summaryrefslogtreecommitdiff
path: root/doc/lexer_comb.html
diff options
context:
space:
mode:
Diffstat (limited to 'doc/lexer_comb.html')
-rw-r--r--doc/lexer_comb.html347
1 files changed, 347 insertions, 0 deletions
diff --git a/doc/lexer_comb.html b/doc/lexer_comb.html
new file mode 100644
index 0000000..eef5c04
--- /dev/null
+++ b/doc/lexer_comb.html
@@ -0,0 +1,347 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <title>Lexer Components - Packrat Docs</title>
+ <link href="default.css" rel="stylesheet">
+ </head>
+
+ <body>
+
+
+ <h2>Lexer Components</h2>
+
+ <a href="index.html">Return to Contents</a>
+
+
+ <p>Lexers are a much simplified version of the parsers. They operate like one giant
+ <em>Choice</em> combinator, the other combinators that you can combine with are more
+ limited, all combinators operate in the more usual greedy consume-as-much-as-possible
+ fashion, and the output is a sequence of tokens instead of a parse graph.<br>
+ <br>
+ Think of it like grouping items of input together and labelling the groups, instead of
+ trying to create any complex parses.<br>
+ <br>
+ Usage of lexers is optional.</p>
+
+ <table>
+ <tr>
+<td><pre>
+generic
+ Components : in Component_Array;
+package Scan_Parts is
+
+ function Scan
+ (Input : in Traits.Element_Array)
+ return Traits.Tokens.Token_Array;
+
+ procedure Reset;
+
+end Scan_Parts;
+</pre></td>
+<td>The scanner version that allows for piecewise scanning. The <em>Scan</em> function can be
+called repeatedly with new input until it is supplied with an empty array, at which point it
+is assumed that the end of input has been reached.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
+declared, with no partial results stored internally.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Components : in Component_Array;
+package Scan_Once is
+
+ function Scan
+ (Input : in Traits.Element_Array)
+ return Traits.Tokens.Token_Array;
+
+ procedure Reset;
+
+end Scan_Once;
+</pre></td>
+<td>This scanner version assumes that the input supplied is the only input available.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
+declared.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Components : in Component_Array;
+package Scan_With is
+
+ function Scan
+ (Input : in With_Input)
+ return Traits.Tokens.Token_Array;
+
+ procedure Reset;
+
+end Scan_With;
+</pre></td>
+<td>This scanner uses the supplied function to retrieve input to be lexed. Like the <em>Scan_Parts</em>
+scanner it will continue to call the function until an empty array is returned, at which point it is
+assumed that the end of input has been reached.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
+declared.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Components : in Component_Array;
+ Pad_In : in Traits.Element_Type;
+ Pad_Out : in Traits.Tokens.Token_Type;
+package Scan_Set is
+
+ procedure Scan
+ (Input : in Traits.Element_Array;
+ Output : out Traits.Tokens.Token_Array);
+
+ procedure Reset;
+
+end Scan_Set;
+</pre></td>
+<td>This scanner allows the use of constant length input and output arrays. Scanning is done piecewise
+until the input array begins with the supplied <em>Pad_In</em> character, upon which it is assumed that
+the end of input has been reached. Any unused space in the output array is padded with the <em>Pad_Out</em>
+character.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.<br>
+<br>
+Note that the input and output array lengths do not have to remain the same between successive calls to
+<em>Scan</em>.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Components : in Component_Array;
+ Pad_In : in Traits.Element_Type;
+ Pad_Out : in Traits.Tokens.Token_Type;
+package Scan_Set_With is
+
+ procedure Scan
+ (Input : in With_Input;
+ Output : out Traits.Tokens.Token_Array);
+
+ procedure Reset;
+
+end Scan_Set_With;
+</pre></td>
+<td>A combination of <em>Scan_With</em> and <em>Scan_Set</em>, this scanner uses the supplied function
+to retrieve input to be lexed, but uses padding characters in the same manner as <em>Scan_Set</em>.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Label : in Traits.Label_Enum;
+ with function Combo
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+function Stamp
+ (Input : in Traits.Element_Array;
+ Context : in out Lexer_Context)
+ return Component_Result;
+</pre></td>
+<td>Runs the supplied combinator, and if successful converts the result into a token with the supplied
+label that then gets added to the output. Scanning then proceeds from just after the input covered by
+the token.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Label : in Traits.Label_Enum;
+ with function Combo
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+function Discard
+ (Input : in Traits.Element_Array;
+ Context : in out Lexer_Context)
+ return Component_Result;
+</pre></td>
+<td>Does the same thing as <em>Stamp</em>, however rather than adding the token to the output it instead
+discards it. Scanning still proceeds from just after the input covered by the token.<br>
+<br>
+Even though the result is ultimately discarded it is nonetheless important that a label is used here so
+that informative error messages can be generated.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Params : in Combinator_Array;
+function Sequence
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Params</em> in order. If they are all successful then the joined
+successful result is returned, otherwise the combinator fails.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ with function Param
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+ Number : in Positive;
+function Count
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Param</em> the supplied <em>Number</em> of times one after another.
+If it is all successful then the joined successful result is returned, otherwise the combinator fails.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ with function Param
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+ Minimum : in Natural := 0;
+function Many
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Param</em> as many times as it can. Succeeds if it manages to do
+so at least <em>Minimum</em> number of times.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ with function Param
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+ with function Test
+ (Item : in Traits.Element_Type)
+ return Boolean;
+ Minimum : in Natural := 0;
+function Many_Until
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Param</em> over and over until <em>Test</em> succeeds on the next
+available item in the input. Succeeds if it manages to lex <em>Param</em> at least <em>Minimum</em> times.
+Fails if <em>Param</em> fails before <em>Test</em> succeeds on the next available item of input.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ with function Test
+ (Item : in Traits.Element_Type)
+ return Boolean;
+function Satisfy
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes one item of input if the supplied <em>Test</em> function succeeds on that item, otherwise fails.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ with function Test
+ (Item : in Traits.Element_Type)
+ return Boolean;
+ with function Change
+ (From : in Traits.Element_Type)
+ return Traits.Element_Type;
+function Satisfy_With
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>First uses the supplied <em>Change</em> function on the first item of input, then acts as in <em>Satisfy</em>
+on the changed item. Upon success the item lexed is the original item.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Item : in Traits.Element_Type;
+function Match
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes one item of input if it matches <em>Item</em>.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Item : in Traits.Element_Type;
+ with function Change
+ (From : in Traits.Element_Type)
+ return Traits.Element_Type;
+function Match_With
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes one item of input if, after applying the supplied <em>Change</em> function to it, it
+matches <em>Item</em>.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Items : in Traits.Element_Array;
+function Multimatch
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes the next <em>Items'Length</em> items of input if they match <em>Items</em>.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ Number : in Positive := 1;
+function Take
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes the next <em>Number</em> items of input.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ with function Test
+ (Item : in Traits.Element_Type)
+ return Boolean;
+function Take_While
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes successive items of input as long as they satisfy the supplied <em>Test</em> function.</td>
+ </tr>
+ <tr>
+<td><pre>
+generic
+ with function Test
+ (Item : in Traits.Element_Type)
+ return Boolean;
+function Take_Until
+ (Input : in Traits.Element_Array;
+ Start : in Positive)
+ return Combinator_Result;
+</pre></td>
+<td>Lexes items of input until the next available item satisfies the supplied <em>Test</em> function.</td>
+ </tr>
+ </table>
+
+
+ </body>
+</html>
+