diff options
Diffstat (limited to 'doc/lexer_comb.html')
| -rw-r--r-- | doc/lexer_comb.html | 347 |
1 files changed, 347 insertions, 0 deletions
diff --git a/doc/lexer_comb.html b/doc/lexer_comb.html new file mode 100644 index 0000000..eef5c04 --- /dev/null +++ b/doc/lexer_comb.html @@ -0,0 +1,347 @@ + +<!DOCTYPE html> + +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>Lexer Components - Packrat Docs</title> + <link href="default.css" rel="stylesheet"> + </head> + + <body> + + + <h2>Lexer Components</h2> + + <a href="index.html">Return to Contents</a> + + + <p>Lexers are a much simplified version of the parsers. They operate like one giant + <em>Choice</em> combinator, the other combinators that you can combine with are more + limited, all combinators operate in the more usual greedy consume-as-much-as-possible + fashion, and the output is a sequence of tokens instead of a parse graph.<br> + <br> + Think of it like grouping items of input together and labelling the groups, instead of + trying to create any complex parses.<br> + <br> + Usage of lexers is optional.</p> + + <table> + <tr> +<td><pre> +generic + Components : in Component_Array; +package Scan_Parts is + + function Scan + (Input : in Traits.Element_Array) + return Traits.Tokens.Token_Array; + + procedure Reset; + +end Scan_Parts; +</pre></td> +<td>The scanner version that allows for piecewise scanning. The <em>Scan</em> function can be +called repeatedly with new input until it is supplied with an empty array, at which point it +is assumed that the end of input has been reached.<br> +<br> +The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just +declared, with no partial results stored internally.</td> + </tr> + <tr> +<td><pre> +generic + Components : in Component_Array; +package Scan_Once is + + function Scan + (Input : in Traits.Element_Array) + return Traits.Tokens.Token_Array; + + procedure Reset; + +end Scan_Once; +</pre></td> +<td>This scanner version assumes that the input supplied is the only input available.<br> +<br> +The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just +declared.</td> + </tr> + <tr> +<td><pre> +generic + Components : in Component_Array; +package Scan_With is + + function Scan + (Input : in With_Input) + return Traits.Tokens.Token_Array; + + procedure Reset; + +end Scan_With; +</pre></td> +<td>This scanner uses the supplied function to retrieve input to be lexed. Like the <em>Scan_Parts</em> +scanner it will continue to call the function until an empty array is returned, at which point it is +assumed that the end of input has been reached.<br> +<br> +The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just +declared.</td> + </tr> + <tr> +<td><pre> +generic + Components : in Component_Array; + Pad_In : in Traits.Element_Type; + Pad_Out : in Traits.Tokens.Token_Type; +package Scan_Set is + + procedure Scan + (Input : in Traits.Element_Array; + Output : out Traits.Tokens.Token_Array); + + procedure Reset; + +end Scan_Set; +</pre></td> +<td>This scanner allows the use of constant length input and output arrays. Scanning is done piecewise +until the input array begins with the supplied <em>Pad_In</em> character, upon which it is assumed that +the end of input has been reached. Any unused space in the output array is padded with the <em>Pad_Out</em> +character.<br> +<br> +The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.<br> +<br> +Note that the input and output array lengths do not have to remain the same between successive calls to +<em>Scan</em>.</td> + </tr> + <tr> +<td><pre> +generic + Components : in Component_Array; + Pad_In : in Traits.Element_Type; + Pad_Out : in Traits.Tokens.Token_Type; +package Scan_Set_With is + + procedure Scan + (Input : in With_Input; + Output : out Traits.Tokens.Token_Array); + + procedure Reset; + +end Scan_Set_With; +</pre></td> +<td>A combination of <em>Scan_With</em> and <em>Scan_Set</em>, this scanner uses the supplied function +to retrieve input to be lexed, but uses padding characters in the same manner as <em>Scan_Set</em>.<br> +<br> +The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.</td> + </tr> + <tr> +<td><pre> +generic + Label : in Traits.Label_Enum; + with function Combo + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +function Stamp + (Input : in Traits.Element_Array; + Context : in out Lexer_Context) + return Component_Result; +</pre></td> +<td>Runs the supplied combinator, and if successful converts the result into a token with the supplied +label that then gets added to the output. Scanning then proceeds from just after the input covered by +the token.</td> + </tr> + <tr> +<td><pre> +generic + Label : in Traits.Label_Enum; + with function Combo + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +function Discard + (Input : in Traits.Element_Array; + Context : in out Lexer_Context) + return Component_Result; +</pre></td> +<td>Does the same thing as <em>Stamp</em>, however rather than adding the token to the output it instead +discards it. Scanning still proceeds from just after the input covered by the token.<br> +<br> +Even though the result is ultimately discarded it is nonetheless important that a label is used here so +that informative error messages can be generated.</td> + </tr> + <tr> +<td><pre> +generic + Params : in Combinator_Array; +function Sequence + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes the supplied combinator <em>Params</em> in order. If they are all successful then the joined +successful result is returned, otherwise the combinator fails.</td> + </tr> + <tr> +<td><pre> +generic + with function Param + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; + Number : in Positive; +function Count + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes the supplied combinator <em>Param</em> the supplied <em>Number</em> of times one after another. +If it is all successful then the joined successful result is returned, otherwise the combinator fails.</td> + </tr> + <tr> +<td><pre> +generic + with function Param + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; + Minimum : in Natural := 0; +function Many + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes the supplied combinator <em>Param</em> as many times as it can. Succeeds if it manages to do +so at least <em>Minimum</em> number of times.</td> + </tr> + <tr> +<td><pre> +generic + with function Param + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; + with function Test + (Item : in Traits.Element_Type) + return Boolean; + Minimum : in Natural := 0; +function Many_Until + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes the supplied combinator <em>Param</em> over and over until <em>Test</em> succeeds on the next +available item in the input. Succeeds if it manages to lex <em>Param</em> at least <em>Minimum</em> times. +Fails if <em>Param</em> fails before <em>Test</em> succeeds on the next available item of input.</td> + </tr> + <tr> +<td><pre> +generic + with function Test + (Item : in Traits.Element_Type) + return Boolean; +function Satisfy + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes one item of input if the supplied <em>Test</em> function succeeds on that item, otherwise fails.</td> + </tr> + <tr> +<td><pre> +generic + with function Test + (Item : in Traits.Element_Type) + return Boolean; + with function Change + (From : in Traits.Element_Type) + return Traits.Element_Type; +function Satisfy_With + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>First uses the supplied <em>Change</em> function on the first item of input, then acts as in <em>Satisfy</em> +on the changed item. Upon success the item lexed is the original item.</td> + </tr> + <tr> +<td><pre> +generic + Item : in Traits.Element_Type; +function Match + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes one item of input if it matches <em>Item</em>.</td> + </tr> + <tr> +<td><pre> +generic + Item : in Traits.Element_Type; + with function Change + (From : in Traits.Element_Type) + return Traits.Element_Type; +function Match_With + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes one item of input if, after applying the supplied <em>Change</em> function to it, it +matches <em>Item</em>.</td> + </tr> + <tr> +<td><pre> +generic + Items : in Traits.Element_Array; +function Multimatch + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes the next <em>Items'Length</em> items of input if they match <em>Items</em>.</td> + </tr> + <tr> +<td><pre> +generic + Number : in Positive := 1; +function Take + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes the next <em>Number</em> items of input.</td> + </tr> + <tr> +<td><pre> +generic + with function Test + (Item : in Traits.Element_Type) + return Boolean; +function Take_While + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes successive items of input as long as they satisfy the supplied <em>Test</em> function.</td> + </tr> + <tr> +<td><pre> +generic + with function Test + (Item : in Traits.Element_Type) + return Boolean; +function Take_Until + (Input : in Traits.Element_Array; + Start : in Positive) + return Combinator_Result; +</pre></td> +<td>Lexes items of input until the next available item satisfies the supplied <em>Test</em> function.</td> + </tr> + </table> + + + </body> +</html> + |