Basic HTML documentation added

author: Jed Barber <jjbarber@y7mail.com> 2021-01-19 02:25:44 +1100
committer: Jed Barber <jjbarber@y7mail.com> 2021-01-19 02:25:44 +1100
commit: da389927ddf9240bbba10b819eb782e80a5d6bf7 (patch)
tree: 88b167b892de8c419151ca9a02c16f8d4ed96bec /doc/lexer_comb.html
parent: 901c7e61278f16dc5c08cbebf74c2a816cd4b4b6 (diff)
1 files changed, 347 insertions, 0 deletions
diff --git a/doc/lexer_comb.html b/doc/lexer_comb.html
new file mode 100644
index 0000000..eef5c04
--- /dev/null
+++ b/doc/lexer_comb.html
@@ -0,0 +1,347 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <title>Lexer Components - Packrat Docs</title>
+    <link href="default.css" rel="stylesheet">
+  </head>
+
+  <body>
+
+
+  <h2>Lexer Components</h2>
+
+  <a href="index.html">Return to Contents</a>
+
+
+  <p>Lexers are a much simplified version of the parsers. They operate like one giant
+  <em>Choice</em> combinator, the other combinators that you can combine with are more
+  limited, all combinators operate in the more usual greedy consume-as-much-as-possible
+  fashion, and the output is a sequence of tokens instead of a parse graph.<br>
+  <br>
+  Think of it like grouping items of input together and labelling the groups, instead of
+  trying to create any complex parses.<br>
+  <br>
+  Usage of lexers is optional.</p>
+
+  <table>
+    <tr>
+<td><pre>
+generic
+    Components : in Component_Array;
+package Scan_Parts is
+
+    function Scan
+           (Input : in Traits.Element_Array)
+        return Traits.Tokens.Token_Array;
+
+    procedure Reset;
+
+end Scan_Parts;
+</pre></td>
+<td>The scanner version that allows for piecewise scanning. The <em>Scan</em> function can be
+called repeatedly with new input until it is supplied with an empty array, at which point it
+is assumed that the end of input has been reached.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
+declared, with no partial results stored internally.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Components : in Component_Array;
+package Scan_Once is
+
+    function Scan
+           (Input : in Traits.Element_Array)
+        return Traits.Tokens.Token_Array;
+
+    procedure Reset;
+
+end Scan_Once;
+</pre></td>
+<td>This scanner version assumes that the input supplied is the only input available.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
+declared.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Components : in Component_Array;
+package Scan_With is
+
+    function Scan
+           (Input : in With_Input)
+        return Traits.Tokens.Token_Array;
+
+    procedure Reset;
+
+end Scan_With;
+</pre></td>
+<td>This scanner uses the supplied function to retrieve input to be lexed. Like the <em>Scan_Parts</em>
+scanner it will continue to call the function until an empty array is returned, at which point it is
+assumed that the end of input has been reached.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
+declared.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Components : in Component_Array;
+    Pad_In     : in Traits.Element_Type;
+    Pad_Out    : in Traits.Tokens.Token_Type;
+package Scan_Set is
+
+    procedure Scan
+           (Input  : in     Traits.Element_Array;
+            Output :    out Traits.Tokens.Token_Array);
+
+    procedure Reset;
+
+end Scan_Set;
+</pre></td>
+<td>This scanner allows the use of constant length input and output arrays. Scanning is done piecewise
+until the input array begins with the supplied <em>Pad_In</em> character, upon which it is assumed that
+the end of input has been reached. Any unused space in the output array is padded with the <em>Pad_Out</em>
+character.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.<br>
+<br>
+Note that the input and output array lengths do not have to remain the same between successive calls to
+<em>Scan</em>.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Components : in Component_Array;
+    Pad_In     : in Traits.Element_Type;
+    Pad_Out    : in Traits.Tokens.Token_Type;
+package Scan_Set_With is
+
+    procedure Scan
+           (Input  : in     With_Input;
+            Output :    out Traits.Tokens.Token_Array);
+
+    procedure Reset;
+
+end Scan_Set_With;
+</pre></td>
+<td>A combination of <em>Scan_With</em> and <em>Scan_Set</em>, this scanner uses the supplied function
+to retrieve input to be lexed, but uses padding characters in the same manner as <em>Scan_Set</em>.<br>
+<br>
+The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Label : in Traits.Label_Enum;
+    with function Combo
+           (Input : in Traits.Element_Array;
+            Start : in Positive)
+        return Combinator_Result;
+function Stamp
+       (Input   : in     Traits.Element_Array;
+        Context : in out Lexer_Context)
+    return Component_Result;
+</pre></td>
+<td>Runs the supplied combinator, and if successful converts the result into a token with the supplied
+label that then gets added to the output. Scanning then proceeds from just after the input covered by
+the token.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Label : in Traits.Label_Enum;
+    with function Combo
+           (Input : in Traits.Element_Array;
+            Start : in Positive)
+        return Combinator_Result;
+function Discard
+       (Input   : in     Traits.Element_Array;
+        Context : in out Lexer_Context)
+    return Component_Result;
+</pre></td>
+<td>Does the same thing as <em>Stamp</em>, however rather than adding the token to the output it instead
+discards it. Scanning still proceeds from just after the input covered by the token.<br>
+<br>
+Even though the result is ultimately discarded it is nonetheless important that a label is used here so
+that informative error messages can be generated.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Params : in Combinator_Array;
+function Sequence
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Params</em> in order. If they are all successful then the joined
+successful result is returned, otherwise the combinator fails.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    with function Param
+           (Input : in Traits.Element_Array;
+            Start : in Positive)
+        return Combinator_Result;
+    Number : in Positive;
+function Count
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Param</em> the supplied <em>Number</em> of times one after another.
+If it is all successful then the joined successful result is returned, otherwise the combinator fails.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    with function Param
+           (Input : in Traits.Element_Array;
+            Start : in Positive)
+        return Combinator_Result;
+    Minimum : in Natural := 0;
+function Many
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Param</em> as many times as it can. Succeeds if it manages to do
+so at least <em>Minimum</em> number of times.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    with function Param
+           (Input : in Traits.Element_Array;
+            Start : in Positive)
+        return Combinator_Result;
+    with function Test
+           (Item : in Traits.Element_Type)
+        return Boolean;
+    Minimum : in Natural := 0;
+function Many_Until
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes the supplied combinator <em>Param</em> over and over until <em>Test</em> succeeds on the next
+available item in the input. Succeeds if it manages to lex <em>Param</em> at least <em>Minimum</em> times.
+Fails if <em>Param</em> fails before <em>Test</em> succeeds on the next available item of input.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    with function Test
+           (Item : in Traits.Element_Type)
+        return Boolean;
+function Satisfy
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes one item of input if the supplied <em>Test</em> function succeeds on that item, otherwise fails.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    with function Test
+           (Item : in Traits.Element_Type)
+        return Boolean;
+    with function Change
+           (From : in Traits.Element_Type)
+        return Traits.Element_Type;
+function Satisfy_With
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>First uses the supplied <em>Change</em> function on the first item of input, then acts as in <em>Satisfy</em>
+on the changed item. Upon success the item lexed is the original item.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Item : in Traits.Element_Type;
+function Match
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes one item of input if it matches <em>Item</em>.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Item : in Traits.Element_Type;
+    with function Change
+           (From : in Traits.Element_Type)
+        return Traits.Element_Type;
+function Match_With
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes one item of input if, after applying the supplied <em>Change</em> function to it, it
+matches <em>Item</em>.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Items : in Traits.Element_Array;
+function Multimatch
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes the next <em>Items'Length</em> items of input if they match <em>Items</em>.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    Number : in Positive := 1;
+function Take
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes the next <em>Number</em> items of input.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    with function Test
+           (Item : in Traits.Element_Type)
+        return Boolean;
+function Take_While
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes successive items of input as long as they satisfy the supplied <em>Test</em> function.</td>
+    </tr>
+    <tr>
+<td><pre>
+generic
+    with function Test
+           (Item : in Traits.Element_Type)
+        return Boolean;
+function Take_Until
+       (Input : in Traits.Element_Array;
+        Start : in Positive)
+    return Combinator_Result;
+</pre></td>
+<td>Lexes items of input until the next available item satisfies the supplied <em>Test</em> function.</td>
+    </tr>
+  </table>
+
+
+  </body>
+</html>
+
author	Jed Barber <jjbarber@y7mail.com>	2021-01-19 02:25:44 +1100
committer	Jed Barber <jjbarber@y7mail.com>	2021-01-19 02:25:44 +1100
commit	da389927ddf9240bbba10b819eb782e80a5d6bf7 (patch)
tree	88b167b892de8c419151ca9a02c16f8d4ed96bec /doc/lexer_comb.html
parent	901c7e61278f16dc5c08cbebf74c2a816cd4b4b6 (diff)