1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
|
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Lexer Components - Packrat Docs</title>
<link href="default.css" rel="stylesheet">
</head>
<body>
<h2>Lexer Components</h2>
<a href="index.html">Return to Contents</a>
<p>Lexers are a much simplified version of the parsers. They operate like one giant
<em>Choice</em> combinator, the other combinators that you can combine with are more
limited, all combinators operate in the more usual greedy consume-as-much-as-possible
fashion, and the output is a sequence of tokens instead of a parse graph.<br>
<br>
Think of it like grouping items of input together and labelling the groups, instead of
trying to create any complex parses.<br>
<br>
Usage of lexers is optional.</p>
<table>
<tr>
<td><pre>
generic
Components : in Component_Array;
package Scan_Parts is
function Scan
(Input : in Traits.Element_Array)
return Traits.Tokens.Token_Array;
procedure Reset;
end Scan_Parts;
</pre></td>
<td>The scanner version that allows for piecewise scanning. The <em>Scan</em> function can be
called repeatedly with new input until it is supplied with an empty array, at which point it
is assumed that the end of input has been reached.<br>
<br>
The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
declared, with no partial results stored internally.</td>
</tr>
<tr>
<td><pre>
generic
Components : in Component_Array;
package Scan_Once is
function Scan
(Input : in Traits.Element_Array)
return Traits.Tokens.Token_Array;
procedure Reset;
end Scan_Once;
</pre></td>
<td>This scanner version assumes that the input supplied is the only input available.<br>
<br>
The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
declared.</td>
</tr>
<tr>
<td><pre>
generic
Components : in Component_Array;
package Scan_With is
function Scan
(Input : in With_Input)
return Traits.Tokens.Token_Array;
procedure Reset;
end Scan_With;
</pre></td>
<td>This scanner uses the supplied function to retrieve input to be lexed. Like the <em>Scan_Parts</em>
scanner it will continue to call the function until an empty array is returned, at which point it is
assumed that the end of input has been reached.<br>
<br>
The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just
declared.</td>
</tr>
<tr>
<td><pre>
generic
Components : in Component_Array;
Pad_In : in Traits.Element_Type;
Pad_Out : in Traits.Tokens.Token_Type;
package Scan_Set is
procedure Scan
(Input : in Traits.Element_Array;
Output : out Traits.Tokens.Token_Array);
procedure Reset;
end Scan_Set;
</pre></td>
<td>This scanner allows the use of constant length input and output arrays. Scanning is done piecewise
until the input array begins with the supplied <em>Pad_In</em> character, upon which it is assumed that
the end of input has been reached. Any unused space in the output array is padded with the <em>Pad_Out</em>
character.<br>
<br>
The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.<br>
<br>
Note that the input and output array lengths do not have to remain the same between successive calls to
<em>Scan</em>.</td>
</tr>
<tr>
<td><pre>
generic
Components : in Component_Array;
Pad_In : in Traits.Element_Type;
Pad_Out : in Traits.Tokens.Token_Type;
package Scan_Set_With is
procedure Scan
(Input : in With_Input;
Output : out Traits.Tokens.Token_Array);
procedure Reset;
end Scan_Set_With;
</pre></td>
<td>A combination of <em>Scan_With</em> and <em>Scan_Set</em>, this scanner uses the supplied function
to retrieve input to be lexed, but uses padding characters in the same manner as <em>Scan_Set</em>.<br>
<br>
The <em>Reset</em> procedure reverts the scanner to a clean state the same as if it was just declared.</td>
</tr>
<tr>
<td><pre>
generic
Label : in Traits.Label_Enum;
with function Combo
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
function Stamp
(Input : in Traits.Element_Array;
Context : in out Lexer_Context)
return Component_Result;
</pre></td>
<td>Runs the supplied combinator, and if successful converts the result into a token with the supplied
label that then gets added to the output. Scanning then proceeds from just after the input covered by
the token.</td>
</tr>
<tr>
<td><pre>
generic
Label : in Traits.Label_Enum;
with function Combo
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
function Discard
(Input : in Traits.Element_Array;
Context : in out Lexer_Context)
return Component_Result;
</pre></td>
<td>Does the same thing as <em>Stamp</em>, however rather than adding the token to the output it instead
discards it. Scanning still proceeds from just after the input covered by the token.<br>
<br>
Even though the result is ultimately discarded it is nonetheless important that a label is used here so
that informative error messages can be generated.</td>
</tr>
<tr>
<td><pre>
generic
Params : in Combinator_Array;
function Sequence
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes the supplied combinator <em>Params</em> in order. If they are all successful then the joined
successful result is returned, otherwise the combinator fails.</td>
</tr>
<tr>
<td><pre>
generic
with function Param
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
Number : in Positive;
function Count
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes the supplied combinator <em>Param</em> the supplied <em>Number</em> of times one after another.
If it is all successful then the joined successful result is returned, otherwise the combinator fails.</td>
</tr>
<tr>
<td><pre>
generic
with function Param
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
Minimum : in Natural := 0;
function Many
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes the supplied combinator <em>Param</em> as many times as it can. Succeeds if it manages to do
so at least <em>Minimum</em> number of times.</td>
</tr>
<tr>
<td><pre>
generic
with function Param
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
with function Test
(Item : in Traits.Element_Type)
return Boolean;
Minimum : in Natural := 0;
function Many_Until
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes the supplied combinator <em>Param</em> over and over until <em>Test</em> succeeds on the next
available item in the input. Succeeds if it manages to lex <em>Param</em> at least <em>Minimum</em> times.
Fails if <em>Param</em> fails before <em>Test</em> succeeds on the next available item of input.</td>
</tr>
<tr>
<td><pre>
generic
with function Test
(Item : in Traits.Element_Type)
return Boolean;
function Satisfy
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes one item of input if the supplied <em>Test</em> function succeeds on that item, otherwise fails.</td>
</tr>
<tr>
<td><pre>
generic
with function Test
(Item : in Traits.Element_Type)
return Boolean;
with function Change
(From : in Traits.Element_Type)
return Traits.Element_Type;
function Satisfy_With
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>First uses the supplied <em>Change</em> function on the first item of input, then acts as in <em>Satisfy</em>
on the changed item. Upon success the item lexed is the original item.</td>
</tr>
<tr>
<td><pre>
generic
Item : in Traits.Element_Type;
function Match
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes one item of input if it matches <em>Item</em>.</td>
</tr>
<tr>
<td><pre>
generic
Item : in Traits.Element_Type;
with function Change
(From : in Traits.Element_Type)
return Traits.Element_Type;
function Match_With
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes one item of input if, after applying the supplied <em>Change</em> function to it, it
matches <em>Item</em>.</td>
</tr>
<tr>
<td><pre>
generic
Items : in Traits.Element_Array;
function Multimatch
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes the next <em>Items'Length</em> items of input if they match <em>Items</em>.</td>
</tr>
<tr>
<td><pre>
generic
Number : in Positive := 1;
function Take
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes the next <em>Number</em> items of input.</td>
</tr>
<tr>
<td><pre>
generic
with function Test
(Item : in Traits.Element_Type)
return Boolean;
function Take_While
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes successive items of input as long as they satisfy the supplied <em>Test</em> function.</td>
</tr>
<tr>
<td><pre>
generic
with function Test
(Item : in Traits.Element_Type)
return Boolean;
function Take_Until
(Input : in Traits.Element_Array;
Start : in Positive)
return Combinator_Result;
</pre></td>
<td>Lexes items of input until the next available item satisfies the supplied <em>Test</em> function.</td>
</tr>
</table>
</body>
</html>
|
