Packrat.Lexer.Combinators specification

1 files changed, 47 insertions, 4 deletions

diff --git a/packrat_parser_lib_notes.txt b/packrat_parser_lib_notes.txt
index d9aca31..40b2820 100644
--- a/packrat_parser_lib_notes.txt
+++ b/packrat_parser_lib_notes.txt
@@ -8,9 +8,13 @@ Packrat.Parser.Combinators
 Packrat.Lexer (generic over stamp enum, input item type, array of input items, array of output items wrapped as tokens)
 Packrat.Lexer.Combinators
 Packrat.Util
-Packrat.Error (nested)
+Packrat.Errors (nested)
 Packrat.Graphs (nested, generic over leaf array type)
 Packrat.Tokens (nested, generic over contained array)
+Packrat.Instant
+Packrat.Instant.Standard (nested, generic over parser/lexer label enums)
+Packrat.Instant.Wide (nested, generic over parser/lexer label enums)
+Packrat.Instant.Wide_Wide (nested, generic over parser/lexer label enums)
 
 Ratnest.Tests
 Ratnest.Examples
@@ -27,14 +31,17 @@ planned order of writing:
 (Ratnest not mentioned since that's where all the testing functions will go)
 
 Packrat.Util
-Packrat.Error
+Packrat.Errors
 Packrat.Tokens
 Packrat.Lexer
 Packrat.Lexer.Combinators
 Packrat.Graphs
 Packrat.Parser
 Packrat.Parser.Combinators
-Packrat (any remaining)
+Packrat.Instant
+Packrat.Instant.Standard
+Packrat.Instant.Wide
+Packrat.Instant.Wide_Wide
 Calculator
 Tomita
 Off_Side
@@ -152,16 +159,53 @@ Packrat.Lexer
    sized output array of tokens (and possibly a statically sized input array)
 
 List of funcs:
+(each of these is generic over an array of lexer_component functions, either Stamp or Ignore as below)
 Scan
+ - function that returns an array of lexed tokens
+ - uses first applicable lexer component to lex each token
+ - if all lexer components return "partial" then also returns with a partial status
+ - if all lexer components fail then raises a lexer_error
+ - lexer status can be fed back into any of these Scan functions to resume with further input
+ - if end of input is reached without any lexer components returning "partial" then returns
+   a complete status and feeding the lexer status back into these functions will be the same
+   as starting with a blank status
+Scan_Set
+ - as above, except is a procedure that uses a fixed size array as output with a padding token
 Scan_Only
+ - function that returns an array of lexed tokens
+ - takes a lexer status as input to resuem a lex, but will treat it as a constant unlike the others
+ - if all lexer components return "partial" or fail then raises a lexer_error
+Scan_Set_Only
+ - as above, except is a procedure that uses a fixed size array as output with a padding token
 Scan_With
+ - function that returns an array of lexed tokens
+ - when it runs out of input it uses the supplied function to get more input until that function
+   returns an empty array
+ - may also return with a partial status as with Scan
+Scan_Set_With
+ - as above, except is a procedure that uses a fixed size array as output with a padding token
+
+(type signature of these are:
+  input of an opaque lex_component_input type
+  output of an opaque lex_component_output type
+  return of a Fail/Partial/Success result enum)
 Stamp
+ - one of two available lexer component functions
+ - generic over a value of the enum used for labelling lexemes and a lexer combinator
+Ignore
+ - the other available lexer component function
+ - generic over a lexer combinator, as this function will scan for the given token and then ignore it
 
 
 
 
 Packrat.Lexer.Combinators
 
+type signature of these are:
+  inputs of input_array and a starting position index
+  outputs of the number of elements consumed and the value lexed
+  return of a Fail/Partial/Success result enum
+
 List of funcs:
 Sequence
 Count
@@ -173,7 +217,6 @@ Satisfy_With
 Match
 Match_With
 Multimatch
-Multimatch_With
 Take
 Take_While
 Take_Until