From 67edc77677d366d4dc0ff42154c79a1d87c2dd29 Mon Sep 17 00:00:00 2001 From: Jed Barber Date: Thu, 10 Jan 2019 15:27:56 +1100 Subject: Packrat.Lexer.Combinators specification --- packrat_parser_lib_notes.txt | 51 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 47 insertions(+), 4 deletions(-) (limited to 'packrat_parser_lib_notes.txt') diff --git a/packrat_parser_lib_notes.txt b/packrat_parser_lib_notes.txt index d9aca31..40b2820 100644 --- a/packrat_parser_lib_notes.txt +++ b/packrat_parser_lib_notes.txt @@ -8,9 +8,13 @@ Packrat.Parser.Combinators Packrat.Lexer (generic over stamp enum, input item type, array of input items, array of output items wrapped as tokens) Packrat.Lexer.Combinators Packrat.Util -Packrat.Error (nested) +Packrat.Errors (nested) Packrat.Graphs (nested, generic over leaf array type) Packrat.Tokens (nested, generic over contained array) +Packrat.Instant +Packrat.Instant.Standard (nested, generic over parser/lexer label enums) +Packrat.Instant.Wide (nested, generic over parser/lexer label enums) +Packrat.Instant.Wide_Wide (nested, generic over parser/lexer label enums) Ratnest.Tests Ratnest.Examples @@ -27,14 +31,17 @@ planned order of writing: (Ratnest not mentioned since that's where all the testing functions will go) Packrat.Util -Packrat.Error +Packrat.Errors Packrat.Tokens Packrat.Lexer Packrat.Lexer.Combinators Packrat.Graphs Packrat.Parser Packrat.Parser.Combinators -Packrat (any remaining) +Packrat.Instant +Packrat.Instant.Standard +Packrat.Instant.Wide +Packrat.Instant.Wide_Wide Calculator Tomita Off_Side @@ -152,16 +159,53 @@ Packrat.Lexer sized output array of tokens (and possibly a statically sized input array) List of funcs: +(each of these is generic over an array of lexer_component functions, either Stamp or Ignore as below) Scan + - function that returns an array of lexed tokens + - uses first applicable lexer component to lex each token + - if all lexer components return "partial" then also returns with a partial status + - if all lexer components fail then raises a lexer_error + - lexer status can be fed back into any of these Scan functions to resume with further input + - if end of input is reached without any lexer components returning "partial" then returns + a complete status and feeding the lexer status back into these functions will be the same + as starting with a blank status +Scan_Set + - as above, except is a procedure that uses a fixed size array as output with a padding token Scan_Only + - function that returns an array of lexed tokens + - takes a lexer status as input to resuem a lex, but will treat it as a constant unlike the others + - if all lexer components return "partial" or fail then raises a lexer_error +Scan_Set_Only + - as above, except is a procedure that uses a fixed size array as output with a padding token Scan_With + - function that returns an array of lexed tokens + - when it runs out of input it uses the supplied function to get more input until that function + returns an empty array + - may also return with a partial status as with Scan +Scan_Set_With + - as above, except is a procedure that uses a fixed size array as output with a padding token + +(type signature of these are: + input of an opaque lex_component_input type + output of an opaque lex_component_output type + return of a Fail/Partial/Success result enum) Stamp + - one of two available lexer component functions + - generic over a value of the enum used for labelling lexemes and a lexer combinator +Ignore + - the other available lexer component function + - generic over a lexer combinator, as this function will scan for the given token and then ignore it Packrat.Lexer.Combinators +type signature of these are: + inputs of input_array and a starting position index + outputs of the number of elements consumed and the value lexed + return of a Fail/Partial/Success result enum + List of funcs: Sequence Count @@ -173,7 +217,6 @@ Satisfy_With Match Match_With Multimatch -Multimatch_With Take Take_While Take_Until -- cgit