diff --git a/README.md b/README.md index 3eb301e6..df910ecb 100644 --- a/README.md +++ b/README.md @@ -152,21 +152,8 @@ fn main() { The generated parser module contains several generated components tailored to your start symbol: - **`Parser`**: A lightweight struct containing the static parsing tables. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/trait.Parser.html) - **`Context`**: A mutable state context that keeps track of the stack and parsed symbol values. [(LR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/deterministic/struct.Context.html) [(GLR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/nondeterministic/struct.Context.html) -- **`State`**: An internal type representing individual states in the parser. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/state/trait.State.html) -- **`Rule`**: An internal enum representing production rules. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/rule/struct.ProductionRule.html) -- **`NonTerminals`**: An enum representing all non-terminal symbols. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/nonterminal/trait.NonTerminal.html) -### Interacting with the Parsing Context -The `Context` offers helpful utilities for inspecting and tracing: -```rust -let mut context = ExprContext::with_default_userdata(); - -// ... feed tokens ... - -context.expected_token(); // Returns the expected symbols for the current state -context.can_feed(&token); // Checks if a terminal can be fed next -println!("{}", context); // Formats the state tree (requires 'tree' feature) -``` +The generated module also includes internal state, rule, and non-terminal types used by the runtime and debugging APIs. ### Feeding Tokens You can feed terminal symbols either with or without location information: @@ -235,6 +222,18 @@ rustylr --state src/grammar.rs ![State Machine Debug](images/state_option.png) +The `Context` also offers inspection utilities that are useful while debugging a parser: + +```rust +let mut context = ExprContext::with_default_userdata(); + +// ... feed tokens ... + +context.expected_token(); // Returns the expected symbols for the current state +context.can_feed(&token); // Checks if a terminal can be fed next +println!("{}", context); // Formats the state tree (requires 'tree' feature) +``` + --- ## Examples diff --git a/SYNTAX.md b/SYNTAX.md index ed3ee84a..94469a37 100644 --- a/SYNTAX.md +++ b/SYNTAX.md @@ -19,12 +19,15 @@ This document provides a comprehensive guide to the grammar definition syntax us - [Conflict Resolution](#resolving-conflicts) - [Panic-Mode Error Recovery (`error`)](#panic-mode-error-recovery) - [Operator Precedence (`%left`, `%right`, `%precedence`, `%prec`)](#operator-precedence) -- [Error Type (`%err` / `%error`)](#error-type-optional) +- [Error Type (`%error`)](#error-type-optional) - [Disabling Table Optimization (`%nooptim`)](#no-optimization) - [Dense Parser Tables (`%dense`)](#dense-parser-table) - [Location Tracking (`%location`)](#location-tracking) - [Variable Substitution](#variable-substitution) - [Advanced Parser Controls](#advanced-parser-controls) + - [Memory Optimization with `box`](#memory-optimization-with-box) + - [Custom Token Ranges](#custom-token-ranges) + - [Dynamic Precedence with Non-Terminals](#dynamic-precedence-with-non-terminals) - [GLR Parser Generation (`%glr`)](#glr-parser-generation) - [Advanced GLR Reduce Controls](#advanced-glr-reduce-controls) - [Rule Priority (`%dprec`)](#rule-priority) @@ -191,21 +194,7 @@ RustyLR supports rich regular expression patterns on the right-hand side of prod - **`"abcd"` / `b"abcd"`** : String/byte string literals (only valid if `%tokentype` is `char` or `u8`). - **`P - TerminalSet`** : Matches pattern `P` but excludes any terminal in the `TerminalSet`. -### Range Patterns (`[first-last]`) -When defining ranges of custom tokens (e.g. `[zero-nine]`), the range ordering is determined by the **declaration order of your `%token` directives**, rather than the inherent values of the enum elements. - -#### Example -If tokens are declared as: -```rust -%token zero Token::Num(0); -%token one Token::Num(1); -%token two Token::Num(2); -// ... -%token nine Token::Num(9); -``` -The range `[zero-two]` matches `zero`, `one`, and `two`. - -If the `%tokentype` is `char` or `u8`, literal character ranges like `['0'-'9']` or `[b'0'-b'9']` are resolved using standard ASCII values. +Literal character and byte ranges such as `['0'-'9']` or `[b'0'-b'9']` are resolved by their literal values. Custom token ranges are also supported for advanced grammars; see [Custom Token Ranges](#custom-token-ranges). --- @@ -229,24 +218,7 @@ Expr(_): Term; > [!WARNING] > If a circular dependency exists (e.g., two rules trying to infer their types from one another without a base type), RustyLR will fail with a compilation error. -### Memory Optimization with `Box` / `box` keyword -Internally, the generated parser stores all semantic values (the `%tokentype` and all non-terminals' `RuleType`s) in a single unified `enum` representing the parser's data stack. - -Because the memory footprint of a Rust `enum` is dictated by its largest variant, if even one `RuleType` is exceptionally large (e.g., a large AST struct), the size of *every* stack slot will inflate. This can result in significant memory waste and performance degradation. - -To avoid this, you can write the `box` keyword in front of `%tokentype` or any non-terminal's `RuleType` (inside the parentheses, e.g. `(box Type)`). The parser generator will automatically box that type in the data enum (generating `Box`), and will automatically wrap (`Box::new(...)`) or unwrap (`*val`) it in reduce actions so that you do not have to write `Box::new` or dereference it yourself: - -```rust -%tokentype box MyToken; - -Expr (box MyLargeASTNode) - : left=Expr '+' right=Term { - // `left` and `right` are automatically unboxed (of type `MyLargeASTNode`), - // and the returned value is automatically wrapped in a `Box::new`. - MyLargeASTNode::Binary(left, right) - } - ; -``` +For large semantic values, RustyLR also supports boxing selected stack variants. See [Memory Optimization with `box`](#memory-optimization-with-box). --- @@ -459,18 +431,11 @@ Expr ; ``` -You can also use a non-terminal in `%prec` if the operator is determined dynamically: -```rust -Expr : Expr op=BinOp Expr %prec op { ... }; - -BinOp : '+' | '*'; -``` -Here, the rule's precedence matches the specific operator that `BinOp` resolved to. +For cases where the operator is represented by a non-terminal, see [Dynamic Precedence with Non-Terminals](#dynamic-precedence-with-non-terminals). ## Error Type (Optional) ``` -%err ; %error ; ``` @@ -532,7 +497,7 @@ You can use variables prefixed with `$` inside any RustCode block in the grammar - `%tokentype` - `%location` - `%userdata` -- `%errortype` (or `%error`) +- `%error` - `%token` terminal definitions - Non-terminal rule types - Reduce actions @@ -599,6 +564,58 @@ This syntax sugar is extremely useful for reducing boilerplate code when process The controls in this section are useful for ambiguous grammars, parser-mode tuning, or conflict cases that cannot be expressed cleanly with ordinary precedence declarations. Most grammars can start without these directives. +### Memory Optimization with `box` + +Internally, the generated parser stores all semantic values (the `%tokentype` and all non-terminals' `RuleType`s) in a single unified `enum` representing the parser's data stack. + +Because the memory footprint of a Rust `enum` is dictated by its largest variant, if even one `RuleType` is exceptionally large (e.g., a large AST struct), the size of *every* stack slot will inflate. This can result in significant memory waste and performance degradation. + +To avoid this, you can write the `box` keyword in front of `%tokentype` or any non-terminal's `RuleType` (inside the parentheses, e.g. `(box Type)`). The parser generator will automatically box that type in the data enum (generating `Box`), and will automatically wrap (`Box::new(...)`) or unwrap (`*val`) it in reduce actions so that you do not have to write `Box::new` or dereference it yourself: + +```rust +%tokentype box MyToken; + +Expr (box MyLargeASTNode) + : left=Expr '+' right=Term { + // `left` and `right` are automatically unboxed (of type `MyLargeASTNode`), + // and the returned value is automatically wrapped in a `Box::new`. + MyLargeASTNode::Binary(left, right) + } + ; +``` + +### Custom Token Ranges + +When defining ranges of custom tokens (e.g. `[zero-nine]`), the range ordering is determined by the **declaration order of your `%token` directives**, rather than the inherent values of the enum elements. + +If tokens are declared as: + +```rust +%token zero Token::Num(0); +%token one Token::Num(1); +%token two Token::Num(2); +// ... +%token nine Token::Num(9); +``` + +The range `[zero-two]` matches `zero`, `one`, and `two`. + +For ordinary grammars, prefer explicit terminal sets like `[zero one two]` unless declaration-order ranges materially simplify the grammar. + +### Dynamic Precedence with Non-Terminals + +You can use a non-terminal in `%prec` if the operator is determined dynamically: + +```rust +Expr : Expr op=BinOp Expr %prec op { ... }; + +BinOp : '+' | '*'; +``` + +Here, the rule's precedence matches the specific operator that `BinOp` resolved to. + +For ordinary unary and binary operators, a named precedence marker such as `UnaryMinus` is usually clearer. + ### GLR Parser Generation ``` diff --git a/example/calculator/src/parser.rs b/example/calculator/src/parser.rs index f23cc145..1c993400 100644 --- a/example/calculator/src/parser.rs +++ b/example/calculator/src/parser.rs @@ -26,7 +26,7 @@ pub enum Token { %start E; // error type -%err String; +%error String; // define tokens %token num Token::Num(_); // `num` maps to `Token::Num(0)` diff --git a/example/glr/src/parser.rs b/example/glr/src/parser.rs index 065f4091..b66835a5 100644 --- a/example/glr/src/parser.rs +++ b/example/glr/src/parser.rs @@ -1,7 +1,7 @@ use rusty_lr::lr1; lr1! { - %err String; + %error String; %glr; %tokentype char; %start E; diff --git a/rusty_lr_executable/README.md b/rusty_lr_executable/README.md index 053cbf95..ce174e23 100644 --- a/rusty_lr_executable/README.md +++ b/rusty_lr_executable/README.md @@ -9,7 +9,16 @@ cargo install rustylr ## Usage ```bash -$ rustylr --help +$ rustylr my_grammar.rs my_parser.rs +``` + +The first argument is the grammar file. The optional second argument is the generated Rust output file; when omitted, RustyLR writes `out.tab.rs`. + +## Advanced CLI Options + +The options below are mainly useful for formatting control, parser-mode overrides, and diagnostics. + +```bash Usage: rustylr [OPTIONS] [OUTPUT_FILE] Arguments: @@ -59,6 +68,12 @@ Options: Print version ``` +### Diagnostic and Debug Options + +- `--state ` prints details for a specific generated state. +- `--no-conflict`, `--no-conflict-resolve`, and `--no-backtrace` reduce conflict diagnostic output. +- `--glr ` and `--dense ` override parser-mode settings from the grammar file. + ## Grammar File Format The program searches for `%%` in the input file to separate Rust code from grammar definitions. diff --git a/rusty_lr_parser/src/grammar.rs b/rusty_lr_parser/src/grammar.rs index 312526b1..e4716560 100644 --- a/rusty_lr_parser/src/grammar.rs +++ b/rusty_lr_parser/src/grammar.rs @@ -78,7 +78,7 @@ pub struct Grammar { /// %userdata pub(crate) userdata_typename: TokenStream, - /// %err + /// %error pub(crate) error_typename: TokenStream, /// %start diff --git a/rusty_lr_parser/src/parser/lexer.rs b/rusty_lr_parser/src/parser/lexer.rs index 3c7f692e..f8e653cd 100644 --- a/rusty_lr_parser/src/parser/lexer.rs +++ b/rusty_lr_parser/src/parser/lexer.rs @@ -56,7 +56,7 @@ pub enum Lexed { Start(Ident), // %start TokenType(Ident), // %tokentype UserData(Ident), // %userdata - ErrorType(Ident), // %err %error + ErrorType(Ident), // %error ModulePrefix(Ident), // %moduleprefix Lalr(Ident), // %lalr Glr(Ident), // %glr @@ -222,7 +222,7 @@ fn ident_to_keyword(ident: Ident) -> Option { "start" => Some(Lexed::Start(ident)), "tokentype" => Some(Lexed::TokenType(ident)), "userdata" => Some(Lexed::UserData(ident)), - "err" | "error" => Some(Lexed::ErrorType(ident)), + "error" => Some(Lexed::ErrorType(ident)), "moduleprefix" => Some(Lexed::ModulePrefix(ident)), "lalr" => Some(Lexed::Lalr(ident)), "glr" => Some(Lexed::Glr(ident)),