diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md new file mode 100644 index 00000000..c2e44136 --- /dev/null +++ b/.agents/AGENTS.md @@ -0,0 +1,31 @@ +# Agent Guidelines for RustyLR + +This document outlines rules and instructions that AI agents must follow when modifying or contributing to the RustyLR codebase. + +--- + +## 1. Keep Documentation Synchronized +Whenever a change is made to the parser generator's specifications, APIs, or runtime behavior, you must update the corresponding documentation files (`README.md`, `SYNTAX.md`, or `GLR.md`) immediately. + +## 2. Documenting Syntax Modifications +In particular, when new grammar rules, directives, pattern operators, or reduce action features are introduced: +- You must add a detailed explanation with examples inside [SYNTAX.md](../SYNTAX.md). +- Ensure the documentation style matches the formatting, tone, and organization of existing syntax references. +- Preserve the exact header anchor names that are referenced as diagnostic URLs within the codebase (e.g., in [parser.rs](../rusty_lr_parser/src/parser/parser.rs)). + +## 3. Verify and Bootstrap +After completing any implementation work or documentation refactoring: +- You must run the project's bootstrap test script: + ```bash + ./scripts/bootstrap_test.sh + ``` +- This script verifies that the parser compiles its own grammar correctly, checks output identity, compiles example crates, and runs all workspace unit/integration tests to ensure no regressions are introduced. + +## 4. Writing Pull Requests (PRs) +When requested by the user to write a Pull Request (PR): +- You must write the PR title and description in clean, professional English. +- Output the content formatted inside raw markdown blocks so that the user can copy and paste it directly. + +## 5. Outputting the Implementation Plan +Due to environment display issues (such as WSL rendering problems that prevent the `implementation_plan.md` artifact from rendering correctly in the UI): +- You must print the full contents of the Implementation Plan directly into the chat/output window in addition to saving it to the artifact file. diff --git a/GLR.md b/GLR.md index e3b57700..5a1ef937 100644 --- a/GLR.md +++ b/GLR.md @@ -1,113 +1,144 @@ # GLR Parsing in RustyLR -RustyLR supports Generalized LR (GLR) parsing, enabling it to handle ambiguous or nondeterministic grammars that traditional LR(1) or LALR(1) parsers cannot process. -When a GLR parser encounters a conflict (such as shift/reduce or reduce/reduce), -it forks the current parsing state into multiple branches, -each representing a different possible interpretation of the input. -These branches are processed in parallel, and invalid paths are pruned as parsing progresses. +RustyLR supports Generalized LR (GLR) parsing, enabling it to process ambiguous or non-deterministic grammars that traditional deterministic LR(1) or LALR(1) parsers cannot handle. + +When a GLR parser encounters a conflict (such as a shift/reduce or reduce/reduce conflict), it does not fail. Instead, it forks the current parsing state into multiple parallel branches. Each branch represents a different possible interpretation of the input. These branches are processed concurrently, and invalid paths are pruned dynamically as parsing progresses. + +--- ## Enabling GLR Parsing -To use GLR parsing in RustyLR, include the `%glr;` directive in your grammar definition. -This directive instructs RustyLR to generate a GLR parser, -which can handle ambiguous grammars by exploring multiple parsing paths. -Once the `%glr` directive is added, any conflicts in the grammar will not be reported as errors. -It's important to be aware of points in your grammar where shift/reduce or reduce/reduce conflicts occur, as each divergence increases computational complexity. +To generate a GLR parser in RustyLR, add the `%glr;` directive to your grammar definition. This directive instructs the parser generator to construct a non-deterministic parsing table and runtime structure capable of maintaining multiple parser stacks. + +Once the `%glr;` directive is active, grammar conflicts will not be reported as compiler errors. However, you should still be mindful of conflict points in your grammar, as excessive non-determinism increases memory usage and parsing complexity at runtime. -**Tip:** If you are using the `rustylr` executable, you can use the `--verbose` option to see any conflicts in the grammar and their divergent paths. +> [!TIP] +> If you are using the `rustylr` CLI, run it with the `--verbose` flag to inspect any grammar conflicts and trace the resulting non-deterministic paths. -## Example: Ambiguous Grammar +--- + +## Example: Ambiguous Expression Grammar + +Below is a simple grammar illustrating shift/reduce conflicts caused by operator ambiguity: ```rust %glr; %tokentype char; %start E; -Digit(char): ['0'-'9'] ; +Digit(char) : ch=['0'-'9'] { ch }; -E(i32): E '+' E { E + E } - | E '*' E { E * E } - | Digit { Digit.to_digit(10).unwrap() as i32 }; +E(i32) + : E '+' e2=E { E + e2 } // Shift/Reduce conflict: + vs + or * + | E '*' e2=E { E * e2 } // Shift/Reduce conflict: * vs + or * + | Digit { Digit.to_digit(10).unwrap() as i32 } + ; ``` -In this grammar, the expression `1 + 2 * 3 + 4` has multiple valid parse trees due to the ambiguity in operator precedence and associativity: - - `((1 + 2) * 3) + 4` - - `(1 + (2 * 3)) + 4` - - `1 + ((2 * 3) + 4)` - - `1 + (2 * (3 + 4))` - - `(1 + 2) * (3 + 4)` +For the input string `1 + 2 * 3 + 4`, this grammar generates multiple valid parse trees due to the absence of operator precedence and associativity: +- `((1 + 2) * 3) + 4` +- `(1 + (2 * 3)) + 4` +- `1 + ((2 * 3) + 4)` +- `1 + (2 * (3 + 4))` +- `(1 + 2) * (3 + 4)` + +A GLR parser will execute all branches concurrently, tracking all possible parse trees in a shared parse forest. -The GLR parser will explore all possible parsing paths to construct the parse forest. +--- -## Resolving Ambiguities -RustyLR allows you to resolve ambiguities dynamically within reduce actions. -Simply returning `Err` from a reduce action will prune the current branch of the parse tree. -By inspecting the lookahead token or other context, you can decide whether to proceed with a particular reduction. +## Resolving Ambiguities Dynamically -For example, to enforce operator precedence (e.g., `*` has higher precedence than `+`), you can modify the reduce actions as follows: +RustyLR allows you to resolve grammar conflicts and prune parsing branches dynamically within your reduce actions. + +### 1. Pruning Paths via `Err` +If a reduce action evaluates to a semantic error and returns `Err`, the parser immediately discards that active parsing branch. By checking the lookahead token or parsing context, you can selectively fail unwanted paths. + +### 2. Disabling Shift Actions (`*shift = false;`) +You can control lookahead behavior directly by modifying the mutable `shift` flag (exposed in reduce actions as `shift: &mut bool`). Setting `*shift = false;` instructs the parser not to perform a shift action on the current lookahead token, effectively forcing a reduction and pruning the shift branch. + +### Example: Enforcing Precedence via Actions +Here is how you can resolve the operator conflicts in the expression grammar dynamically inside the reduce actions: ```rust -E : E '+' E { - match *lookahead.to_term().unwrap() { - '*' => { - // Don't reduce if the next token is '*' - // This prevents: - // E + E / * - // ^ lookahead - // from becoming: E * ... - // ^ (E + E) - return Err("".to_string()); - } - _ => { - // Revoke the shift action - // This prevents: - // E + E / + - // ^ lookahead - // from becoming: E + E + ... - // and enforces only the reduce action: - // E + ... - // ^ (E + E) - *shift = false; - } - } - E + E // Return the result of the addition -} +E(i32) + : E '+' e2=E { + match lookahead.to_term() { + Some('*') => { + // If the next token is '*', return an error to prevent reducing + // this addition first. This prunes the '+' reduction branch and + // allows '*' to shift first (multiplying 2 * 3 before adding 1). + return Err("defer addition for multiplication".to_string()); + } + _ => { + // Otherwise, prevent shifting further tokens and force the reduction + *shift = false; + E + e2 + } + } + } + | E '*' e2=E { + // Enforce left-associative reduction for multiplication + *shift = false; + E * e2 + } + | Digit { Digit.to_digit(10).unwrap() as i32 } + ; ``` ### Predefined Variables in Reduce Actions -- `lookahead: &TerminalSymbol` - refers to the next token in the input stream. either a terminal fed by the user or an special token like `error` -- `shift: &mut bool` - controls whether a shift action should be performed +- **`lookahead`**: A reference of type `&TerminalSymbol` pointing to the next token in the input stream (either a user-supplied terminal or a special symbol like `error`). +- **`shift`**: A mutable reference of type `&mut bool` that controls whether the parser is allowed to shift the next token. -### Ambiguity Resolution Rules -- Returning `Err` from the reduce action will discard the current parsing path -- Setting `*shift = false;` prevents the parser from performing a shift action, enforcing the desired reduction +--- ## Parsing with the GLR Parser -RustyLR provides a consistent parsing interface for both deterministic and GLR parsers. -After generating the parser, you can feed tokens to the parser context and retrieve the parsing results. + +The GLR parser shares a similar interface to the deterministic parser, but instead of producing a single result, its `accept()` method returns an iterator over all successful parse tree results. You can feed tokens using either `feed` (basic) or `feed_location` (location-aware). ```rust -let parser = EParser::new(); // Create Parser instance -let mut context = EContext::new(); // Create Context instance +// Include the generated parser module +mod parser; -for token in input_sequence { - match context.feed(&parser, token) { - Ok(_) => {} - Err(e) => { - println!("Parse error: {}", e); +fn main() { + let parser = parser::EParser::new(); + let mut context = parser::EContext::new(); + let mut userdata = (); // Custom userdata if defined by %userdata + + let input = vec!['1', '+', '2', '*', '3', '+', '4']; + + // Feed tokens to the GLR parser + for token in input { + // basic feeding: + if let Err(e) = context.feed(&parser, token, &mut userdata) { + eprintln!("Fatal parse error: {}", e); return; } + + // or location-aware feeding (if %location is configured in the grammar): + // let span = MySpan { start: ..., end: ... }; + // if let Err(e) = context.feed_location(&parser, token, &mut userdata, span) { + // eprintln!("Fatal parse error: {}", e); + // return; + // } } -} -// Retrieve all possible parse results -for result in context.accept(&parser).unwrap() { - println!("Parse result: {:?}", result); + // Retrieve all valid parse tree results + match context.accept(&parser, &mut userdata) { + Ok(results) => { + for result in results { + println!("Parse tree result: {:?}", result); + } + } + Err(e) => { + eprintln!("Parsing failed: {}", e); + } + } } ``` -### Key Components: -- `EParser::new()` - Creates a new parser instance -- `EContext::new()` - Initializes the parsing context -- `context.feed(&parser, token)` - Feeds tokens to the parser -- `context.accept(&parser)` - Returns all possible values of the `%start` symbol from every parse path \ No newline at end of file +### Key API Components +- **`EParser::new()`**: Creates the static parser table instance. +- **`EContext::new()`**: Initializes a new GLR state context. +- **`context.feed(&parser, token, &mut userdata)`**: Feeds a token into all active parsing stacks. +- **`context.feed_location(&parser, token, &mut userdata, location)`**: Feeds a token with its location span into all active parsing stacks (requires `%location` in the grammar). +- **`context.accept(&parser, &mut userdata)`**: Finalizes parsing (feeding the end-of-file symbol) and returns an iterator over all successful parse results from all active branches. \ No newline at end of file diff --git a/README.md b/README.md index 8ecf2fd9..17e95565 100644 --- a/README.md +++ b/README.md @@ -1,260 +1,294 @@ # rusty_lr + [![crates.io](https://img.shields.io/crates/v/rusty_lr.svg)](https://crates.io/crates/rusty_lr) [![docs.rs](https://docs.rs/rusty_lr/badge.svg)](https://docs.rs/rusty_lr) -***A Bison-like Parser generator & Compiler frontend for Rust generating optimised IELR(1), LALR(1) parser tables, with deterministic LR and non-deterministic LR (GLR) parsing.*** +***A Bison-like parser generator and compiler frontend for Rust. It generates optimized IELR(1) and LALR(1) parsing tables, supporting both deterministic LR and non-deterministic Generalized LR (GLR) parsing.*** + +RustyLR is a robust parser generator that converts context-free grammars into optimized IELR(1) or LALR(1) tables. It seamlessly integrates with the Rust ecosystem, allowing you to write custom reduce actions directly in Rust with rich, diagnostic-driven error reporting. -RustyLR is a parser generator that converts context-free grammars into IELR(1)/LALR(1) tables and supporting deterministic LR and non-deterministic GLR parsing strategies. It supports custom reduce actions in Rust, with beautiful diagnostics. -Highly inspired by tools like *bison*, it uses a similar syntax while integrating seamlessly with Rust's ecosystem. -It constructs optimized state machines, ensuring efficient and reliable parsing. +Highly inspired by classic tools like *Bison* and *Yacc*, RustyLR uses a familiar syntax while offering modern features such as state optimization, location tracking, generalized parsing, and compile-time/runtime conflict resolution. ![title](images/title.png) ## Features - - **Custom Reduce Actions:** Define custom actions in Rust, allowing you to build custom data structures easily. - - **Automatic Optimization:** Reduces parser table size and improves performance by grouping terminals with identical behavior across parser states. - - **Multiple Parsing Strategies:** Supports minimal-LR(1), LALR(1) parser tables, and GLR parsing strategy. - - **Detailed Diagnostics:** Detects grammar conflicts, verbose conflict resolution stages, and optimization stages. - - **Static & Runtime Conflict Resolution:** Provides mechanisms to resolve conflicts at compile time or runtime. - - **Location Tracking:** Tracks the location of every token in the parse tree, useful for error reporting and debugging. - - **State Machine Debugging:** The `rustylr` executable provides a `--state` option that allows you to debug and visualize the generated state machine. This is useful for understanding how the parser will behave and for identifying potential issues in the grammar. -## Quick Start: Using the `rustylr` Executable +- **Custom Reduce Actions:** Define actions directly in Rust to build abstract syntax trees (ASTs) or custom data structures easily. +- **Automatic Parser Optimization:** Shrinks parsing tables and boosts runtime performance by grouping terminal symbols that exhibit identical behavior across parser states. +- **Multiple Parsing Strategies:** Supports minimal-LR(1) (IELR-style), LALR(1) tables, and Generalized LR (GLR) parsing. +- **Detailed Diagnostics:** Reports grammar conflicts, provides verbose traces of conflict resolution stages, and logs optimization passes. +- **Static & Runtime Conflict Resolution:** Resolve grammar conflicts at compile time (precedence/associativity) or dynamically at runtime. +- **Location Tracking:** Automatically tracks positions of tokens and non-terminals, simplifying error reporting in compiler diagnostics. +- **State Machine Debugging:** The `rustylr` CLI provides a `--state` flag to inspect and visualize the generated state machine, making conflict debugging straightforward. -The recommended way to use RustyLR is with the standalone `rustylr` executable. It's faster, provides richer grammar diagnostics, and includes commands for debugging state machines directly. +--- -Here is a step-by-step guide to get you started. +## Quick Start: Using the `rustylr` CLI -**1. Add `rusty_lr` to your dependencies** +The recommended way to use RustyLR is via the standalone `rustylr` CLI executable. It offers faster compilation, comprehensive grammar diagnostics, and interactive tools for debugging state machines. -First, add the `rusty_lr` runtime library to your project's `Cargo.toml`. The generated parser code will depend on it. +### 1. Add `rusty_lr` to your dependencies +Add the runtime library to your `Cargo.toml`. The generated parser will depend on it. ```toml [dependencies] -rusty_lr = "..." # Use the same version as the executable +rusty_lr = "..." # Ensure this matches the version of the CLI executable ``` -**2. Install the `rustylr` executable** - -You can install the executable from crates.io using `cargo`: +### 2. Install the `rustylr` CLI +Install the command-line generator from crates.io using Cargo: ```bash cargo install rustylr ``` -**3. Create a grammar file** - -Create a file named `src/grammar.rs`. This file will contain your token definitions and grammar rules. Any Rust code above the `%%` separator will be copied directly to the generated output file. +### 3. Create a Grammar File +Create a grammar file, e.g., `src/grammar.rs`. Any Rust code placed *above* the `%%` delimiter is copied directly to the generated parser file. The section *below* `%%` defines directives and grammar rules. ```rust // src/grammar.rs -// This code is copied to the generated file. -pub enum MyToken { + +#[derive(Debug, Clone, Copy)] +pub enum Token { Num(i32), Plus, + Minus, + Mul, + Div, + LParen, + RParen, } -%% // Grammar rules start here. - -%tokentype MyToken; -%start E; -%left plus; // Specify left-associativity for the 'plus' token. - -// Define tokens and how they map to MyToken variants. -%token num MyToken::Num(_); -%token plus MyToken::Plus; - -// Define grammar rules and their return types. -// E(i32) means the non-terminal E returns an i32. -// In the action blocks `{ ... }`, you can refer to the values of symbols -// on the right-hand side by their names (e.g., `e1`, `e2`, `num`). -E(i32): e1=E plus e2=E { e1 + e2 } - | num { let MyToken::Num(num) = num else { unreachable!(); }; - num - } - ; +%% + +// Define the token type and the start symbol +%tokentype Token; +%start Expr; + +// Declare operator precedence and associativity (lowest to highest) +%left plus minus; +%left mul div; + +// Map grammar terminals to Token enum variants +%token num Token::Num(_); +%token plus Token::Plus; +%token minus Token::Minus; +%token mul Token::Mul; +%token div Token::Div; +%token lparen Token::LParen; +%token rparen Token::RParen; + +// Production rules +// Expr(i32) means the non-terminal Expr returns an i32. +// In the action block `{ ... }`, reference RHS symbols by their names. +Expr(i32) + : e1=Expr plus e2=Expr { e1 + e2 } + | e1=Expr minus e2=Expr { e1 - e2 } + | e1=Expr mul e2=Expr { e1 * e2 } + | e1=Expr div e2=Expr { e1 / e2 } + | lparen e=Expr rparen { e } + | num { + if let Token::Num(val) = num { + val + } else { + unreachable!() + } + } + ; ``` -**4. Generate the parser code** - -Run the `rustylr` executable to process your grammar file. This command will generate `src/parser.rs` from `src/grammar.rs`. +### 4. Generate the Parser Code +Run the CLI to compile your grammar into a Rust module: ```bash rustylr src/grammar.rs src/parser.rs ``` -**5. Use the generated parser in your code** - -Finally, include the newly generated `src/parser.rs` as a module in your `main.rs` or `lib.rs` and use it to parse a token stream. +### 5. Parse a Token Stream +Include the generated `src/parser.rs` file in your project and feed it a stream of tokens: ```rust -// In src/main.rs - -// Include the generated parser module. +// src/main.rs mod parser; -// Bring the token enum into scope. -use parser::MyToken; +use parser::Token; fn main() { - // Example token stream for "1 + 2" - let tokens = vec![MyToken::Num(1), MyToken::Plus, MyToken::Num(2)]; - - let parser = parser::EParser::new(); // Assumes 'E' is your start symbol - let mut context = parser::EContext::new(); - let mut userdata = (); // No userdata in this example. + // Represents the expression: 3 + 4 * 2 + let tokens = vec![ + Token::Num(3), + Token::Plus, + Token::Num(4), + Token::Mul, + Token::Num(2), + ]; + + let parser = parser::ExprParser::new(); + let mut context = parser::ExprContext::new(); + let mut userdata = (); // No custom user data needed for token in tokens { - match context.feed(&parser, token, &mut userdata) { - Ok(_) => {} - Err(e) => { - eprintln!("Parse error: {}", e); - return; - } + if let Err(err) = context.feed(&parser, token, &mut userdata) { + eprintln!("Parse error: {}", err); + return; } } - // Get the final parsed result. - match context.accept(&parser) { + match context.accept(&parser, &mut userdata) { Ok(result) => { - let final_result: i32 = result; - println!("Parsed result: {}", final_result); // Should print "3" - }, - Err(e) => { - eprintln!("Failed to produce a final result: {}", e); + println!("Parsed result: {}", result); // Output: 11 + } + Err(err) => { + eprintln!("Failed to finalize parsing: {}", err); } } } ``` -**Important:** Ensure the version of the `rustylr` executable you run matches the version of the `rusty_lr` crate in your `Cargo.toml`. Mismatched versions can lead to build errors. - +> [!IMPORTANT] +> The version of the `rustylr` CLI executable must exactly match the version of the `rusty_lr` library in your `Cargo.toml`. Version mismatches may result in build failures. +--- ## Generated Code Structure -The generated code will include several structs and enums: - - `Parser`: A lightweight struct that references the static parser table. [(LR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/lr/trait.Parser.html) [(GLR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/glr/trait.Parser.html) - - `Context`: A struct that maintains the current parsing state and symbol values. [(LR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/lr/struct.Context.html) [(GLR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/glr/struct.Context.html) - - `State`: A type representing a parser state and its associated table. - - `Rule`: A type representing a production rule. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/struct.ProductionRule.html) - - `NonTerminals`: An enum representing all non-terminal symbols in the grammar. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/trait.NonTerminal.html) +The generated parser module contains several generated components tailored to your start symbol: +- **`Parser`**: A lightweight struct containing the static parsing tables. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/trait.Parser.html) +- **`Context`**: A mutable state context that keeps track of the stack and parsed symbol values. [(LR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/deterministic/struct.Context.html) [(GLR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/nondeterministic/struct.Context.html) +- **`State`**: An internal type representing individual states in the parser. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/state/trait.State.html) +- **`Rule`**: An internal enum representing production rules. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/rule/struct.ProductionRule.html) +- **`NonTerminals`**: An enum representing all non-terminal symbols. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/nonterminal/trait.NonTerminal.html) - -### Working with Context -You can also get contextual information from the `Context` struct: +### Interacting with the Parsing Context +The `Context` offers helpful utilities for inspecting and tracing: ```rust -let mut context = Context::new(); +let mut context = ExprContext::new(); -// ... parsing ... +// ... feed tokens ... -context.expected_token(); // Get expected (terminal, non-terminal) symbols for current state -context.can_feed(&term); // Check if a terminal symbol can be fed -context.trace(); // Get all `%trace` non-terminals currently being parsed -println!("{}", context.backtrace()); // Print backtrace of the parser state -println!("{}", context); // Print tree structure of the parser state (`tree` feature) +context.expected_token(&parser); // Returns the expected symbols for the current state +context.can_feed(&parser, &token); // Checks if a terminal can be fed next +context.trace(&parser); // Retrieves active `%trace` non-terminals +println!("{}", context.backtrace(&parser)); // Prints the stack trace of parser states +println!("{}", context); // Formats the state tree (requires 'tree' feature) ``` -### The Feed Method -The generated code includes a `feed` method that processes tokens: - +### Feeding Tokens +You can feed terminal symbols either with or without location information: ```rust -context.feed(&parser, term, &mut userdata); // Feed a terminal symbol and update the state machine -context.feed_location(&parser, term, &mut userdata, term_location); // Feed a terminal symbol with location tracking -``` +// Basic feeding +context.feed(&parser, token, &mut userdata); -This method returns `Ok(())` if the token was successfully parsed, or an `Err` if there was an error. +// Location-aware feeding (requires %location in grammar) +context.feed_location(&parser, token, &mut userdata, token_location); +``` -**Note:** The actual method signatures differ slightly when building a GLR parser. +--- ## GLR Parsing -RustyLR offers built-in support for Generalized LR (GLR) parsing, enabling it to handle ambiguous or nondeterministic grammars that traditional LR(1) or LALR(1) parsers cannot process. -See [GLR.md](GLR.md) for details. + +RustyLR provides native support for Generalized LR (GLR) parsing. When you add the `%glr;` directive to a grammar, RustyLR generates a non-deterministic parser that forks state branches upon encountering shift/reduce or reduce/reduce conflicts. This is particularly useful for ambiguous grammars or complex programming languages. + +For more details, see [GLR.md](GLR.md). + +--- ## Error Handling and Conflict Resolution -RustyLR provides multiple mechanisms for handling semantic errors and resolving conflicts during parsing: - - **Panic Mode Error Recovery:** Use the `error` token for panic-mode error recovery - - **Operator Precedence:** Set precedence with `%left`, `%right`, `%precedence` for terminals - - **Reduce Rule Priority:** Set priority with `%dprec` for production rules - - **Runtime Errors:** Return `Err` from reduce actions to handle semantic errors -See [SYNTAX.md - Resolving Conflicts](SYNTAX.md#resolving-conflicts) for detailed information. +RustyLR provides multiple tools to resolve grammar ambiguities and handle parsing failures: +- **Panic-Mode Error Recovery:** Use the special `error` terminal to catch and recover from syntax errors. +- **Operator Precedence:** Disambiguate expressions with `%left`, `%right`, and `%precedence` directives. +- **Reduce Rule Priority:** Explicitly set reduce priorities with `%dprec`. +- **Runtime Error Propagation:** Return custom `Err` payloads from reduce actions to signal semantic or parsing errors. + +See [SYNTAX.md - Resolving Conflicts](SYNTAX.md#resolving-conflicts) for in-depth information. + +--- ## Location Tracking -Track the location of tokens and non-terminals for better error reporting and debugging: + +Track input spans automatically across tokens and non-terminals to print helpful compiler errors: ```rust -Expr: exp1=Expr '+' exp2=Expr { - println!("Location of exp1: {:?}", @exp1); - println!("Location of exp2: {:?}", @exp2); - println!("Location of this expression: {:?}", @$); // @$ (or @0) is the location of the non-terminal itself - exp1 + exp2 -} -| Expr error Expr { - println!("Error at: {:?}", @error); // @error is the location of the error token - 0 // Return a default value -} +Expr(i32) + : e1=Expr '+' e2=Expr { + println!("Span of e1: {:?}", @e1); + println!("Span of e2: {:?}", @e2); + println!("Span of this Expr: {:?}", @$); // @$ (or @0) represents the current non-terminal's span + e1 + e2 + } + | Expr error Expr { + println!("Syntax error recovery span: {:?}", @error); + 0 // Fallback value + } + ; ``` -See [SYNTAX.md - Location Tracking](SYNTAX.md#location-tracking) for detailed information. +See [SYNTAX.md - Location Tracking](SYNTAX.md#location-tracking) for configuration details. + +--- ## State Machine Debugging -The `rustylr` executable includes a powerful `--state` option for debugging the generated parser's state machine. This feature allows you to inspect the details of each state, including its production rules, expected tokens, and transitions to other states. It is an invaluable tool for diagnosing grammar ambiguities, understanding shift/reduce conflicts, and verifying that the parser behaves as expected. -To use it, run `rustylr` with the `--state` flag, followed by your grammar file: +You can inspect the generated parser states using the `--state` option. This outputs a color-coded state listing showing core items, lookahead sets, transitions, and conflict reports. ```bash rustylr --state src/grammar.rs ``` -This will output a detailed, color-coded representation of the state machine directly in your terminal, making it easy to trace the parser's logic. - ![State Machine Debug](images/state_option.png) -This visualization helps you understand the parsing process step-by-step and is particularly useful for debugging complex grammars. +--- ## Examples - - [Calculator (enum version)](https://github.com/ehwan/RustyLR/blob/main/example/calculator/src/parser.rs): A numeric expression parser using custom token enums - - [Calculator (u8 version)](https://github.com/ehwan/RustyLR/blob/main/example/calculator_u8/src/parser.rs): A numeric expression parser using byte tokens - - [JSON Validator](https://github.com/ehwan/RustyLR/blob/main/example/json/src/parser.rs): A JSON syntax validator - - [Lua 5.4 syntax parser](https://github.com/ehwan/lua_rust/blob/main/parser/src/parser.rs): A complete Lua language parser - - [C language parser](https://github.com/ehwan/C-language-Parser-In-Rust/blob/main/src/ast/parser_lr.rs): A C language parser - - [Bootstrap parser](https://github.com/ehwan/RustyLR/blob/main/rusty_lr_parser/src/parser/parser.rs): RustyLR's own syntax parser is written in RustyLR itself + +- [Calculator (enum tokens)](https://github.com/ehwan/RustyLR/blob/main/example/calculator/src/parser.rs): A numeric expression parser using custom token enums. +- [Calculator (u8 tokens)](https://github.com/ehwan/RustyLR/blob/main/example/calculator_u8/src/parser.rs): A byte-stream numeric calculator. +- [JSON Validator](https://github.com/ehwan/RustyLR/blob/main/example/json/src/parser.rs): A validator checking JSON syntax. +- [Lua 5.4 Parser](https://github.com/ehwan/lua_rust/blob/main/parser/src/parser.rs): A complete parser for the Lua 5.4 programming language. +- [C Parser](https://github.com/ehwan/C-language-Parser-In-Rust/blob/main/src/ast/parser_lr.rs): An LR-based parser for the C programming language. +- [Bootstrap Parser](https://github.com/ehwan/RustyLR/blob/main/rusty_lr_parser/src/parser/parser.rs): RustyLR's own grammar parser, written using RustyLR. + +--- ## Cargo Features - - `build`: Enables build script tools for generating parsers at compile time. - - `tree`: Enables automatic syntax tree construction for debugging purposes. Makes `Context` implement `Display` for pretty-printing. + +- **`build`**: Enables helper functions in `rusty_lr_buildscript` for compiling grammars inside `build.rs` scripts. +- **`tree`**: Enables automatic syntax tree rendering for debugging. Implementing `Display` for `Context` outputs a formatted parse tree. + +--- ## Grammar Syntax -RustyLR's grammar syntax is inspired by traditional Yacc/Bison formats. -See [SYNTAX.md](SYNTAX.md) for detailed grammar definition syntax. + +RustyLR's syntax builds upon standard Yacc/Bison design but is optimized for Rust. + +See [SYNTAX.md](SYNTAX.md) for the complete reference. ### Type Inference with `_` -Assigning a type to a non-terminal can be done automatically using the `_` placeholder. +You can omit explicit rule types using the `_` placeholder. RustyLR will infer the type based on identity rules and reduce actions: ```rust -E(_): A; +Expr(_): Term; ``` -When `_` is used, RustyLR will infer the type by examining `Identity` reduce actions. If a circular dependency prevents type resolution, a compilation error is raised. + +If a circular dependency prevents inference, RustyLR will report a compilation error. + +--- ## Contributing -Contributions are welcome! Please feel free to open an issue or submit a pull request. + +We welcome issues and pull requests! ### Project Structure -This project is organized as a Cargo workspace with the following crates: - - - **`rusty_lr/`**: The main end-user library that provides the public API. This is what users add to their `Cargo.toml`. - - **`rusty_lr_core/`**: Core parsing engine containing the fundamental data structures, algorithms, and runtime components for both deterministic (`src/parser/deterministic`) and non-deterministic (`src/parser/nondeterministic`) parsing. - - **`rusty_lr_parser/`**: The main code generation engine that parses RustyLR's grammar syntax, builds parser tables, and generates the actual parser code. This is the core of the parser generation process. - - **`rusty_lr_derive/`**: Procedural macro interface that wraps `rusty_lr_parser` to provide the `lr1!` macro for inline grammar definitions. - - **`rusty_lr_buildscript/`**: Build script interface that wraps `rusty_lr_parser` for generating parser code at compile time when using the `build` feature. - - **`rusty_lr_executable/`**: Standalone `rustylr` executable for command-line parser generation. - - **`scripts/`**: Development and testing scripts - -The crates have the following dependency relationships: -- `rusty_lr` depends on `rusty_lr_core`, `rusty_lr_derive`, and `rusty_lr_buildscript` (optional) -- `rusty_lr_derive` and `rusty_lr_buildscript` depend on `rusty_lr_parser` -- `rusty_lr_parser` depends on `rusty_lr_core` -- `rusty_lr_executable` depends on `rusty_lr_buildscript` + +This repository is organized as a Cargo workspace: + +- **`rusty_lr/`**: The main user-facing library. Add this to your `Cargo.toml`. +- **`rusty_lr_core/`**: The runtime engine, defining stack logic, deterministic parsing (`src/parser/deterministic`), and GLR parsing (`src/parser/nondeterministic`). +- **`rusty_lr_parser/`**: The grammar compilation engine. Parses RustyLR files, constructs parsing tables, and generates Rust output. +- **`rusty_lr_derive/`**: Procedural macro wrapper around `rusty_lr_parser`, providing the `lr1!` macro. +- **`rusty_lr_buildscript/`**: Helper API for running RustyLR in cargo build scripts. +- **`rusty_lr_executable/`**: The standalone `rustylr` CLI executable. +- **`scripts/`**: Automation, regression test suites, and helper scripts. ```mermaid graph TD; @@ -282,17 +316,17 @@ graph TD; rusty_lr_parser --> rusty_lr_core; ``` +### Versioning Policy +RustyLR separates its components into two parts: +1. The compiler CLI (`rustylr`) +2. The runtime library (`rusty_lr`) -### About the Versioning -RustyLR consists of two big parts: - - executable (`rustylr`), the code generator - - runtime (`rusty_lr`), the main library +To maintain Cargo compatibility, patch versions are incremented when changes are backwards-compatible (meaning previously generated parser files compile without errors with the new library version). If a change to the code generator requires updates to the runtime library API that break older generated code, a minor version bump is performed. -Since the `cargo` automatically uses the latest patch in `major.minor.patch` version of a crate, we increase the patch number only if the generated code is compatible with the runtime. That is, for any user who is not using buildscript or proc-macro, and using the executable-generated code itself, -any code change that could make compile errors with the previous generated code will result in a minor version bump. +--- ## License -This project is dual-licensed under either of the following licenses, at your option: - - MIT License ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT) - - Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0) +Dual-licensed under either: +- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0) +- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT) diff --git a/SYNTAX.md b/SYNTAX.md index 2ad99e55..9d88e67f 100644 --- a/SYNTAX.md +++ b/SYNTAX.md @@ -1,567 +1,549 @@ # Syntax + +This document provides a comprehensive guide to the grammar definition syntax used by RustyLR. The syntax is heavily inspired by parser generators like *Yacc* and *Bison*, but tailored to integrate seamlessly with the Rust programming language. + +--- + ## Quick Reference - - [Token type - `%tokentype`](#token-type-must-defined) - - [Defining tokens - `%token`](#token-definition-must-defined) - - [`%filter`](#filter-directive) - - [Production rules](#production-rules) - - [Regex patterns](#regex-pattern) - - [RuleType](#ruletype-optional) - - [ReduceAction](#reduceaction-optional) - - [Accessing token data in ReduceAction](#accessing-token-data-in-reduceaction) - - [Exclamation mark `!`](#exclamation-mark-) - - [Traceable Non-Terminals - `%trace`](#tracing-non-terminals) - - [Start symbol - `%start`](#start-symbol-must-defined) - - [User data type - `%userdata`](#userdata-type-optional) - - [Resolving Conflicts](#resolving-conflicts) - - [Panic Mode Error Recovery - `error`](#panic-mode-error-recovery) - - [Shift/Reduce conflicts - `%left`, `%right`, `%precedence`, `%prec`](#operator-precedence) - - [Reduce/Reduce conflicts - `%dprec`](#rule-priority) - - [Error variants - `%err`, `%error`](#error-type-optional) - - [GLR parser - `%glr`](#glr-parser-generation) - - [LALR parser - `%lalr`](#lalr-parser-generation) - - [Disable Optimization - `%nooptim`](#no-optimization) - - [Make dense parser table - `%dense`](#dense-parser-table) - - [Location tracking - `%location`](#location-tracking) +- [Token Type (`%tokentype`)](#token-type-must-defined) +- [Token Definition (`%token`)](#token-definition-must-defined) +- [Filter Directive (`%filter`)](#filter-directive) +- [Production Rules](#production-rules) +- [Patterns](#patterns) +- [RuleType (Non-Terminal Types)](#ruletype-optional) +- [Reduce Actions](#reduceaction-optional) +- [Accessing Data in Reduce Actions](#accessing-token-data-in-reduceaction) +- [Exclamation Mark (`!`) Value Discard](#exclamation-mark-) +- [Tracing Non-Terminals (`%trace`)](#tracing-non-terminals) +- [Start Symbol (`%start`)](#start-symbol-must-defined) +- [Userdata Type (`%userdata`)](#userdata-type-optional) +- [Conflict Resolution](#resolving-conflicts) + - [Panic-Mode Error Recovery (`error`)](#panic-mode-error-recovery) + - [Operator Precedence (`%left`, `%right`, `%precedence`, `%prec`)](#operator-precedence) + - [Rule Priority (`%dprec`)](#rule-priority) +- [Error Type (`%err` / `%error`)](#error-type-optional) +- [LALR(1) Parser Generation (`%lalr`)](#lalr-parser-generation) +- [GLR Parser Generation (`%glr`)](#glr-parser-generation) +- [Disabling Table Optimization (`%nooptim`)](#no-optimization) +- [Dense Parser Tables (`%dense`)](#dense-parser-table) +- [Location Tracking (`%location`)](#location-tracking) + +--- ## Overview -RustyLR's grammar syntax is inspired by parser generators like Yacc and Bison. -Grammars are defined using a combination of directives, token definitions, and production rules. -In procedural macros, the grammar is defined using the `lr1!` macro. -In build script files, the grammar section is separated from Rust code using `%%`. Everything before `%%` is treated as regular Rust code and is copied as-is to the generated output. +RustyLR grammars can be defined in two ways: +1. **Procedural Macros:** Using the `lr1!` macro inline in your Rust code. +2. **Build Scripts:** Using a standalone grammar file (e.g., `src/grammar.rs`) processed by the `rustylr` command-line tool. stand-alone files use the `%%` delimiter to separate Rust helper code (imports, custom enums) from the grammar definition. Everything preceding `%%` is copied as-is into the generated parser file. + +--- + +## Token Type (Must Defined) -## Token type (must defined) ``` %tokentype ; ``` -Define the type of terminal symbols. -`` must be accessible at the point where the macro is called. +Defines the Rust type representing the input terminal symbols (tokens). The `` must be in scope at the place where the parser is generated. + +### Example ```rust -enum MyTokenType { - Digit, - Ident, - ... - VariantWithGeneric +#[derive(Debug, Clone)] +pub enum Token { + Ident(String), + Num(i32), + Plus, + Minus, } -lr! { -... -%tokentype MyTokenType; -} +// In the grammar section: +%tokentype Token; ``` +--- + +## Token Definition (Must Defined) -## Token definition (must defined) ``` %token name ; ``` -Defines the terminal symbol `name` for further use in the grammar, -and `` will be used in the terminal classification `match` statement: + +Binds a grammar terminal symbol (`name`) to a pattern (``) used to classify tokens. Under the hood, RustyLR generates a `match` statement to identify the token: + ```rust -match terminal_symbol { - => { classification } +match terminal_token { + => { /* Classified as `name` */ }, ... } ``` -**Example:** +### Example ```rust -%tokentype MyToken; +%tokentype Token; -%token num MyToken::Num(_); -%token plus MyToken::Punct('+'); -%token minus MyToken::Punct('-'); -... +%token id Token::Ident(_); // Matches Token::Ident("foo") +%token num Token::Num(_); // Matches Token::Num(42) +%token plus Token::Plus; // Matches Token::Plus +%token minus Token::Minus; // Matches Token::Minus -E: num plus num - | num minus num - ; +Expr + : id plus num + | num minus id + ; ``` -**Notes:** -- If `%tokentype` is either `char` or `u8`, you can't use this directive. You must use literal values in the grammar directly. -- This directive is not for defining the *complete token space*. Any token not defined here can also be captured by `[^ term1 ...]`-like negation patterns. +### Notes +- **Literal Tokens:** If `%tokentype` is set to `char` or `u8`, you do not need to define tokens using `%token`. Instead, use character literals (`'+'`, `'a'`) or byte literals (`b'+'`, `b'a'`) directly in the grammar. +- **Incomplete Token Space:** You do not need to exhaustively define every single possible enum variant. Any tokens not explicitly matched can still be captured using negation sets (e.g., `[^ plus minus]`). +--- -## `%filter` directive -For `%tokentype` that cannot be used in `match` statement directly, -you can define a filter function using the `%filter` directive. -```rust -%filter ::my::filter_fn ; +## %filter Directive + +``` +%filter ; ``` -Now the `match` statement will be generated as follows: + +When your `%tokentype` cannot be matched directly in a simple Rust `match` pattern (e.g., if it contains generic parameters, references, or requires complex validation), you can define a custom filter function. + +When specified, the generated classification `match` will wrap the input terminal in the filter function: + ```rust -match ::my::filter_fn(terminal_symbol) { - => { classification } +match filter_fn(terminal_token) { + => { /* ... */ } ... } ``` -The signature of the filter function must be `fn (&Terminal) -> MatchType`. +The filter function signature must be: `fn(&Terminal) -> MatchType` or `fn(Terminal) -> MatchType` depending on your wrapper design. + +### Example +```rust +fn my_filter(token: &Token) -> TokenKind { + match token { + Token::Ident(_) => TokenKind::Ident, + Token::Num(_) => TokenKind::Num, + Token::Plus => TokenKind::Plus, + } +} + +// In the grammar: +%filter my_filter; +``` +--- ## Production Rules -Each production rule defines how a non-terminal symbol can be derived from a sequence of patterns. + +Production rules define how non-terminal symbols are constructed from sequences of other symbols or pattern groups. + ``` NonTerminalName : Pattern1 Pattern2 ... PatternN %prec OpName { ReduceAction } | Pattern1 Pattern2 ... PatternN { ReduceAction } - ... + ... ; ``` -**Components:** - - **NonTerminalName:** The name of the non-terminal symbol being defined - - **PatternX:** A terminal or non-terminal symbol, or a pattern as defined below - - **ReduceAction:** Optional Rust code executed when the rule is reduced - - **OpName:** Use this symbol as an operator for this production rule. `OpName` can be defined with `%token` or literal, or any unique identifier just for this rule. See [RuleType](#ruletype-optional) for more details. -## Patterns -Patterns define the structure of the input that matches a production rule. +### Components +- **`NonTerminalName`**: The name of the non-terminal being defined. +- **`PatternX`**: A symbol (terminal or non-terminal) or a regex-like pattern (see [Patterns](#patterns)). +- **`ReduceAction`**: An optional block of Rust code executed when the production rule is reduced. +- **`%prec OpName`**: Explicitly assigns the operator precedence of the terminal `OpName` to this rule (see [Operator Precedence](#operator-precedence)). - - `.` : Any single terminal symbol - - `name` : Non-terminal or terminal symbol `name` defined in the grammar - - `[term1 term_start-term_last]`, `[^term1 term_start-term_last]` : Set of terminal symbols. - - `P*` : Zero or more repetitions of `P` - - `P+` : One or more repetitions of `P` - - `P?` : Zero or one repetition of `P` - - `$sep( P, P_separator, repetition )`: A repetition of `P` separated by `P_separator`. The `repetition` can be `*`, or `+` to indicate zero or more, or one or more repetitions respectively - - `(P1 P2 P3 | P4 | P5 P6 ...)` : Grouping of patterns - - `P / term` or `P / [term1 term_start-term_last]`: Pattern `P` followed by lookaheads. Lookaheads will not be consumed - - `'a'` or `b'a'`: Single character literal or byte literal. This is only supported if the `%tokentype` is `char` or `u8` - - `"abcd"` or `b"abcd"`: String literal or byte string literal. This is only supported if the `%tokentype` is `char` or `u8` - - `P - TerminalSet`: `P` must be a subset of terminal symbols. This pattern matches `P` but not any of the terminal symbols in `TerminalSet` +--- -**Important Note about Range Patterns:** -When using range patterns like `[first-last]`, the range is determined by the order of `%token` directives, not by the actual values of the tokens. +## Patterns -**Example:** -If you define tokens in the following order: -``` -%token one '1'; -%token two '2'; -... -%token zero '0'; -%token nine '9'; +RustyLR supports rich regular expression patterns on the right-hand side of production rules: + +- **`.`** : Matches any single terminal symbol. +- **`name`** : Matches the terminal or non-terminal symbol `name`. +- **`[term1 term_start-term_last]`** : Matches any terminal symbol in the specified set. +- **`[^term1 term_start-term_last]`** : Negated set. Matches any terminal symbol *not* in the specified set. +- **`P*`** : Matches zero or more repetitions of pattern `P` (binds as a `Vec

`). +- **`P+`** : Matches one or more repetitions of pattern `P` (binds as a `Vec

`). +- **`P?`** : Matches zero or one occurrence of pattern `P` (binds as an `Option

`). +- **`$sep(P, P_separator, repetition)`** : Matches repetitions of `P` separated by `P_separator`. The `repetition` argument can be `*` (zero or more) or `+` (one or more). Binds as a `Vec

`. +- **`(P1 P2 | P3)`** : Grouping and alternation. +- **`P / term`** or **`P / [term1 term2]`** : Lookahead assertion. Matches `P` only if followed by the lookahead symbol(s), without consuming them. +- **`'a'` / `b'a'`** : Character/byte literals (only valid if `%tokentype` is `char` or `u8`). +- **`"abcd"` / `b"abcd"`** : String/byte string literals (only valid if `%tokentype` is `char` or `u8`). +- **`P - TerminalSet`** : Matches pattern `P` but excludes any terminal in the `TerminalSet`. + +### Range Patterns (`[first-last]`) +When defining ranges of custom tokens (e.g. `[zero-nine]`), the range ordering is determined by the **declaration order of your `%token` directives**, rather than the inherent values of the enum elements. + +#### Example +If tokens are declared as: +```rust +%token zero Token::Num(0); +%token one Token::Num(1); +%token two Token::Num(2); +// ... +%token nine Token::Num(9); ``` -The range `[zero-nine]` will be `['0', '9']`, not `['0'-'9']`. +The range `[zero-two]` matches `zero`, `one`, and `two`. -If you are using `char` or `u8` as `%tokentype`, you can use the range pattern like this: -``` -['0'-'9'] -``` -This exactly matches the range of characters from '0' to '9'. +If the `%tokentype` is `char` or `u8`, literal character ranges like `['0'-'9']` or `[b'0'-b'9']` are resolved using standard ASCII values. +--- +## RuleType (Optional) + +You can assign a semantic return type to any non-terminal symbol. -## RuleType (optional) -Assigning a type to a non-terminal allows the parser to carry semantic information. ``` -E(MyType): ... ; +NonTerminal(RustType) : ... ; ``` -- `MyType`: The Rust type associated with the non-terminal `E` -### Type Inference with Placeholder `_` -You can use `_` as a placeholder to let `rusty_lr` automatically infer the type of a non-terminal. +- **`RustType`**: The Rust type that this non-terminal evaluates to when reduced. + +### Type Inference with `_` +If you do not want to specify types manually, you can use the `_` placeholder. RustyLR will inspect the rule's reduce actions and identity transitions to infer the correct type automatically: + ```rust -E(_): A; +Expr(_): Term; ``` -If `_` is used, the type will be inferred by analyzing the rules (specifically focusing on `Identity` actions/rules that map directly to another typed token or non-terminal). If a circular dependency prevents inference, the parser will return a compilation error. -The actual value of `E` is evaluated by the result of the ReduceAction. +> [!WARNING] +> If a circular dependency exists (e.g., two rules trying to infer their types from one another without a base type), RustyLR will fail with a compilation error. ### Memory Optimization with `Box` -Internally, the generated parser stores all semantic values—including the `%tokentype` (terminal token type) and all non-terminals' `RuleType`s—within a single unified `enum` that represents the data stack. +Internally, the generated parser stores all semantic values (the `%tokentype` and all non-terminals' `RuleType`s) in a single unified `enum` representing the parser's data stack. -Because the size of a Rust enum is determined by its largest variant, if even a single `RuleType` (or the token type itself) has a large memory footprint, it will inflate the size of the entire enum. Consequently, the data stack will consume significantly more memory, as every element pushed onto the stack will allocate space corresponding to that largest variant. +Because the memory footprint of a Rust `enum` is dictated by its largest variant, if even one `RuleType` is exceptionally large (e.g., a large AST struct), the size of *every* stack slot will inflate. This can result in significant memory waste and performance degradation. -To avoid this overhead, it is highly recommended to wrap large data types in a `Box` (e.g., `Box`). This keeps the variant size minimal (the size of a single pointer) and optimizes the memory consumption of the data stack as a whole. +To avoid this, wrap large AST nodes or structures in a `Box` (e.g., `Box`). This ensures the enum variant only takes up the size of a single pointer, optimizing stack memory usage. +--- -## ReduceAction (optional) -A ReduceAction is Rust code executed when a production rule is reduced. +## ReduceAction (Optional) -**Rules:** -- If a `RuleType` is defined, the ReduceAction must evaluate to that type -- If no `RuleType` is defined and only one token holds a value, the ReduceAction can be omitted -- Reduce actions can return `Result<(), ErrorType>` to handle errors during parsing -- Reduce actions can be written in Rust code. They are executed when the rule is matched and reduced +A reduce action is Rust code executed when a rule is matched and reduced. -**Example:** -```rust -%err String; +### General Rules +- If the non-terminal has a `RuleType`, the reduce action block must evaluate to that type. +- If no `RuleType` is defined and only one symbol in the rule has a value, the reduce action can be omitted (it automatically forwards that value). +- Actions can return a `Result` to propagate runtime parser errors. -E(i32): A div a2=A { - if a2 == 0 { - return Err("Division by zero".to_string()); - } - A / a2 // new value of E -}; +### Named Variables +Assign names to elements on the right-hand side using the `=` syntax to access their values inside the action block: + +```rust +Expr(i32) + : left=Expr '+' right=Term { left + right } + ; ``` -You can reference any data with below patterns: - - `data: &mut %UserData`: the user data passed to `feed()` function - - `var_name: %RuleType`: token data associated with `var_name` - - `@var_name: %LocationType`: location data associated with `var_name` - - `@$: &mut %LocationType`: location data of current non-terminal in this reduce action - - `$1`, `$2`, ...: index-based token data ($1 is the leftmost symbol, $2 is the second, etc.) - - `@1`, `@2`, ...: index-based location data (@1 is the location of the leftmost symbol, @2 is the second, etc.) - - `@0`: location data of current non-terminal in this reduce action (equivalent to `@$`) - - `lookahead: &TerminalSymbol<%TokenType>`: lookahead token that caused this reduce action - - `shift: &mut bool`: for non-deterministic GLR parser, set this value to `false` to revoke the shift action +--- ## Accessing Token Data in ReduceAction -Within a ReduceAction, you can access the data associated with tokens and non-terminals: -**Named Patterns:** Assign names to patterns to access their values. +Within a `ReduceAction` block, you can access values and metadata from the matched symbols: + +### 1. Named Bindings +Explicitly bind a pattern to a variable name: ```rust -E(i32): left=A '+' right=A { left + right }; +Expr(i32) : left=Expr '+' right=Term { left + right }; ``` -**Default Names:** Use default names when obvious. +### 2. Default Bindings +If a symbol name is a valid identifier, you can reference it directly without an explicit binding: ```rust -E(i32): A '+' right=A { A + right }; // use A directly +Expr(i32) : Expr '+' Term { Expr + Term }; ``` -This is also possible for advanced patterns: + +For regex repetitions (e.g., `Term*`), referencing the base name `Term` returns a `Vec`: ```rust -E(i32): A* { A.iter().sum() }; // sum all values in A +Sum(i32) : Term* { Term.iter().sum() }; ``` -Here, `A` is a `Vec` and you can access its values directly. -**Bison-style Positional Variables:** You can reference values and locations by their indices in the rule. - - `$1`, `$2`, ... represent the semantic values of RHS symbols. - - `@1`, `@2`, ... represent the location of RHS symbols. - - `@0` represents the location of LHS (equivalent to `@$`). +### 3. Bison-style Positional Variables +You can access values and locations using index numbers corresponding to the position of the symbol on the RHS (1-indexed): +- **`$1`, `$2`, ...**: The semantic value of the 1st, 2nd, etc. symbol. +- **`@1`, `@2`, ...**: The location/span of the 1st, 2nd, etc. symbol. +- **`@$` (or `@0`)**: The location of the entire reduced non-terminal. ```rust -E(i32): A '+' A { $1 + $3 }; // sum first and third token +Expr(i32) : Expr '+' Expr { $1 + $3 }; ``` - -**User Data:** Access mutable user-defined data passed to the parser. +### 4. User Data (`data`) +Access mutable user data passed to the `feed()` function. It is exposed in reduce actions as `data` (of type `&mut UserData`): ```rust -E(i32): A '+' right=A { - *data += 1; // data: &mut UserData - A + right +Expr(i32) : Term { + *data += 1; // Increment a parse counter + Term }; ``` -Here, `data` is `&mut UserData`, which is defined by the `%userdata` directive. -**Lookahead Token:** Inspect the next token without consuming it. +### 5. Lookahead Token (`lookahead`) +You can inspect the lookahead token that triggered the current reduction. It is exposed as `lookahead` (of type `&TerminalSymbol`): ```rust -match *lookahead.to_term().unwrap() { // lookahead: &TerminalType - '+' => { /* ... */ }, - _ => { /* ... */ }, -} +Expr(i32) : Term { + if let Some(next_token) = lookahead.to_term() { + println!("Lookahead is: {:?}", next_token); + } + Term +}; ``` -Here, `lookahead` is a `&TerminalSymbol<%TokenType>`, it is either a terminal symbol fed by the user or a special token like `error`. -**Shift Control:** Control whether to perform a shift operation (for GLR parser). +### 6. Shift Control (`shift`) +For GLR parsing, you can dynamically control whether the parser should perform a shift action or prune the branch by modifying the mutable `shift` boolean: ```rust -*shift = false; // Prevent shift action +*shift = false; // Disable shifting the next token, forcing reduction ``` -### Variable Types for Regex Patterns -For some regex patterns, the type of variable will be modified as follows: - - `P*` : `Vec

` - - `P+` : `Vec

` - - `P?` : `Option

` - - `$sep(P, P_separator, repetetion)` : `Vec

` - -You can still access the `Vec` or `Option` by using the base name of the pattern. -```rust -E(i32) : A* { - println!("Value of A: {:?}", A); // Vec -}; -``` +### Variable Types for Patterns +When matching regex patterns, the bindings yield the following Rust types: +- **`P*`** : `Vec

` +- **`P+`** : `Vec

` +- **`P?`** : `Option

` +- **`$sep(P, Sep, Rep)`** : `Vec

` ### Terminal Sets -For terminal set `[term1 term_start-term_end]`, `[^term1 term_start-term_end]`, there is no predefined variable name. You must explicitly define the variable name. +For terminal sets (like `[plus minus]`), no default variable name is generated. You must assign a variable name to capture the token value: ```rust -E: digit=[zero-nine] { - println!("Value of digit: {:?}", digit); // %tokentype -}; +Op(Token) : op=[plus minus] { op }; ``` ### Pattern Groups -For group `(P1 P2 P3 | P4 P5 | P6)`: - - For each line `P1 P2 P3`, `P4 P5`, and `P6`, if every line holds a same type of value `T`, the group will hold a `T` value. Else, the group will not hold any value. - - Rule type for each line will be determined by following rules: - - If none of the tokens in the line holds a value, the group will not hold any value - - If only one of the tokens in the line holds a value, the group will hold that value - - If there are multiple tokens in the line holding value, the group will hold a `Tuple` of those values - - There is no default variable name for the group, you must define the variable name explicitly with the `=` operator - - ```rust - NoRuleType: ... ; - - I(i32): ... ; - - // I will be chosen - A: i=(NoRuleType I NoRuleType | I) { - println!("Value of I: {:?}", i); // can access by 'i' - i - }; - -// (I I) and A does not match, so it will not hold any value - B: ( I NoRuleType I | A ) { - }; - - ``` - -## Exclamation Mark `!` -An exclamation mark `!` can be used right after the token to ignore the value of the token. -The token will be treated as if it is not holding any value. +For groupings like `(P1 P2 | P3)`: +- If every branch in the group evaluates to the same type `T`, the group yields a value of type `T`. Otherwise, it yields no value. +- The type of a branch is: + - Empty if no symbol yields a value. + - `T` if exactly one symbol yields a value. + - A tuple `(T1, T2, ...)` if multiple symbols yield values. +- Groups have no default variable name. You must explicitly bind them: + ```rust + Expr(i32) : val=( Term | '+' Term ) { + match val { + // ... + } + }; + ``` + +--- + +## Exclamation Mark (`!`) + +An exclamation mark (`!`) appended immediately after a symbol tells the parser to discard its value. This is useful for omitting unneeded tokens (like punctuation or delimiters) from the default variable scope. ```rust -A(i32) : ... ; - -// A in the middle will be chosen, since other A's are ignored -E(i32) : A! A A!; +// Only the middle `Expr` is kept; the parentheses values are discarded +Expr(i32) : '('! Expr ')'! ; ``` -## Tracing Non-Terminals -Putting non-terminals in `%trace` directive will enable tracing for that non-terminal. -By calling `context.trace(): HashSet`, you can get the set of tracing non-terminals -that the current context is trying to parse. +--- - - Tracing non-terminals will not be automatically removed from the grammar by optimization. +## Tracing Non-Terminals ``` -%trace NonTerm1 NonTerm2 ...; +%trace NonTerm1 NonTerm2 ... ; ``` +Registers non-terminals for tracing. When registered, you can query the active parsing goals at runtime using `context.trace(&parser)`, which returns a `HashSet`. + +Additionally, tracing prevents the optimization engine from merging or optimizing away these non-terminal states. + +--- + +## Start Symbol (Must Defined) -## Start symbol (must defined) ``` %start NonTerminalName ; ``` -Set the start symbol of the grammar as `NonTerminalName`. -```rust -%start E; -// This internally generates augmented rule Augmented -> E eof - -E: ... ; -``` +Defines the entry point of the grammar. RustyLR automatically creates an augmented rule `Augmented -> NonTerminalName eof`. +--- +## Userdata Type (Optional) -## Userdata type (optional) ``` %userdata ; ``` -Define the type of user data passed to the `feed()` function. - -**Example:** -```rust -struct MyUserData { ... } -... +Specifies a custom state type passed to the parser's `feed` and `accept` functions. This allows you to thread compilation state, symbol tables, or diagnostics through your reduce actions. -%userdata MyUserData; +### Example +```rust +struct ParserState { + errors: Vec, +} -... +// In the grammar: +%userdata ParserState; -fn main() { - ... - let mut userdata = MyUserData { ... }; - parser.feed(..., token, &mut userdata); // <-- userdata fed here -} +// In your reduce action: +Expr(i32) : Term { + if Term < 0 { + data.errors.push("Negative number encountered".to_string()); + } + Term +}; ``` +--- ## Resolving Conflicts -### Panic Mode Error Recovery -``` -JsonObject: '{' JsonKeyValue* '}' - | '{' error '}' { println!("recovering with '}}' at {}", @error); } - ; +### Panic-Mode Error Recovery + +```rust +Expr: '{' Stmt* '}' + | '{' error '}' { println!("Recovered from syntax error at: {:?}", @error); } + ; ``` -The `error` token is a reserved terminal symbol that can be matched with **any zero or more tokens**. -In the above example, if the parser encounters an invalid token while parsing a JSON object, it will enter panic mode and discard all tokens until it finds a closing brace `}`. -**How it works:** -When an invalid token is encountered, the parser enters panic mode and starts discarding symbols from the parsing stack until it finds a point where the special `error` token is allowed by the grammar. -At that point, it shifts the invalid fed token as the `error` token, then tries to complete the rule that contains the `error` token. +The reserved terminal `error` is used to implement panic-mode recovery. When the parser encounters an unexpected token: +1. It pops states off the parser stack until it finds a state that can shift the `error` symbol. +2. It shifts the `error` symbol. +3. It discards input tokens until it encounters a token that can legally follow `error` in the grammar (in this case, `}`). +4. It reduces the recovery rule, letting the parser resume normal execution. -**Important notes:** -- The `error` token does not have any value, no associated rule-type -- `error` token does have location data, which can be accessed in the reduce action by `@error`. The location data is merged from the invalid tokens that consist of the `error` token. -- In GLR parsing, the `error` path will be ignored if there are any other valid paths. In other words, it enters panic-mode only if there is no other way to feed the terminal symbol +#### Notes +- The `error` token does not carry a semantic value. +- You can access the span of all discarded tokens using the location binder `@error`. +- In GLR mode, the parser will prefer non-error paths and only trigger error recovery if all other active branches fail. ### Operator Precedence + +``` +%left term1 term2 ... ; +%right term3 term4 ... ; +%precedence term5 term6 ... ; ``` -// reduce first -%left term1 term2 term3 ...; -// shift first -%right term4 term5 term6 ... ; +Used to resolve shift/reduce conflicts in expressions. The precedence of terminals is defined by the declaration order of the `%left`, `%right`, and `%precedence` directives (lower to higher). -// only precedence -%precedence term7 term8 term9 ... ; -``` -For shift/reduce conflicts, the `%left`, `%right`, and `%precedence` directives are used to resolve the conflicts. -These directives define the associativity and precedence of operators. -As in `bison`, the order of precedence is determined by the order in which `%left`, `%right`, or `%precedence` directives appear. +- **`%left`**: Declares left-associative operators (e.g., `a + b + c` parses as `(a + b) + c`). +- **`%right`**: Declares right-associative operators (e.g., `a ^ b ^ c` parses as `a ^ (b ^ c)`). +- **`%precedence`**: Declares precedence without associativity. Banning chained usage of the operator without parentheses. -**Conflict Resolution:** -When a conflict occurs, the parser will compare the precedence of the shift terminal and the *operator* in the reduce rule. If both precedences are defined, either the shift or reduce will be chosen based on the precedence of the operator. - - If the shift terminal has a higher precedence than the reduce operator, the shift will be chosen - - If the reduce operator has a higher precedence than the shift terminal, the reduce will be chosen - - If both have the same precedence, the `%left` or `%right` directive will be used to determine the resolving process +When a conflict arises between shifting a terminal and reducing a rule: +1. The parser compares the precedence of the lookahead terminal to the precedence of the rule's *operator*. +2. The rule's operator is the rightmost terminal in the rule that has a precedence assigned. +3. If the lookahead terminal has higher precedence, the parser **shifts**. +4. If the rule operator has higher precedence, the parser **reduces**. +5. If precedences are equal: + - For `%left`, the parser **reduces**. + - For `%right`, the parser **shifts**. + - For `%precedence`, the parser reports a syntax error. -The *operator* of the reduce rule is the rightmost terminal symbol in the production rule that has a precedence defined by `%left`, `%right`, or `%precedence` directive. Alternatively, the operator of the reduce rule can be defined explicitly by using the `%prec` directive. +#### Explicit Precedence (`%prec`) +You can override a rule's default operator using the `%prec` directive followed by a terminal name: -**Examples:** ```rust -// left reduction for binary operator '+' %left '+'; +%left '*'; +%left UnaryMinus; -// right reduction for binary operator '^' -%right '^'; +Expr + : Expr '+' Expr + | Expr '*' Expr + | '-' Expr %prec UnaryMinus // Gives unary minus higher precedence than '*' + ; ``` +You can also use a non-terminal in `%prec` if the operator is determined dynamically: ```rust -%left '+'; -%left '*'; -%left UnaryMinus; // << highest priority +Expr : Expr op=BinOp Expr %prec op { ... }; -E: E '+' E { E + E } - | E '*' E { E * E } - | '-' E %prec UnaryMinus { -E } // make operator for this production rule `UnaryMinus` - ; +BinOp : '+' | '*'; ``` +Here, the rule's precedence matches the specific operator that `BinOp` resolved to. -You can also set the precedence of a rule by referencing a non-terminal symbol with `%prec`. -This is useful when an operator is represented by a non-terminal that produces different actual operators. +### Rule Priority -**Example:** ```rust -%left '+'; -%left '*'; - -E: E op=BinOp E %prec op { ... } - | ... - ; - -// The precedence of `BinOp` is determined by the production rule that `BinOp` was derived from. -BinOp: '+' - | '*' - ; +Expr + : Expr '+' Expr %dprec 2 + | Expr '*' Expr %dprec 1 + ; ``` -In this example, the `E: E op=BinOp E` rule's precedence is determined by the `BinOp` non-terminal. -When the parser needs to resolve a conflict involving this rule, it will look at what `BinOp` was reduced from. -If `BinOp` was reduced from `+`, the precedence of `BinOp` will be the precedence of `+`, which is defined by `%left '+'`. -otherwise, if `BinOp` was reduced from `*`, the precedence of `BinOp` will be the precedence of `*`, which is defined by `%left '*'`. - -If any non-terminal symbol was referenced in the `%prec` directive, -every production rule in that non-terminal must have operator precedence. +Assigns static priority numbers to rules to resolve reduce/reduce conflicts. If two rules can be reduced at the same time, the parser resolves the conflict by choosing the rule with the highest `%dprec` value (default is `0`). -### Rule priority -``` -E: - P1 P2 P3 %dprec 2 - | P4 P5 P6 %dprec 1 - ; -``` -For reduce/reduce conflicts, rule with the highest priority will be chosen. -The priority is defined by the `%dprec` directive. Default priority is `0`. +--- +## Error Type (Optional) -## Error type (optional) ``` %err ; %error ; ``` -Define the type of `Err` variant in `Result<(), Err>` returned from [`ReduceAction`](#reduceaction-optional). If not defined, `DefaultReduceActionError` will be used. - -**Example:** -```rust -enum MyErrorType { - ErrVar1, - ErrVar2, - ErrVar3(T), -} -... +Defines a custom error type returned by reduce actions. If your reduce action returns a `Result`, returning an `Err(custom_error)` will stop execution (or prune the GLR branch) and bubble the error up wrapped in `ParseError::ReduceAction`. -%err MyErrorType ; - -... - -match parser.feed( ... ) { - Ok(_) => {} - Err(err) => { - match err { - ParseError::ReduceAction( err ) => { - // do something with err - } - _ => {} - } - } -} -``` +--- +## LALR Parser Generation -## LALR parser generation ``` %lalr; ``` -Switch generated parser table to LALR parser. -By default, the parser will be generated as minimal-LR(1) parser. +Forces RustyLR to generate LALR(1) parsing tables. By default, RustyLR builds minimal-LR(1) (IELR-style) tables, which provide full LR(1) parsing strength while keeping table sizes close to LALR(1). +--- + +## GLR Parser Generation -## GLR parser generation ``` %glr; ``` -Switch to GLR parser generation. -If you want to generate a GLR parser, add the `%glr;` directive to your grammar. -With this directive, any Shift/Reduce, Reduce/Reduce conflicts will not be treated as errors. +Enables Generalized LR (GLR) parser generation. With `%glr;` enabled, shift/reduce and reduce/reduce conflicts are not treated as compiler errors. Instead, the parser splits into parallel execution branches at runtime. + +--- -See [GLR Parser](GLR.md) section for more details. +## No Optimization -## No optimization ``` %nooptim; ``` -Disable grammar optimization. -## Dense parser table +Disables optimization passes on the generated table (which merge states and group equivalent terminals). Use this for debugging or to speed up the parser generation compilation phase itself. + +--- + +## Dense Parser Table + ``` %dense; ``` -Normally, the generated code will use `HashMap` to store the parser table. -This directive will force the parser to use `Vec` instead of `HashMap`. -**Be careful:** this could increase memory usage significantly. +Forces the generated parser table to use flat `Vec` indexes instead of `HashMap` lookups. This significantly speeds up token feeding, but can drastically increase the binary size of the generated parser. + +--- + +## Location Tracking -## Location tracking ``` -%location ; +%location ; ``` -The location type must implement `rusty_lr::Location` trait. -User must explicitly feed the location data to the parser. +Enables location tracking in the parser. The specified `` must implement the `rusty_lr::Location` trait: + ```rust -context.feed_location(parser, terminal, user_data, terminal_location); -context.feed(parser, terminal, user_data); // this is equivalent to using `Location::default()` for the location +pub trait Location: Default + Clone { + fn merge(&self, other: &Self) -> Self; +} ``` -And the location data can be accessed in the reduce action by `@name` syntax. +To feed locations, use `feed_location` instead of `feed`: ```rust -Expr: exp1=Expr '+' exp2=Expr { - println!("Location of exp1: {:?}", @exp1); - println!("Location of exp2: {:?}", @exp2); - println!("Location of this expression: {:?}", @$); // @$ is the location of the non-terminal itself - exp1 + exp2 -}; +context.feed_location(&parser, token, &mut userdata, span); ``` -Default location type is `rusty_lr::DefaultLocation`. This does not hold any data. \ No newline at end of file +Within reduce actions, access symbol spans using the `@` prefix: +```rust +Expr(i32) : e1=Expr '+' e2=Expr { + println!("Start/End position: {:?}", @$); // Location of this Expr + println!("Left operand position: {:?}", @e1); + e1 + e2 +}; +``` \ No newline at end of file