Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 13 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,21 +152,8 @@ fn main() {
The generated parser module contains several generated components tailored to your start symbol:
- **`<Start>Parser`**: A lightweight struct containing the static parsing tables. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/trait.Parser.html)
- **`<Start>Context`**: A mutable state context that keeps track of the stack and parsed symbol values. [(LR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/deterministic/struct.Context.html) [(GLR docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/nondeterministic/struct.Context.html)
- **`<Start>State`**: An internal type representing individual states in the parser. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/state/trait.State.html)
- **`<Start>Rule`**: An internal enum representing production rules. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/rule/struct.ProductionRule.html)
- **`<Start>NonTerminals`**: An enum representing all non-terminal symbols. [(docs)](https://docs.rs/rusty_lr/latest/rusty_lr/parser/nonterminal/trait.NonTerminal.html)

### Interacting with the Parsing Context
The `<Start>Context` offers helpful utilities for inspecting and tracing:
```rust
let mut context = ExprContext::with_default_userdata();

// ... feed tokens ...

context.expected_token(); // Returns the expected symbols for the current state
context.can_feed(&token); // Checks if a terminal can be fed next
println!("{}", context); // Formats the state tree (requires 'tree' feature)
```
The generated module also includes internal state, rule, and non-terminal types used by the runtime and debugging APIs.

### Feeding Tokens
You can feed terminal symbols either with or without location information:
Expand Down Expand Up @@ -235,6 +222,18 @@ rustylr --state src/grammar.rs

![State Machine Debug](images/state_option.png)

The `<Start>Context` also offers inspection utilities that are useful while debugging a parser:

```rust
let mut context = ExprContext::with_default_userdata();

// ... feed tokens ...

context.expected_token(); // Returns the expected symbols for the current state
context.can_feed(&token); // Checks if a terminal can be fed next
println!("{}", context); // Formats the state tree (requires 'tree' feature)
```

---

## Examples
Expand Down
103 changes: 60 additions & 43 deletions SYNTAX.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,15 @@ This document provides a comprehensive guide to the grammar definition syntax us
- [Conflict Resolution](#resolving-conflicts)
- [Panic-Mode Error Recovery (`error`)](#panic-mode-error-recovery)
- [Operator Precedence (`%left`, `%right`, `%precedence`, `%prec`)](#operator-precedence)
- [Error Type (`%err` / `%error`)](#error-type-optional)
- [Error Type (`%error`)](#error-type-optional)
- [Disabling Table Optimization (`%nooptim`)](#no-optimization)
- [Dense Parser Tables (`%dense`)](#dense-parser-table)
- [Location Tracking (`%location`)](#location-tracking)
- [Variable Substitution](#variable-substitution)
- [Advanced Parser Controls](#advanced-parser-controls)
- [Memory Optimization with `box`](#memory-optimization-with-box)
- [Custom Token Ranges](#custom-token-ranges)
- [Dynamic Precedence with Non-Terminals](#dynamic-precedence-with-non-terminals)
- [GLR Parser Generation (`%glr`)](#glr-parser-generation)
- [Advanced GLR Reduce Controls](#advanced-glr-reduce-controls)
- [Rule Priority (`%dprec`)](#rule-priority)
Expand Down Expand Up @@ -191,21 +194,7 @@ RustyLR supports rich regular expression patterns on the right-hand side of prod
- **`"abcd"` / `b"abcd"`** : String/byte string literals (only valid if `%tokentype` is `char` or `u8`).
- **`P - TerminalSet`** : Matches pattern `P` but excludes any terminal in the `TerminalSet`.

### Range Patterns (`[first-last]`)
When defining ranges of custom tokens (e.g. `[zero-nine]`), the range ordering is determined by the **declaration order of your `%token` directives**, rather than the inherent values of the enum elements.

#### Example
If tokens are declared as:
```rust
%token zero Token::Num(0);
%token one Token::Num(1);
%token two Token::Num(2);
// ...
%token nine Token::Num(9);
```
The range `[zero-two]` matches `zero`, `one`, and `two`.

If the `%tokentype` is `char` or `u8`, literal character ranges like `['0'-'9']` or `[b'0'-b'9']` are resolved using standard ASCII values.
Literal character and byte ranges such as `['0'-'9']` or `[b'0'-b'9']` are resolved by their literal values. Custom token ranges are also supported for advanced grammars; see [Custom Token Ranges](#custom-token-ranges).

---

Expand All @@ -229,24 +218,7 @@ Expr(_): Term;
> [!WARNING]
> If a circular dependency exists (e.g., two rules trying to infer their types from one another without a base type), RustyLR will fail with a compilation error.

### Memory Optimization with `Box` / `box` keyword
Internally, the generated parser stores all semantic values (the `%tokentype` and all non-terminals' `RuleType`s) in a single unified `enum` representing the parser's data stack.

Because the memory footprint of a Rust `enum` is dictated by its largest variant, if even one `RuleType` is exceptionally large (e.g., a large AST struct), the size of *every* stack slot will inflate. This can result in significant memory waste and performance degradation.

To avoid this, you can write the `box` keyword in front of `%tokentype` or any non-terminal's `RuleType` (inside the parentheses, e.g. `(box Type)`). The parser generator will automatically box that type in the data enum (generating `Box<Type>`), and will automatically wrap (`Box::new(...)`) or unwrap (`*val`) it in reduce actions so that you do not have to write `Box::new` or dereference it yourself:

```rust
%tokentype box MyToken;

Expr (box MyLargeASTNode)
: left=Expr '+' right=Term {
// `left` and `right` are automatically unboxed (of type `MyLargeASTNode`),
// and the returned value is automatically wrapped in a `Box::new`.
MyLargeASTNode::Binary(left, right)
}
;
```
For large semantic values, RustyLR also supports boxing selected stack variants. See [Memory Optimization with `box`](#memory-optimization-with-box).

---

Expand Down Expand Up @@ -459,18 +431,11 @@ Expr
;
```

You can also use a non-terminal in `%prec` if the operator is determined dynamically:
```rust
Expr : Expr op=BinOp Expr %prec op { ... };

BinOp : '+' | '*';
```
Here, the rule's precedence matches the specific operator that `BinOp` resolved to.
For cases where the operator is represented by a non-terminal, see [Dynamic Precedence with Non-Terminals](#dynamic-precedence-with-non-terminals).

## Error Type (Optional)

```
%err <RustType> ;
%error <RustType> ;
```

Expand Down Expand Up @@ -532,7 +497,7 @@ You can use variables prefixed with `$` inside any RustCode block in the grammar
- `%tokentype`
- `%location`
- `%userdata`
- `%errortype` (or `%error`)
- `%error`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since %errortype is no longer supported as a directive (and has been replaced by %error), please also update the description of $error or $errortype in the Supported Variables section below (around line 509) to remove the reference to %errortype for consistency.

- `%token` terminal definitions
- Non-terminal rule types
- Reduce actions
Expand Down Expand Up @@ -599,6 +564,58 @@ This syntax sugar is extremely useful for reducing boilerplate code when process

The controls in this section are useful for ambiguous grammars, parser-mode tuning, or conflict cases that cannot be expressed cleanly with ordinary precedence declarations. Most grammars can start without these directives.

### Memory Optimization with `box`

Internally, the generated parser stores all semantic values (the `%tokentype` and all non-terminals' `RuleType`s) in a single unified `enum` representing the parser's data stack.

Because the memory footprint of a Rust `enum` is dictated by its largest variant, if even one `RuleType` is exceptionally large (e.g., a large AST struct), the size of *every* stack slot will inflate. This can result in significant memory waste and performance degradation.

To avoid this, you can write the `box` keyword in front of `%tokentype` or any non-terminal's `RuleType` (inside the parentheses, e.g. `(box Type)`). The parser generator will automatically box that type in the data enum (generating `Box<Type>`), and will automatically wrap (`Box::new(...)`) or unwrap (`*val`) it in reduce actions so that you do not have to write `Box::new` or dereference it yourself:

```rust
%tokentype box MyToken;

Expr (box MyLargeASTNode)
: left=Expr '+' right=Term {
// `left` and `right` are automatically unboxed (of type `MyLargeASTNode`),
// and the returned value is automatically wrapped in a `Box::new`.
MyLargeASTNode::Binary(left, right)
}
;
```

### Custom Token Ranges

When defining ranges of custom tokens (e.g. `[zero-nine]`), the range ordering is determined by the **declaration order of your `%token` directives**, rather than the inherent values of the enum elements.

If tokens are declared as:

```rust
%token zero Token::Num(0);
%token one Token::Num(1);
%token two Token::Num(2);
// ...
%token nine Token::Num(9);
```

The range `[zero-two]` matches `zero`, `one`, and `two`.

For ordinary grammars, prefer explicit terminal sets like `[zero one two]` unless declaration-order ranges materially simplify the grammar.

### Dynamic Precedence with Non-Terminals

You can use a non-terminal in `%prec` if the operator is determined dynamically:

```rust
Expr : Expr op=BinOp Expr %prec op { ... };

BinOp : '+' | '*';
```

Here, the rule's precedence matches the specific operator that `BinOp` resolved to.

For ordinary unary and binary operators, a named precedence marker such as `UnaryMinus` is usually clearer.

### GLR Parser Generation

```
Expand Down
2 changes: 1 addition & 1 deletion example/calculator/src/parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ pub enum Token {
%start E;

// error type
%err String;
%error String;

// define tokens
%token num Token::Num(_); // `num` maps to `Token::Num(0)`
Expand Down
2 changes: 1 addition & 1 deletion example/glr/src/parser.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
use rusty_lr::lr1;

lr1! {
%err String;
%error String;
%glr;
%tokentype char;
%start E;
Expand Down
17 changes: 16 additions & 1 deletion rusty_lr_executable/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,16 @@ cargo install rustylr

## Usage
```bash
$ rustylr --help
$ rustylr my_grammar.rs my_parser.rs
```

The first argument is the grammar file. The optional second argument is the generated Rust output file; when omitted, RustyLR writes `out.tab.rs`.

## Advanced CLI Options

The options below are mainly useful for formatting control, parser-mode overrides, and diagnostics.

```bash
Usage: rustylr [OPTIONS] <INPUT_FILE> [OUTPUT_FILE]

Arguments:
Expand Down Expand Up @@ -59,6 +68,12 @@ Options:
Print version
```

### Diagnostic and Debug Options

- `--state <STATE>` prints details for a specific generated state.
- `--no-conflict`, `--no-conflict-resolve`, and `--no-backtrace` reduce conflict diagnostic output.
- `--glr <true|false>` and `--dense <true|false>` override parser-mode settings from the grammar file.

## Grammar File Format
The program searches for `%%` in the input file to separate Rust code from grammar definitions.

Expand Down
2 changes: 1 addition & 1 deletion rusty_lr_parser/src/grammar.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ pub struct Grammar {
/// %userdata
pub(crate) userdata_typename: TokenStream,

/// %err
/// %error
pub(crate) error_typename: TokenStream,

/// %start
Expand Down
4 changes: 2 additions & 2 deletions rusty_lr_parser/src/parser/lexer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ pub enum Lexed {
Start(Ident), // %start
TokenType(Ident), // %tokentype
UserData(Ident), // %userdata
ErrorType(Ident), // %err %error
ErrorType(Ident), // %error
ModulePrefix(Ident), // %moduleprefix
Lalr(Ident), // %lalr
Glr(Ident), // %glr
Expand Down Expand Up @@ -222,7 +222,7 @@ fn ident_to_keyword(ident: Ident) -> Option<Lexed> {
"start" => Some(Lexed::Start(ident)),
"tokentype" => Some(Lexed::TokenType(ident)),
"userdata" => Some(Lexed::UserData(ident)),
"err" | "error" => Some(Lexed::ErrorType(ident)),
"error" => Some(Lexed::ErrorType(ident)),
"moduleprefix" => Some(Lexed::ModulePrefix(ident)),
"lalr" => Some(Lexed::Lalr(ident)),
"glr" => Some(Lexed::Glr(ident)),
Expand Down