DataStack to use a single unified enum stack & document memory optimizations

[ehwan]

This PR refactors the generated parser's DataStack implementation from using multiple type-specific vectors and a tracking tag vector to a single, unified vector containing all token/rule variants wrapped in a single generated enum. This significantly simplifies the generated parser code, reduces generated code size, and improves runtime management of stack operations.
Additionally, this PR adds a documentation section on optimizing memory usage for large rule types.

Key Changes

1. Unified Data Stack Enum (emit.rs):
   • Replaced individual type vectors (__stack0, __stack1, etc.) and the tracking tag vector (__tags) with a single enum (e.g., GrammarData) and a single stack vector (__stack: Vec<GrammarData>).
   • Simplified the generated DataStack trait methods (pop, clear, reserve, split_off, truncate, append) to directly delegate to the single underlying __stack vector.
   • Refactored reduce actions to pop from the single stack and destructure the rule values via pattern matching.
   • Adjusted pop_start to pop the trailing EOF (Empty token) before extracting the final parse result.
2. Simplified Variant Tracking:
   • Cleaned up generator helpers (variant_names_for_nonterm and ruletype_variant_map) to store and return Ident directly instead of wrapping them in Option<Ident>, mapping untyped symbols to the Empty variant.
3. Documentation (SYNTAX.md):
   • Added a new subsection Memory Optimization with Box under the RuleType section. It clarifies that since all parser semantic values are stored in a single unified enum, wrapping large data types in a Box (e.g., Box<MyLargeStruct>) is recommended to prevent stack memory bloat.

Verification

• Re-bootstrapped the parser generator locally.
• Verified that all generated parser modules (such as calculator, JSON, and GLR examples) build correctly and all automated unit tests pass successfully.

[gemini-code-assist]

Code Review

This pull request introduces a memory optimization to the generated parser by replacing multiple typed stacks and a tag stack with a single unified data stack containing a newly introduced unified enum. This change simplifies stack operations like clear, split_off, truncate, and append into single vector operations, and the documentation is updated to recommend wrapping large types in a Box to minimize enum variant sizes. Feedback on the changes highlights two critical compilation issues in the code generator: first, several streams (such as debug_tag_check_stream and extract_data_stream) are defined inside an if block but are referenced in the corresponding else block; second, the Empty variant is conditionally generated, yet unconditionally referenced by methods like push_empty and pop_start, which will cause compilation failures when no empty tags are used.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The variables debug_tag_check_stream, extract_data_stream, alias_stream, and custom_reduce_action_stream are defined inside the if rule.is_used block, but they are referenced in the else block (around line 1384). This will cause a compilation error in the generator. Defining them as empty TokenStreams before the if block resolves this issue.

                let debug_tag_check_stream = TokenStream::new();
                let extract_data_stream = TokenStream::new();
                let alias_stream = TokenStream::new();
                let custom_reduce_action_stream = TokenStream::new();

                if rule.is_used {

[gemini-code-assist]

The Empty variant is conditionally added to the data enum only if empty_tag_used is true. However, generated methods like push_empty and pop_start unconditionally reference #data_enum_typename::Empty. If a grammar has no empty tags used, the generated parser will fail to compile due to the missing Empty variant. The Empty variant should be unconditionally included in the enum.

            variants.extend(quote! {
                Empty,
            });

[ehwan]

added test codes