Skip to content

Comments

Implement token-aware PHP transformer with type declaration preservation#3

Merged
archisgore merged 1 commit intomainfrom
archis/token-scrambling
Jan 25, 2026
Merged

Implement token-aware PHP transformer with type declaration preservation#3
archisgore merged 1 commit intomainfrom
archis/token-scrambling

Conversation

@archisgore
Copy link

Summary

This PR implements a token-aware PHP transformer that properly preserves PHP type declarations, replacing the simple text-replacement approach that was incorrectly transforming type names.

Key Features

Type-Aware: Preserves PHP type names (int, string, bool, float, etc.) in type declarations
Context-Sensitive: Understands when keywords are used as types vs. code
Grammar-Correct: Uses PHP's token_get_all() for accurate parsing
Comprehensive: Handles all PHP 7.4+ type declaration features including:

  • Function parameter types
  • Return types
  • Property types (PHP 7.4+)
  • Union types (PHP 8.0+)
  • Nullable types
  • Special types (self, parent, static)

What Gets Preserved

  • Scalar types: int, float, string, bool
  • Compound types: array, object, callable, iterable
  • Special types: void, mixed, never, null, false, true
  • Class references: self, parent, static

Critical Bug Fix

Fixed handling of the static keyword which tokenizes as T_STATIC (not T_STRING). The transformer now correctly preserves static in type contexts while still transforming it as a keyword elsewhere.

Implementation

The transformer uses:

  1. Tokenization: token_get_all() for proper PHP parsing
  2. Context Tracking: TransformContext class maintains state about function signatures, parameter lists, return types
  3. Type Detection: Identifies when tokens appear in type declaration contexts
  4. Selective Transformation: Only transforms tokens that are not in type positions

Files Added

  • encrypted-execution/tools/scrambler/token-aware-transformer.php - Main transformer implementation
  • encrypted-execution/tools/scrambler/README-token-aware.md - Documentation
  • encrypted-execution/tools/scrambler/test-type-declarations.php - Comprehensive test suite
  • encrypted-execution/tools/scrambler/test-dictionary.json - Test dictionary

Testing

Comprehensive tests validate:

  • Function parameter types preserved
  • Return types preserved
  • Property types preserved
  • Union types preserved
  • Nullable types preserved
  • Special keywords (self, parent, static) preserved in type contexts
  • Keywords correctly transformed in non-type contexts

Example

// Input
function foo(int $x, ?string $y): static {
    return new static();
}

// Output (All type names preserved!)
FUNC foo(int $x| ?string $y): static {
    RET NEW_KW static()~
}

This fixes the root cause of the WordPress PR failures where type names were being scrambled, causing PHP parse errors.

🤖 Generated with Claude Code

This transformer uses PHP's built-in tokenizer to properly parse PHP code
and preserve type declarations while transforming keywords and operators.

Key Features:
- Uses token_get_all() for accurate PHP parsing
- Tracks context (parameter lists, return types, property types)
- Preserves all PHP type names in type declarations
- Handles PHP 7.4+ and 8.0+ type features
- Special handling for T_STATIC token

Files Added:
- token-aware-transformer.php: Main transformer implementation
- snip-transform-v2.php: Updated snip-transform with type awareness
- transform-php-file-v2.php: Standalone file transformer
- README-token-aware.md: Comprehensive documentation
- Test files for validation

This solves the fatal error issue where type names were being scrambled:
  Cannot use int as default value for parameter of type ?XYZ

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@archisgore archisgore merged commit cd680a5 into main Jan 25, 2026
1 check passed
@archisgore archisgore deleted the archis/token-scrambling branch January 25, 2026 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant