Non-terminals, e.g. expr, have diverged between parser and macro matcher

[pnkfelix]

Spawned off of PR #84364, based on discussion from lang team meeting (minutes, youtube).

The meaning of the non-terminal "expr" according to the rustc parser has changed (or would like to change) from its meaning according to the current rustc macro matcher.

Concretely:

• The inline_const feature has already added a production roughly of the form EXPR ::= const { EXPR }
• The let_chains feature may want (in the future) to add a production roughly of the form COND_EXPR ::= let PAT = EXPR, where COND_EXPR can occur as a sub-expression in the condition that drives an if expression or while statement.

In both cases, for backwards compatibility, the expr fragment specifier does not accept these new productions. Examples of the kinds of problems this causes follow:

• example for inline_const,
• example for let_chains.

In other words, today you simply cannot write a macro fragment specifier denoting expressions that will match const { EXPR } or let PAT = EXPR, other than something like $e:tt, which would also match arbitrary token-trees, which won't catch errors as effectively as a more precise fragment specifier.

So, the main question this raises is: How should we resolve this divergence between the parser and macro-rules matcher?

• E.g. we could add a new fragment specifier (e.g. expr2024) that covers the new forms and call it a day.
• Or we could, at an edition boundary, change the meaning of the expr fragment specifier to cover the new forms. (This would probably need to be coupled with an addition of a new fragment specifier that would capture the old semantics.)
• In any case, we need a clear summary of the grammar changes, so that we can understand what expr matches now that it did not before.
• I think any change we make here would be well-served to be documented by writing down the actual grammar being implemented, or rather, the changes to the grammar.
• We also may need a crater run.

[pnkfelix]

Also, lang team discussion has concluded that we don't have to solve this problem until one of inline_const (#76001) or let_chains (#53667) is being stabilized.

[petrochenkov]

It's not only expr, other matchers may be affected too.
The matcher logic needs to be audited and implemented in terms of "reused parser code" + "explicit exceptions for backward compatibility", instead of the inconsistent ad hoc mess that is used now.
UPD: At least expr and ty use can_begin_expr/can_begin_type from the parser now, IIRC it was worse in the past.

The "explicit exceptions for backward compatibility" part can be made conditional on an edition, but that's a technicality.

[bstrie]

Another option is to do the maximally backwards-compatible thing with macros 1.0 (either nothing at all, or introduce a unique fragment specifier), and then fix this properly in macros 2.0 while they're still unstable.

[bstrie]

Worth noting that this appears to be the one and only remaining issue blocking the stabilization of inline_const, which itself is the only remaining issue blocking the stabilization of asm_const, so getting this sorted out would be very impactful.

[memoryruins]

@pnkfelix

Also, lang team discussion has concluded that we don't have to solve this problem until one of inline_const (#76001) or let_chains (#53667) is being stabilized.

The stabilization proposal for let_chains is currently entering its final comment period #94927 (comment). I believe its macro concern in #94927 (comment) was resolved by forbidding let in certain places in #97295 (unless if there was another change that helped resolved it).

Would it be possible to make a similar change to unblock inline_const? and/or now that let_chains is in the process of being stabilized, should this issue be prioritized?

cc @joshtriplett since you opened/resolved the macro concerns on let_chains stabilization, opened an FCP for a different asm feature recently, and are lang team :)

[joshtriplett]

Since let PAT = expr isn't allowed in an arbitrary expression, only in if let and while let, it sholdn't be added to expr. And AFAICT you currently have to match if-let and while-let explicitly in macro matches if you want their contents. So, people would need to add support for let chains in those matches. We could add something like cond to help with that, but I don't think that's a blocker.