`rustc_parser` incorrectly parses groups with `Delimiter::None`

[petrochenkov]

Such groups can currently be created by proc macros.

Reproduction, proc macro crate:

#[proc_macro]
pub fn add_mul(_: TokenStream) -> TokenStream {
    // ⟪ 2 + 2 ⟫ * 2
    vec![
        TokenTree::from(Group::new(Delimiter::None, "2 + 2".parse().unwrap())),
        TokenTree::from(Punct::new('*', Spacing::Alone)),
        TokenTree::from(Literal::u8_unsuffixed(2)),
    ].into_iter().collect()
}

User crate:

fn main() {
    let x = add_mul!();
    assert_eq!(x, 8); // FAIL: the result is 6 != 8
}

This is not a huge issue right now because proc macros have no reasons to created Delimiter::None delimiters themselves (but they can still get them from input and return them if they return their input partially or fully, see the example below).

However, this will became a huge issue when interpolated tokens like $e: expr migrate from AST pieces to token streams, because in the token stream model they are represented exactly like groups with Delimiter::None delimiters (proc macros already see them in this way).

[petrochenkov]

How to reproduce this without constructing Delimiter::None by yourself.

Proc macro crate:

#[proc_macro]
pub fn add_mul(input: TokenStream) -> TokenStream {
    let mul_2 = vec![
        TokenTree::from(Punct::new('*', Spacing::Alone)),
        TokenTree::from(Literal::u8_unsuffixed(2)),
    ];
    input.into_iter().chain(mul_2.into_iter()).collect()
}

User crate:

macro_rules! mbe {
    ($e: expr) => ( add_mul!($e) )
}

fn main() {
    let x = mbe!(2 + 2);
    assert_eq!(x, 8); // FAIL: the result is 6 != 8
}

[petrochenkov]

What it means to parse Delimiter::None correctly is a big language design question in general.

However, it's clear that in pieces of grammar with expression-like priorities they should behave more or less like parentheses to keep the "operator priority hygiene" property of our macros.

"How would an interpolated token behave here?" should probably be the primary intuition here.

[petrochenkov]

cc @dtolnay @alexcrichton
To make you aware of this issue, and perhaps proc-macro2 and syn could attempt hiding this bug somehow.

[petrochenkov]

Encountered in practice in #74036.