Tracking Issue for RFC 3101: Reserved Prefixes

[nikomatsakis]

This is a tracking issue for the RFC "Reserved Prefixes" (rust-lang/rfcs#3101).
The feature gate for the issue is #![feature(reserved_prefixes)].

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement the RFC (implementation details are tracked in Implement RFC: Reserved prefixes in the 2021 edition #84599)
• Adjust documentation (see instructions on rustc-dev-guide)
• Stabilization FCP (see instructions on rustc-dev-guide) -- Stabilize reserved prefixes #88140

Unresolved Questions

• How to manage the API proc_macro::TokenStream::from_str, which does not take any edition information? (raised here) --> Discussed in Reserved Prefixes RFC: How to handle TokenStream::from_str? #84979

Implementation history

[lrh2000]

@rustbot claim

I'm interested at implementing this. But there are some questions.

1. The RFC says foo# bar is treated as three tokens. But currently r# bar failed with found invalid character.

• The error is weird and unnecessary (in my opinion).
• It may lead to bad consistency. For example, we may reserve the prefixxr# and accept xr# bar as three tokens (that's what the RFC specified). Once we decide to use xr# represents something like a raw string, xr# bar may not compile anymore (due to the error found invalid character).

Possible solutions:

• Remove the error found invalid character to improve consistency. I don't think it breaks anything.
• Treat everything starting with foo# as an error (suggested by @m-ou-se). It is easy to implement but it does not conform to the requirements of the RFC.

2. The RFC does not describe some corner cases clearly. I give my expected results for them below. (It's according to the RFC, and obviously we get different results if we deny everything starting from foo#.)

• foo## bar
  4 tokens foo, #, #, bar (NOTE: However,r## bar cannot compile now.)
• foo#! bar
  4 tokens foo, #, !, bar (NOTE: However, r#! bar cannot compile now.)
• foo##! bar
  5 tokens foo, #, #, !, bar (NOTE: However, r##! bar cannot compile now.)
• foo##" bar
  error: reserved prefixes
• foo##bar
  error: reserved prefixes
• r#foo#bar
  error: reserved prefixes
• r"foo"#bar
  3 tokens "foo", #, bar

(See also the discussion at zulip https://rust-lang.zulipchat.com/#narrow/stream/268952-edition-2021/topic/reserved.20prefixes/near/238470099.)

[nikomatsakis]

@lrh2000 great! I'm going to spin off an implementation issue, I think to track this discussion. (I should've done this from the start, perhaps.)

I take that back. These are kind of design questions for now, and this is simple enough. Let's just keep it here.

I take THAT back. There was already an issue here #84599

[steffahn]

Some relevant examples regarding string (and similarly char) literals – the first two copied from the linked zulip discussion (archive) – include

match"foo" { _ => () }

if"foo".eq("foo") {}

while"foo".ne("foo") {}

if let"foo" = "foo" {}

loop {
    break"foo"
};

for _ in"foo".chars() {}

return"foo"

(playground)

which raises the question whether we want to exclude all keywords or maybe certain keywords from the disallowed prefixes.

[petrochenkov]

whether we want to exclude all keywords or maybe certain keywords from the disallowed prefixes

Note that lexer isn't currently aware of the keyword list (*), and making it aware is a step in the wrong direction, IMO.
I generally support the @matklad's idea of working with literal prefixes using jointness (#84979 (comment)), this is the direction in which multi-character punctuation and "compound" float literals are supposed to move in accordance with #71322 (comment).

(*) After all, every left-hand side of a macro is able to introduce its own syntax entirely independent from Rust syntax, with its own keyword list, and lexer works with macro inputs too.

[nikomatsakis]

@petrochenkov I was wondering about "jointness". The idea here would be that we produce the same tokens, but in the Rust parser we check for (e.g.) k#foo and interpret it differently than k# foo? (By inspecting the "is there adjoining whitespace" info?)

[petrochenkov]

Yeah, it should be applicable to prefix#ident too in some form (with less urgency), but I was talking about prefixLITERAL specifically (as an answer to #84978 (comment)) because it has larger compatibility issues.

[lrh2000]

@petrochenkov So do you mean that we should emit errors for reserved prefixes in the parser instead of the lexer?

I think it does not solve the problem about TokenStream::from_str. Instead, it only defers the problem. We may not alter the lexing rules now, but in the future we have to change the lexing rules if we want to introduce another prefix that is also used to denote raw strings (or something like them). Am I missing anything?

But I think it sounds good because it can easily solve almost all problems pointed out by @steffahn . For example, we can distinguish lifetimes from string literals and we can also distinguish keywords from identifiers. And we can have better diagnostics messages like

error: unknown prefix on string literal: bar
 --> file.rs:x:y
  |
1 | foo!(bar"qux");
  |      ^^^ help: try using whitespace here: `bar "qux"`
  |
  = note: prefixed string literals are reserved for future use

But I've found currently parsing the macro arguments and expanding macros is done in rustc_expand, so it means that we will emit errors for reserved prefixes both from rustc_parse and rustc_expand, does it sound good?

[petrochenkov]

@lrh2000

So do you mean that we should emit errors for reserved prefixes in the parser instead of the lexer?

Yes, at least for breakage-prone things like match"foo".
For anything involving raw strings with #s the errors can reported in the lexer just to save time and defer the final decision, I'm pretty sure that code like match#"foo"# is not used in practice.
What I'd really like to see is a crater result for the most conservative reservation, that would help to make decisions.

I think it does not solve the problem about TokenStream::from_str

Could you remind what is the problem with TokenStream::from_str exactly?

but in the future we have to change the lexing rules if we want to introduce another prefix that is also used to denote raw strings (or something like them). Am I missing anything?

String escaping is done during parsing (proc macros can do custom escaping in particular), so lexer doesn't need to discern between raw and non-raw strings unless #s are involved, but #s should be covered by the first item.

But I've found currently parsing the macro arguments and expanding macros is done in rustc_expand, so it means that we will emit errors for reserved prefixes both from rustc_parse and rustc_expand, does it sound good?

rustc_expand should always delegate to rustc_parse for parsing (maybe unless compatibility shims for tt matchers are involved?).
Sorry for the brief answers, I can't spend much time on this right now.