remove language-level UB for non-UTF-8 str

[RalfJung]

This is the Rust-side issue for rust-lang/reference#792 just so that we can use fcpbot. The change description follows.

Ever since Rust 1.0, the reference said that a non-UTF-8 str causes immediate UB. In terms of today's terminology, that means that str has a validity invariant of being valid UTF-8.

However, that seems unnecessary: the compiler does not actually exploit this, nor is there any clear way it could exploit this. Making UTF-8 a library-level safety invariant is more than enough for everything str does. Most likely, it was made a validity invariant because we had not yet properly teased apart those two concepts when the document was initially written.

This is also the conclusion that the UCG WG arrived at in rust-lang/unsafe-code-guidelines#78.

I therefore propose we remove the UTF-8 clause from the language spec, so that str will have the same validity invariant as [u8].

[lcnr]

Would this mean that the following stops being UB?

use std::mem;

fn main() {
    // Being valid utf8 is still a safety invariant of `str`.
    // As any method using `str` may depend on this invariant,
    // it would still be UB to use `str::from_utf8_unchecked`
    // or `str::as_bytes` here. 
    let s: &str = unsafe { std::mem::transmute(b"\xff\xff" as &[u8]) };
    let bytes: &[u8] = unsafe { std::mem::transmute(s) };
    assert_eq!(bytes, &[0xff, 0xff][..]);
}

[RalfJung]

@lcnr yes exactly.

it would still be UB to use str::from_utf8_unchecked
or str::as_bytes here.

To be more precise, it would be library UB -- the way the methods are implemented right now, there is actually no language-level UB and nothing that Miri could possibly find, but the library is permitted to change in the future in ways that would make this language UB.

[Centril]

Dear community and language team.

As Ralf notes, there are no compelling reasons to keep this Undefined Behavior (UB) at the level of the abstract machine in terms of the validity invariant of str. Therefore, keeping it UB at this level only complicates the language definition instead with no notable benefits.

Instead, we can make this a library invariant, and leave it as "library UB" or "unspecified behavior". Indeed, this is probably what we always meant by the note in the reference. I hereby propose that we accept that new definition:

@rfcbot merge

[rfcbot]

Team member @Centril has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Centril
• @cramertj
• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[eddyb]

there is actually no language-level UB and nothing that Miri could possibly find

This leads to an interesting question: what can miri find?
My first guess would be some unsafe { unreachable_unchecked() } call in UTF-8 decoding.

It would be great to figure out if there is something like this that miri does detect, even if miri stops checking str values for UTF-8 validity altogether, and use it as a test case.

[RalfJung]

It would be great to figure out if there is something like this that miri does detect, even if miri stops checking str values for UTF-8 validity altogether, and use it as a test case.

Note that Miri does not check behind references, so while str would be checked, that type is basically unused, and &str is not checked.

[RalfJung]

This leads to an interesting question: what can miri find?
My first guess would be some unsafe { unreachable_unchecked() } call in UTF-8 decoding.

I scrolled over str/mod.rs to see how the invariant gets used. I certainly missed some things, but here is what stood out:

• Calling char::from_u32_unchecked, which must be a valid unicode codepoint. Miri checks this.
• The searching/splitting talks a lot about indices being at unicode boundaries. I do not know what happens if they are not.
• I suspect somewhere it might also lead to out-of-bounds accesses when it thinks there is a 4-byte character following, but only 2 bytes are left in the buffer.
• I did not find any unreachable/unreachable_unchecked.

[joshtriplett]

This seems entirely reasonable to me. If you never call any of str's functions, just storing non-UTF-8 in it shouldn't cause any issue.

I wonder if we might be able, in the future, to carefully exclude a few of str's functions from the "library UB" requirements.

cc @BurntSushi: Would this change potentially simplify bstr?