Stabilize unit tests with non-`()` return type

[nikomatsakis]

CURRENT STATUS:

Awaiting someone to write the stabilization PR! Mentoring instructions here.

---

This is a sub-issue of the larger tracking issue devoted to the ? in main RFC. This issue corresponds to stabilizing the use of unit tests whose return type is something other than () -- it basically an extension of #48453, which was discussing the same thing but for the main function.

What would be stabilized

As before, unit tests that return ():

• Succeed unless they panic
• Can be annotated with #[should_panic]

However, unit tests can now have other return types:

• The return type must implement the Termination trait
  • Test and expected error
• The test passes if the return value is considered a "success" (e.g., an Ok from Result)
  • Test
• If the type is not (), then #[should_panic] is disallowed
  • Test and expected error

Unknowns

• Question: what's up with #[should_panic] and #[bench] anyway? (Example)
• Can we use this with rustdoc yet?

Older proposal

Possible changes needed before stabilizing

• Adjust libtest runner to distinguish #[should_panic] from #[should_error], as suggested by @scottmcm?
• Successful examples of using with rustdoc

What would be stabilized

Unit tests (and #[bench] functions) would join the main function in being allowed to return any value that implements the Termination trait.

This commits us to the following (each link is to a test):

• tests that return Result and other Termination types. In the case of Result:
  • Ok means test passes
  • Err causes test failure
  • more generally, success or failure is determined by invoking report(), although that detail is not being stabilized
• benchmarks work the same way (Ok, Err)
  • note that #[bench] is still effectively unstable though in general

What remains unstable

• The library details: where the trait is and its methods
  • Which methods the test runner invokes on the trait -- this is unstable as the trait itself is unstable

[nikomatsakis]

@rfcbot fcp merge

I propose that we further stabilize unit tests etc that return non-() return types.

[rfcbot]

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged teams:

• @aturon
• @cramertj
• @eddyb
• @nikomatsakis
• @nrc
• @pnkfelix
• @withoutboats

Concerns:

• experience (Stabilize unit tests with non-() return type #48854 (comment))

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[scottmcm]

Thanks for including the test links! Super helpful.

I think letting Ok(()) (and ExitCode::SUCCESS 🙂) be test pass is great.

This one surprised me, though:

#[test]
#[should_panic]
fn not_a_num() -> Result<(), ParseIntError> {
    let _: u32 = "abc".parse()?;
    Ok(())
}

I get cognitive dissonance from that, because I my brain says "what do you mean parse should panic? I thought the point was that it doesn't panic!", and I think it would be unfortunate if the test were to continue to pass should it actually start panicking.

I think I was expecting

• (unannotated): test pass is .report()==SUCCESS, everything else is test fail
• [should_panic]: catch_panic caught a panic is test pass, everything else is test fail
• [should_error] (hypothetical): test pass is .report()!=SUCCESS, everything else is test fail

(Unimportant thought: a -> ! test seems fundamentally useless. But probably not worth prohibiting?)

[cramertj]

Unimportant thought: a -> ! test seems fundamentally useless. But probably not worth prohibiting?

#[test]
#[should_panic]
fn panic_panics() -> ! { panic!() }

😛

[sgrif]

I think this definitely warrants eventually adding an alias for should_panic (perhaps should_error or should_fail), but that seems like it's strictly orthogonal to stabilizing this

[aturon]

cc @jonhoo re: test frameworks

[scottmcm]

@sgrif Note that I'm explicitly not suggesting that it should be an alias: I would like should_panic to only result in a test pass if there was an actual panic, which is something that I don't think we'd want to change after stabilization if there were a bunch of tests using should_panic to check for Err.

[sgrif]

@scottmcm Fair point. I agree.

[jonhoo]

In the context of custom test frameworks, this seems fine. The path we're taking going forward is that what ships with Rust by default is the same thing that libtest does at the time of the change, so committing to this now just means that the libtest we build using ctf will also have to support it (which seems fine).

Note that with ctfs, new test frameworks can implement entirely new testing annotations, and are not bound to the annotations, signatures, and semantics as the current built-in test support.

Also cc @Manishearth .