Build released compiler artifacts as optimized as possible

[michaelwoerister]

At the moment the compiler binaries that we release are not as fast and optimized as they could be. As of ff227c4, they are built with multiple codegen units and ThinLTO again, which makes the compiler around 10% slower than when built with a single CGU per crate. We really should be able to do better here, especially for stable releases:

• At least, the compiler should be built with -Ccodegen-units=1 for stable releases.
• In the medium term, the compiler might gain support for profile-guided optimization (see Add basic PGO support. #48346). Once it is available, we should use it for making the compiler itself faster. (see also symbol ordering: Use section/symbol ordering files for compiling rustc (e.g. BOLT) #50655)
• We don't use full LTO for compiling the compiler, mainly because we don't support it for Rust dylibs. We should review if this restriction is still current, and, if we can lift it, enable full LTO.

@rust-lang/release @rust-lang/infra, how can we decouple builds of stable releases from the regular CI builds that are timing out so much lately. There should be a way of doing these builds without the severe time limits that we have in regular CI.

[alexcrichton]

To make sure we set expectations, the 10% number of perf is not 10% slower, it's "executes 10% more instructions". The change in the number of instructions is often an indicator that there could be a regression but it does not translate to a 10% slowdown in literal wall time. For example the wall-time measurements for that commit shows the worst regression, percentage-wise, as 0.49s to 0.56s. Large benchmarks like servo-style-opt got at worse 3.8% slower in a clean build from scratch, going from 75 to 78 seconds.

I mean to point this out in terms of reducing the number of codegen units or PGO or those sorts of optimizations aren't really silver bullets. They're incredibly expensive optimizations for a few seconds here and there as opposed to major optimizations across the board.

[SimonSapin]

@alexcrichton thanks for clarifying.

[michaelwoerister]

@alexcrichton Yes, I know that this won't make the compiler massively faster. On the other hand, it's not uncommon that we spend weeks of developer time on getting a 5% compile time improvement. If there's the opportunity of making the compiler 10% faster by letting a build machine chew on it for a few hours every six weeks, I think we should take it.

That being said, I don't underestimate the complexity of our CI. I just don't want us to disregard the opportunity from the beginning. Maybe there is a simpler solution that would get us 90% of the way.

[ishitatsuyuki]

Moving to opt-level=3 can speed up up to 2%, but it's blocked on a Windows codegen bug. See also: #48204.

[michaelwoerister]

@andjo403's comments on gitter have given me the idea that we could also try to build LLVM with PGO. I realize of course that this would require lots of new infrastructure support and isn't something that can be implemented quickly.

[michaelwoerister]

Some updates here:

• In an heroic effort, @alexcrichton and @kennytm are working on switching the compiler's C++ code to be built with Clang 6.0 (Compile LLVM with Clang on release builders #50200) which promises to speed up the compiler by a few percent.
• Using Clang will open up the possibility to use linker-based ThinLTO, which does not seem to have problems with Rust dylibs. This should give another few percent in compiler performance.
• Another option for making the compiler faster is optimizing the order in which sections/symbols are emitted into object files (Chrome does this and Firefox might soon do it too).

[michaelwoerister]

I opened a separate issue for symbol ordering: #50655