LTO w/Graphite as kernel profile?

[preney]

These configurations are wonderful! Thanks for sharing!

I am wondering if the LTO w/Graphite configurations might also useful as a Gentoo kernel profile (possibly more than one inheriting from other profiles) so ease the burdens of trying out such. The profile(s) can stipulate some of the overrides, etc. required to have things work (e.g., Perl5 w/no LTO). Over time, presumably, the need for overrides would be eliminated, etc.

Others' thoughts on this (at least as an idea)?

[InBetweenNames]

This would indeed be very cool as a Gentoo profile--I wonder if profiles support "env" overrides as used in this Portage configuration? If so, it could be possible to make an "lto-overlay" for Gentoo that has a continuously updated LTO profile without Graphite and O3 (users can add those in after if they are interested).

[darkbasic]

The idea is super interesting but I'm not sure if profiles allows us to do all the things we need (env overrides, etc..).
I'm also interested about the Linux kernel, is it possible to build a kernel with LTO?

[InBetweenNames]

I actually have never had much luck myself with building the kernel with LTO, but I have enabled -O3 and Graphite on it before without issues. An LTOed kernel would be super nice however.

I just did an experiment on my system where I set up a profile in my local overlay with env overrides, and it looks like Portage didn't see them. We could patch portage to do this, but given the highly specific nature of this project, I'd understand if the main Gentoo devs didn't want to support that functionality. Perhaps we could achieve something close enough that wouldn't be too intrusive?

Say, we could create a new directory, /etc/portage/envs/lto for the LTO configs without any additional optimizations (O3, graphite, whatever the future may hold, etc). Then add a /etc/portage/package.env/lto.conf which is updated in sync with those.

This way users could easily add LTO to their existing systems and choose their own optimization options at will. Those could be checked out directly from this git repo. A small shell script could be used to keep the LTO settings in sync, using git stash functionality to preserve local modifications. What do you think?

[darkbasic]

Uhm... Not sure if it's worth the hassle: the only thing which needs to be kept in sync is ltoworkarounds.conf.

[InBetweenNames]

Something like:

./ltoize.py --install
./ltoize.py --update

Which would merge everything into your own /etc/portage/ in separate directories to avoid colliding with local settings. I'm hesitant to have such a script modify make.conf directly, but perhaps it could provide a snippet of code to replace your CFLAGS, CXXFLAGS, LDFLAGS, etc with that you could customize yourself. The only real requirement being that FLTO should be something of the form -flto[=n].

Put the usual stern warnings in, etc. I would hope that more users would take -O3 and Graphite for a spin, but even having LTO alone would be very useful.

[darkbasic]

It should show to the user which kind of modification it is going to make and ask for confirmation and possibly if he wants to edit before apply. Such a way I see no issue in touching make.conf.

[InBetweenNames]

Indeed... I just thought of this: make this LTOize script and add it to an overlay, say lto-overlay. Have the env/* stuff in there with ltoworkarounds.conf. The LTOize script would install symlinks to those places within your own /etc/portage. Then actually have the LTOize process happen when you emerge the ebuild, like so:

layman -a lto-overlay
emerge --sync
emerge sys-util/ltoize

This way you can use etc-update to review the changes made. Now, any time you emerge --sync, you will automatically get the latest settings, since /etc/portage/env/lto and /etc/portage/package.env/lto would be symlinks to the appropriate places in the overlay.

[darkbasic]

Sounds good.

[darkbasic]

It would be interesting to replicate Clear Linux optimizations too: https://www.phoronix.com/scan.php?page=news_item&px=Clear-Linux-NA2017-Opts

[mgomersbach]

I would recommend to put the symlink creation in a shim package instead of program.

1. It only needs to be run once.
2. From ebuild the change is tracked and undone after uninstall.

[mgomersbach]

Good idea BTW those symlinks ;P

[InBetweenNames]

Awesome, thanks for the suggestion! I should have time to work on this today. If it helps people try out LTO, I'm all for it.

@darkbasic Yes--Intel's approach is the right one IMO. The prospect of distccing a configuration that FMV is interesting--you could share the same binaries on your desktop as on your laptop for example with just some size overhead. Very cool idea. We actually go above and beyond ClearLinux since I don't think that it does LTO! We really are pushing the boundaries here. For users that just have one Gentoo system, they gain all of the benefits of the Intel approach with -march=native (or equivalent) since that turns on -mavx2, etc by default. Especially if you use -O3, like Intel does! In my Graphite flags, I deliberately enable -ftree-vectorize-distribution which helps with this stuff. GCC 8 will enable that by default at -O3.

One thing I think people will find is that when you crank GCC up like this, it actually generates code comparable to what Intel's ICC would generate--especially in numerical code. Very cool stuff!

[InBetweenNames]

The latest commit has paved the way for this symlinked-style configuration. I expect to be able to convert this entire repo to an overlay soon which users could just add with layman.

[InBetweenNames]

Just a question--which category should "ltoize" live in? I was thinking sys-apps, but I'm kind of leaning towards a new category sys-config because it isn't really an application, it's just a configuration modification.

[darkbasic]

Yes probably sys-config is better.