rustup refuses server certificate on openSUSE Tumbleweed

[frgomes]

Problem

rustup refuses the server certificate on openSUSE Tumbleweed.

• firewall is disabled on openSUSE.
• no proxy configuration or any other MITM software.
• curl and openssl accept the server certificate as it should be.
• rustup works fine on a Debian 11 box connected to the same router.
• rustup works fine on a openSUSE instance running on WSL2.

Steps

$ uname -a
Linux mars 5.14.11-2-default #1 SMP Sun Oct 10 08:34:34 UTC 2021 (834dddd) x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/os-release 
NAME="openSUSE Tumbleweed"
# VERSION="20211019"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20211019"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20211019"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"

$ ./rustup -V
rustup 1.24.3 (ce5817a94 2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.53.0 (53cb7b09b 2021-06-17)`
(j11s300) [2021-10-22 00:50:24] rgomes@mars:~/.cargo/bin$ ./rustup -v check
verbose: read metadata version: '12'
verbose: creating temp file: /home/rgomes/.rustup/tmp/5zy78gtql715nkgz_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with reqwest
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' to '/home/rgomes/.rustup/tmp/5zy78gtql715nkgz_file'

Caused by:
    0: failed to make network request
    1: error sending request for url (https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get local issuer certificate)
    2: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get local issuer certificate)
    3: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get local issuer certificate)
    4: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:

curl and openssl accept the server certifcate on this very same machine and terminal session:

$ curl -s https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256
2f57b4a3b6b3463387774cc16bec64167c5d16e94fa1f4de7524c46194a38f6b  channel-rust-stable.toml

$ echo | openssl s_client -servername rust-lang.org -connect static.rust-lang.org:443 2>/dev/null | openssl x509 -noout -issuer -subject -dates
issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
subject=CN = rust-lang.org
notBefore=Jul 24 00:00:00 2021 GMT
notAfter=Aug 22 23:59:59 2022 GMT

I would expect a successful output, like it happens on my Debian 11 box:

$ ./rustup -v check
verbose: read metadata version: '12'
verbose: creating temp file: /home/rgomes/.rustup/tmp/yix3x9ddke03zoxs_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with reqwest
verbose: deleted temp file: /home/rgomes/.rustup/tmp/yix3x9ddke03zoxs_file
stable-x86_64-unknown-linux-gnu - Up to date : 1.56.0 (09c42c458 2021-10-18)
rustup - Up to date : 1.24.3

Possible Solution(s)

Nothing listed below produced any benefit:

1. I've reinstalled all packages on this openSUSE box, and performed a dist-upgrade;
2. I've removed all 32bit packages;
3. I've installed packages which could possibly contribute to mitigate the problem, namely:

$ zypper se -i cert
Loading repository data...
Reading installed packages...

S  | Name                    | Summary                                               | Type
---+-------------------------+-------------------------------------------------------+--------
i+ | ca-certificates         | Utilities for system wide CA certificate installation | package
i+ | ca-certificates-cacert  | CAcert root certificates                              | package
i+ | ca-certificates-mozilla | CA certificates for OpenSSL                           | package
i+ | mozilla-nss-certs       | CA certificates for NSS                               | package
i+ | python38-certifi        | Python package for providing Mozilla's CA Bundle      | package

4. I've updated ca-certs:

$ sudo update-ca-certificates -f -v
running /usr/lib/ca-certificates/update.d/50java.run ..
creating ///var/lib/ca-certificates/java-cacerts ...
running /usr/lib/ca-certificates/update.d/70openssl.run ..
creating ///var/lib/ca-certificates/openssl ...
running /usr/lib/ca-certificates/update.d/80etc_ssl.run ..
running /usr/lib/ca-certificates/update.d/99certbundle.run ..
creating ///var/lib/ca-certificates/ca-bundle.pem ...

i've found this ///var/lib/ca-certificates above suspicious.

Notes

Output of rustup --version:

rustup 1.24.3 (ce5817a94 2021-05-31)

Output of rustup show:

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/rgomes/.rustup

no active toolchain

[facklambda]

Is that local time accurate in the uname -a output accurate?

I'm curious if there's something in the reqwest invocation or whichever TLS backend being used that's not utilizing the local issuer certificates.

What happens if you repeat this process with the environment variable SSL_CERT_FILE="///var/lib/ca-certificates/ca-bundle.pem" set explicitly?

[frgomes]

I was able to install Rust on openSUSE running under WSL. But the problem persists on my regular openSUSE box.

I've compared installed system packages between this openSUSE instance running under WSL2 with my regular openSUSE box.

zypper se -i ssl
zypper se -i tls
zypper se -i ssh
zypper se -i cert

I've then uninstalled several packages which accumulated along the years from my regular openSUSE box, making sure that I try to install Rust once again for every batch of packages which are uninstalled. Even after cleaning packages to the point that both machines look similar... even after that... I'm still not able to run rustup properly on my regular box.

This approach was insufficient for mitigating the issue.

[frgomes]

I've discovered that my system had accumulated a relative large number of package repositories and it has a lot of stuff which probably is colliding to each other in multiple ways. I've decided for reinstalling openSUSE.

Hours later...

After reinstalling openSUSE from scratch I was able to install Rust as usual.

I'll leave this issue open so that you guys decide whether rustup should behave like curl and openssl behave... or maybe not, which translates to a decision involving bug or feature.

Note: I've observed that, in Debian 11, looking for packages containing openssl plus dev, well... this returns several package names. I cannot say that such fact leads to any conclusion, but I would like to share the fact, just in case:

$ apt search openssl | fgrep dev | fgrep openssl | fgrep rust
librust-cargo+openssl-dev/stable 0.43.1-4 amd64
librust-curl+openssl-probe-dev/stable 0.4.33-1 amd64
librust-curl+openssl-sys-dev/stable 0.4.33-1 amd64
librust-curl-sys+openssl-sys-dev/stable 0.4.36-1 amd64
librust-git2+openssl-probe-dev/stable 0.13.11-2 amd64
librust-git2+openssl-sys-dev/stable 0.13.11-2 amd64
librust-openssl-dev/stable 0.10.29-1 amd64
librust-openssl-probe-dev/stable,now 0.1.2-1+b1 amd64 [installed]
librust-openssl-sys-dev/stable 0.9.55-2 amd64

I don't see similar packages on openSUSE, though... at least not after reinstalling the system, which resets the repositories to default settings.

[frgomes]

@facklambda :: I've seen other issues mentioning that date/time must be accurate, NDP client must be active, etc. I can guarantee that my computers are all configured to obtain date/time from NDP servers.

I believe that the problem was related to certificates being obtained from wrong locations. Unfortunately, I could not wait for this issue to be solved, I mean: I had to reinstall openSUSE from scratch and move on, which means that we don't have a computer anymore for exploring whatever went wrong.

For your information, in my brand new openSUSE installation, the command sudo update-ca-certificates -v gives the very same output. However, this time, rustup -v check works properly.

[facklambda]

@frgomes interesting, I wonder if the freshly installed rustup is still using the reqwest backend.

[frgomes]

@facklambda : It is, apparently.

$ rustup -v check
verbose: read metadata version: '12'
verbose: creating temp file: /home/rgomes/.rustup/tmp/c5pvsrolbn7q91pz_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with reqwest
verbose: deleted temp file: /home/rgomes/.rustup/tmp/c5pvsrolbn7q91pz_file
stable-x86_64-unknown-linux-gnu - Up to date : 1.56.0 (09c42c458 2021-10-18)
rustup - Up to date : 1.24.3

[kinnison]

With no ability to investigate (due to perfectly reasonable reinstallation by OP) and no ability to reproduce when I tried, I'm going to have to close this. If someone else has this problem in the future, we should open a fresh issue and work through as much careful detail as possible.