Supporting Elasticsearch 8.0 and its secure by default mode

[spinscale]

With the release of Elasticsearch 8.0 a major change has been done to provide a secure implementation by default - namely enabling TLS by default as well as requiring user/password authentication. As this is the supposed mode of operation in production I think it would be great to provide the same experience within integration tests.

There are a couple of caveats though and before doing any PRs I'd appreciate some guidance. So, let's get started with what works and how.

I have a test up and running that initializes like this

private static final ElasticsearchContainer container =
            new ElasticsearchContainer("docker.elastic.co/elasticsearch/elasticsearch:8.0.0")
                    .withExposedPorts(9200)
                    .withPassword("s3cret");

This is the container and HTTP client setup

      // this is required because the wait strategy uses HttpsURLConnection and does not yet
      // know about the certificate to be copied out of the container, see below
      HttpsURLConnection.setDefaultSSLSocketFactory(SslUtils.trustAllContext().getSocketFactory());

      // remove from environment to have TLS enabled
      container.getEnvMap().remove("xpack.security.enabled");

      // custom wait strategy for tls and basic auth
      container.setWaitStrategy(new HttpWaitStrategy()
              .forPort(9200)
              .usingTls()
              .forStatusCode(HTTP_OK)
              .forStatusCode(HTTP_UNAUTHORIZED)
              .withBasicCredentials("elastic", "secr3t")
              .withStartupTimeout(Duration.ofMinutes(2)));
      container.start();

The Elasticsearch Client setup looks like this and requires extracting the self signed certificate out of the docker container

        // extract the ca cert from the running instance
        byte[] certAsBytes = container.copyFileFromContainer("/usr/share/elasticsearch/config/certs/http_ca.crt", InputStream::readAllBytes);

        HttpHost host = new HttpHost("localhost", container.getMappedPort(9200), "https");
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("elastic", "s3cret"));
        final RestClientBuilder builder = RestClient.builder(host);

        builder.setHttpClientConfigCallback(clientBuilder -> {
            clientBuilder.setSSLContext(SslUtils.createContextFromCaCert(certAsBytes));
            clientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            return clientBuilder;
        });

Let's cover the issues with this and how to resolve it:

First, ElasticsearchContainer.withPassword() sets xpack.security.enabled - this should probably be done with a specific method, as I need to explicitely remove it from the environment to make sure TLS is enabled. I think it makes to have proper methods to enable/disable TLS plus some checks if this is running against an 8.0 release in order to keep BWC compatibility?

Second, the wait strategy uses HttpsURLConnection, which at the point of its first run, has no access on the generated cert of the docker container. I do not consider a trust all cert manager a valid solution here, so the question is, whether it makes sense to have a custom wait strategy that extracts the cert from the docker container as well, and cleans up the UrlConnection to use the default ssl socket factory after the test. This would still mean that there might be issue with parallel running tests that may be setting this as well. Maybe you have a better idea to work around that.

Any help and guidance much appreciated!

[kiview]

Hey, thanks for bringing this to our attention @spinscale.

In some cases, we found that LogMessageWaitStrategy can provide the best experience with the least amount of workarounds, it might be the case here as well. Of course, it has some drawbacks, like tightly coupling against the logging side-effect rather than to the actual container behavior, but often this is a pragmatic tradeoff.

Why does disabling xpack.security enable TLS? This is a bit confusing to me.

Since users need the self-signed cert, I think it makes sense to provide an API for extracting it and document the usage.

[spinscale]

Fantastic, I will take a look at the LogMessageWaitStrategy then as well as adapting the ElasticContainer class then!

Regarding xpack.security. This is not about disabling, but not setting at, as the default options mean that TLS is enabled (that default is the major change in 8.0). The moment you set options manually, you have to set all of them. Hope that clears it up.

Thanks for your help!