gax conflict dependency

[lboudard]

Hi,

I'm trying to use google cloud storage client along with apache beam/dataflow.
Though I'm having some runtime dependency conflicts problems with java-gax library instanciating gcs client.

Caused by: java.lang.ClassNotFoundException: com.google.api.gax.retrying.ExceptionRetryAlgorithm
    at java.net.URLClassLoader.findClass (URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1 (Native Method)
    at java.lang.ClassLoader.defineClass (ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass (SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass (URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100 (URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run (URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run (URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged (Native Method)
    at java.net.URLClassLoader.findClass (URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:357)
    at com.google.cloud.BaseService.<clinit> (BaseService.java:48)
    at com.google.cloud.storage.StorageOptions$DefaultStorageFactory.create (StorageOptions.java:44)
    at com.google.cloud.storage.StorageOptions$DefaultStorageFactory.create (StorageOptions.java:38)
    at com.google.cloud.ServiceOptions.getService (ServiceOptions.java:426)

Below the code calling storage service

Storage storage = StorageOptions.getDefaultInstance().getService();

I've tried adding multiple versions of java-gax, excluding it from apache beam dependencies.

    <dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
      <version>${beam.version}</version>
      <exclusions>
        <exclusion>
          <groupId>com.google.api</groupId>
          <artifactId>gax</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
....
   <dependency>
      <groupId>com.google.api</groupId>
      <artifactId>gax</artifactId>
      <version>1.21.0</version>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-storage</artifactId>
      <version>1.24.0</version>
      <exclusions>
        <exclusion>
          <groupId>com.google.cloud</groupId>
          <artifactId>google-cloud-core</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.api</groupId>
          <artifactId>gax-httpjson</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.auth</groupId>
          <artifactId>google-auth-library-credentials</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

But still having those dependencies issues :(.

Below dependency tree:

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ word-count-beam ---
[INFO] org.example:word-count-beam:jar:0.1
[INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.4.0:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
[INFO] |  +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.8.9:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.9:compile
[INFO] |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  \- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.4.0:compile
[INFO] |  |  +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
[INFO] |  |  +- com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0:compile
[INFO] |  |  \- com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.4.0:compile
[INFO] |  +- io.grpc:grpc-core:jar:1.2.0:compile
[INFO] |  |  +- com.google.errorprone:error_prone_annotations:jar:2.0.11:compile
[INFO] |  |  +- io.grpc:grpc-context:jar:1.2.0:compile
[INFO] |  |  \- com.google.instrumentation:instrumentation-api:jar:0.3.0:compile
[INFO] |  +- com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0:compile
[INFO] |  +- com.google.api:gax-grpc:jar:0.20.0:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:1.2.0:compile
[INFO] |  |  \- com.google.auto.value:auto-value:jar:1.2:compile
[INFO] |  +- com.google.cloud:google-cloud-core-grpc:jar:1.2.0:compile
[INFO] |  |  \- com.google.protobuf:protobuf-java-util:jar:3.2.0:compile
[INFO] |  |     \- com.google.code.gson:gson:jar:2.7:compile
[INFO] |  +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
[INFO] |  +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18:compile
[INFO] |  |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.18:compile
[INFO] |  +- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
[INFO] |  |  +- com.google.api-client:google-api-client-java6:jar:1.20.0:compile
[INFO] |  |  +- com.google.api-client:google-api-client-jackson2:jar:1.20.0:compile
[INFO] |  |  \- com.google.oauth-client:google-oauth-client-java6:jar:1.20.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
[INFO] |  |  \- com.google.http-client:google-http-client-jackson:jar:1.20.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
[INFO] |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:0.1.0:compile
[INFO] |  +- io.grpc:grpc-auth:jar:1.2.0:compile
[INFO] |  +- io.grpc:grpc-netty:jar:1.2.0:compile
[INFO] |  |  +- io.netty:netty-codec-http2:jar:4.1.8.Final:compile (version selected from constraint [4.1.8.Final,4.1.8.Final])
[INFO] |  |  |  \- io.netty:netty-codec-http:jar:4.1.8.Final:compile
[INFO] |  |  \- io.netty:netty-handler-proxy:jar:4.1.8.Final:compile
[INFO] |  |     \- io.netty:netty-codec-socks:jar:4.1.8.Final:compile
[INFO] |  +- io.netty:netty-handler:jar:4.1.8.Final:compile
[INFO] |  |  +- io.netty:netty-buffer:jar:4.1.8.Final:compile
[INFO] |  |  |  \- io.netty:netty-common:jar:4.1.8.Final:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.8.Final:compile
[INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.8.Final:compile
[INFO] |  |  \- io.netty:netty-codec:jar:4.1.8.Final:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.2.0:compile
[INFO] |  +- io.grpc:grpc-all:jar:1.2.0:compile
[INFO] |  |  +- io.grpc:grpc-okhttp:jar:1.2.0:compile
[INFO] |  |  |  +- com.squareup.okhttp:okhttp:jar:2.5.0:compile
[INFO] |  |  |  \- com.squareup.okio:okio:jar:1.6.0:compile
[INFO] |  |  +- io.grpc:grpc-protobuf-lite:jar:1.2.0:compile
[INFO] |  |  \- io.grpc:grpc-protobuf-nano:jar:1.2.0:compile
[INFO] |  |     \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
[INFO] |  |  \- org.json:json:jar:20160810:compile
[INFO] |  +- com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b:compile
[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11:compile
[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
[INFO] |  |  \- com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11:compile
[INFO] |  |     \- com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11:compile
[INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile
[INFO] |  +- com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0:compile
[INFO] |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  +- com.google.auth:google-auth-library-appengine:jar:0.7.0:compile
[INFO] |  |  +- io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0:compile
[INFO] |  |  \- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.22.0:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:0.7.1:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:0.7.1:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:3.2.0:compile
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9:compile
[INFO] |  \- com.google.api.grpc:proto-google-common-protos:jar:0.1.9:compile
[INFO] +- org.apache.beam:beam-examples-java:jar:2.4.0:compile
[INFO] |  \- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
[INFO] +- com.google.api-client:google-api-client:jar:1.22.0:compile
[INFO] +- com.google.http-client:google-http-client:jar:1.22.0:compile
[INFO] |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO] |     \- commons-codec:commons-codec:jar:1.3:compile
[INFO] +- joda-time:joda-time:jar:2.4:compile
[INFO] +- com.google.guava:guava:jar:20.0:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- org.slf4j:slf4j-jdk14:jar:1.7.25:runtime
[INFO] +- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] +- org.hamcrest:hamcrest-library:jar:1.3:compile
[INFO] +- junit:junit:jar:4.12:compile
[INFO] +- org.apache.beam:beam-runners-direct-java:jar:2.4.0:runtime
[INFO] |  \- org.apache.beam:beam-runners-local-java-core:jar:2.4.0:runtime
[INFO] +- org.mockito:mockito-core:jar:1.9.5:test
[INFO] |  \- org.objenesis:objenesis:jar:1.0:test
[INFO] +- com.google.api:gax:jar:1.21.0:compile
[INFO] |  +- org.threeten:threetenbp:jar:1.3.3:compile
[INFO] |  \- com.google.api:api-common:jar:1.5.0:compile
[INFO] +- com.google.cloud:google-cloud-storage:jar:1.24.0:compile
[INFO] |  \- com.google.cloud:google-cloud-core-http:jar:1.24.0:compile
[INFO] |     +- com.google.http-client:google-http-client-appengine:jar:1.23.0:compile
[INFO] |     +- io.opencensus:opencensus-api:jar:0.11.1:compile
[INFO] |     \- io.opencensus:opencensus-contrib-http-util:jar:0.11.1:compile
[INFO] \- org.ahocorasick:ahocorasick:jar:0.4.0:compile

Could you help us with this issue?

Thanks!

[andreamlin]

Hello,

The stack trace says the ClassNotFoundException is thrown in BaseService.java, which is in the google-cloud-core package. Your dependency tree output has the google-cloud-core dep at version
com.google.cloud:google-cloud-core:jar:1.0.2.

Would you be able to update it to the latest, at 1.24.0?