I am seeing this error in my Dataproc job when I am attempting to write to my Spanner table using JDBC. Weird thing is that when I try to read from the same table via the Spanner Connector(https://github.com/GoogleCloudDataproc/spark-spanner-connector) in the same job, it is working. So its not a permissions issue. I am forced to use JDBC for writes as the Spanner connector does not support it. Even though the error makes it look like a an authorization issue, I feel it is something else. Since the Spanner Dataproc connector is indeed working. But the JDBC based connection is not.
My Compute Engine default service account has the following roles.
BigQuery Data Owner, BigQuery Job User ,Cloud Spanner Database Admin ,Compute OS Login, Dataproc Editor, Dataproc Worker ,Service Account User Storage Admin.
I am using Dataproc serverless version 2.1 and Spanner jdbc jar version 2.14.2 I have tried everything from 2.4 to no avail. Although, the errors seem to vary with the jar version. I am building a fatJar and I even tried to store the jdbc jar in a GCS bucket and added the jdbc jar as a runtime dependency in the gcloud submit command. Nothing works.
Caused by: com.google.cloud.spanner.SpannerException: UNKNOWN: io.grpc.netty.NettyChannelProvider: Unsupported credential type: io.grpc.netty.shaded.io.grpc.netty.NettyChannelCredentials; io.grpc.netty.UdsNettyChannelProvider: does not support 1 or more of [class java.net.InetSocketAddress]
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:291)
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:297)
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:61)
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:181)
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:110)
at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:484)
at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:271)
at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create(SpannerOptions.java:475)
at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create(SpannerOptions.java:470)
at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:602)
at com.google.cloud.spanner.SpannerOptions.getSpannerRpcV1(SpannerOptions.java:1417)
at com.google.cloud.spanner.SpannerImpl.<init>(SpannerImpl.java:133)
at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create(SpannerOptions.java:465)
at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create(SpannerOptions.java:460)
at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:582)
at com.google.cloud.spanner.connection.SpannerPool.createSpanner(SpannerPool.java:361)
at com.google.cloud.spanner.connection.SpannerPool.getSpanner(SpannerPool.java:302)
at com.google.cloud.spanner.connection.ConnectionImpl.<init>(ConnectionImpl.java:254)
at com.google.cloud.spanner.connection.ConnectionOptions.getConnection(ConnectionOptions.java:1141)
at com.google.cloud.spanner.jdbc.AbstractJdbcConnection.<init>(AbstractJdbcConnection.java:63)
at com.google.cloud.spanner.jdbc.JdbcConnection.<init>(JdbcConnection.java:67)
at com.google.cloud.spanner.jdbc.JdbcDriver.connect(JdbcDriver.java:196)
... 48 more
Caused by: io.grpc.ManagedChannelRegistry$ProviderNotFoundException: io.grpc.netty.NettyChannelProvider: Unsupported credential type: io.grpc.netty.shaded.io.grpc.netty.NettyChannelCredentials; io.grpc.netty.UdsNettyChannelProvider: does not support 1 or more of [class java.net.InetSocketAddress]
at io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:206)
at io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:157)
at io.grpc.Grpc.newChannelBuilder(Grpc.java:101)
at io.grpc.Grpc.newChannelBuilderForAddress(Grpc.java:111)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:387)
at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:107)
at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:85)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:243)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:237)
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:226)
at com.google.cloud.spanner.v1.stub.GrpcSpannerStub.create(GrpcSpannerStub.java:243)
at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:370)```
Here is my code for reference
// JDBC WRITE --> DOES NOT WORK
String spannerConnectionUrl = String.format(
"jdbc:cloudspanner:/projects/%s/instances/%s/databases/%s?lenient=true",
spannerProjectId, spannerInstanceId, spannerDatabase
);
log.info("Spanner connection url {}",spannerConnectionUrl);
testDF.write()
.option("dbtable", "test_activity")
.format("jdbc")
.option("driver", "com.google.cloud.spanner.jdbc.JdbcDriver")
.option("url", spannerConnectionUrl)
.mode("overwrite")
.save();
// SPANNER CONNECTOR READ --> WORKS
// Read from Cloud Spanner using t
Dataset<Row> testDF = spark.read()
.format("cloud-spanner")
.option("table", "test_activity")
.option("instanceId", spannerInstanceId)
.option("databaseId", spannerDatabase)
.option("projectId", spannerProjectId)
.load()
.filter(col("id").equalTo(id));