AWS, dotnet spark, and redshift are not working

94 views Asked by At

Hi I am having problems to get redshift and dotnet spark working:

This the configuration I use to get it working on debug mode:

C:\bin\spark-2.4.1-bin-hadoop2.7\bin\spark-submit.cmd `
            --jars spark-redshift_2.11-2.0.1.jar,aws-java-sdk-1.11.869.jar,aws-java-sdk-s3-1.11.832.jar,hadoop-aws-3.3.0.jar `
            --class org.apache.spark.deploy.dotnet.DotnetRunner `
            --master local `
            --packages org.apache.spark:spark-avro_2.11:2.4.0 `
            microsoft-spark-2.4.x-0.12.1.jar `
            debug

And here is my code excerpt:

// Create a Spark session.
            SparkSession spark = SparkSession
                .Builder()
                .AppName("DataScraping ETL into Redshift")
                //.Config("spark.hadoop.fs.s3.access.key", awsAccessKey)
                //.Config("spark.hadoop.fs.s3.awsSecretAccessKey", awsSecretKey)
                .GetOrCreate();

            var productState = spark.Read()
                //.Format("com.databricks.spark.redshift")
                //.Jdbc(url, table, new Dictionary<string, string>
                //{
                //    ["temporary_aws_access_key_id"] = awsAccessKey,
                //    ["temporary_aws_secret_access_key"] = awsSecretKey,
                //    ["aws_iam_role"] = iam_role,
                //    ["tempdir"] = s3Bucket,
                //    ["jdbcdriver"] = "com.amazon.redshift.jdbc42.Driver"
                //})
                //.Count();
                //.Format("jdbc")
                .Format("com.databricks.spark.redshift")
                //.Format("com.amazon.redshift.jdbc42.Driver")
                .Option("temporary_aws_access_key_id", awsAccessKey)
                .Option("temporary_aws_secret_access_key", awsSecretKey)
                .Option("url", url)
                .Option("dbtable", "product_track")
                .Option("aws_iam_role", iam_role)
                .Option("tempdir", s3Bucket)
                .Option("spark.hadoop.fs.s3.access.key", awsAccessKey)
                .Option("spark.hadoop.fs.s3.awsSecretAccessKey", awsSecretKey)
                .Option("jdbcdriver", "com.amazon.redshift.jdbc42.Driver")
                .Load();

            productState.Show(10);

Note in the coments the lot of configurations I have proved:

There is not way I have this working. The exception is always the same:

java.io.IOException: No FileSystem for scheme: s3
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at com.databricks.spark.redshift.Utils$.assertThatFileSystemIsNotS3BlockFileSystem(Utils.scala:156)
    at com.databricks.spark.redshift.RedshiftRelation.<init>(RedshiftRelation.scala:52)
    at com.databricks.spark.redshift.DefaultSource.createRelation(DefaultSource.scala:49)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:145)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.handleBackendRequest(DotnetBackendHandler.scala:85)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:28)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:23)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
    at java.lang.Thread.run(Thread.java:748)

Please, hhhhheeeeeeeellllllllppppppppp!!!!

0

There are 0 answers