I want to use spark-redshift libraries for writing data from AWS S3 to AWS Redshift using the following code. Before using this, I would like to know whether spark-redshift libraries are open-source/free to use or it has to be licensed via Databricks.

val query="delete from emp where empno=7790"

//Write data to RedShift

mydf.coalesce(1).write.
format("com.databricks.spark.redshift")
.option("url",redShiftUrl)
.option("dbtable","emp")
.option("tempdir",s3dir)
.option("forward_spark_s3_credentials",true)
.option("preactions",query).
mode("append").
save()
1

There are 1 answers

0
jayrythium On

spark-redshift is a package maintained by Databricks, with community contributions from SwiftKey and other companies. It is free to use no license needed.