I'm having some trouble encoding data when some columns that are of type Option[Seq[String]] are missing from our data source. Ideally I would like the missing column data to be filled with None
.
Scenario:
We have some parquet files that we are reading in that have column1 but not column2.
We load the data in from these parquet files into a Dataset
, and cast it as MyType
.
case class MyType(column1: Option[String], column2: Option[Seq[String]])
sqlContext.read.parquet("dataSource.parquet").as[MyType]
org.apache.spark.sql.AnalysisException: cannot resolve '
column2
' given input columns: [column1];
Is there a way to create the Dataset with column2 data as None
?
In simple cases you can provide an initial schema which is a superset of expected schemas. For example in your case:
This approach can be a little bit fragile so in general you should rather use SQL literals to fill the blanks: