I am reading data from a parquet file, where new columns might get added so want use schema evolution with my existing schema.
The below code does not work
spark.readStream.format('cloudFiles')
.option('cloudFiles.format', 'parquet')
.option('cloudFiles.inferColumnTypes', 'true')
.option('cloudFiles.schemaLocation', schema_path)
.option('cloudFiles.schemaEvolutionMode', 'addNewColumns')
.load(input_path))
But these two code work fine:
spark.readStream.format('cloudFiles')
.option('cloudFiles.format', 'parquet')
.option('cloudFiles.inferColumnTypes', 'true')
.option('cloudFiles.schemaEvolutionMode', 'addNewColumns')
.load(input_path))
spark.readStream.format('cloudFiles')
.option('cloudFiles.format', 'parquet')
.option('cloudFiles.inferColumnTypes', 'true')
.option('cloudFiles.schemaLocation', schema_path)
.load(input_path))