Databricks 'schemaEvolutionMode' and 'schemaLocation' does not work together

121 views Asked by At

I am reading data from a parquet file, where new columns might get added so want use schema evolution with my existing schema.

The below code does not work

spark.readStream.format('cloudFiles')
            .option('cloudFiles.format', 'parquet')
            .option('cloudFiles.inferColumnTypes', 'true')
            .option('cloudFiles.schemaLocation', schema_path)
            .option('cloudFiles.schemaEvolutionMode', 'addNewColumns')
            .load(input_path))

But these two code work fine:

spark.readStream.format('cloudFiles')
            .option('cloudFiles.format', 'parquet')
            .option('cloudFiles.inferColumnTypes', 'true')
            .option('cloudFiles.schemaEvolutionMode', 'addNewColumns')
            .load(input_path))
spark.readStream.format('cloudFiles')
            .option('cloudFiles.format', 'parquet')
            .option('cloudFiles.inferColumnTypes', 'true')
            .option('cloudFiles.schemaLocation', schema_path)
            .load(input_path))
0

There are 0 answers