COPY INTO command is not working in command line terminal

Question

I want to incrementally load data into a delta table. However, my "COPY INTO" command is not working in spark sql prompt of command line terminal.

The same command is working in databricks.

Please let me know does this command only works in databricks environment.

Answer 1

The COPY INTO is a SQL statement specific to Databricks (docs), you can't use it with the open source Spark. But basically it's just a simplification of the following streaming code:

df = spark.readStream.format("parquet").load(input_path)
df = df.select(....) # optionally transform the input data
df.writeStream.format("delta")\ 
  .trigger(availableNow=True) \
  .option("checkpointLocation", "some location") \
  .start(output_path)