I have a delta table that is structured as follows:
id | col1 | .... | colN
"...." implies that there is a large number of columns
A JSON is stored in Storage Account every day. I can read from that STAC using a Volume. The JSON is structured as follows, for example:
[
{ "id": 1, "col_name": "col3", value: "update_me" },
{ "id": 3, "col_name": "col6", value: "update_me_too" }
]
A JSON like this can potentially have up to 100 entries in the array
The delta table must be changed using this JSON. For example, the first entry in the example above states that the entry in col3 must be changed to "update_me" for id 1. The second entry states that for id 3 col6 must be changed to "update_me_to".
The changes should be executed daily in a batch job.
I'm racking my brain as to how I could solve this reasonably efficiently, but have no idea...
I did try to iterate over the JSON entries and .update the Delta Table, but I could not get the syntax right and it was very slow.