We needed to move to databricks-connect>13.x. Now I facing the issue that when I work with a nested dataframe of the structure
root
|-- a: string (nullable = true)
|-- b: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- c: string (nullable = true)
| | |-- f: struct (nullable = true)
| | | |-- g: string (nullable = true)
| | | |-- h: struct (nullable = true)
| | | | |-- supplier_id: string (nullable = true)
| | |-- i: string (nullable = true)
| | |-- j: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- k: string (nullable = true)
| | | | |-- l: array (nullable = true)
| | | | | |-- element: struct (containsNull = true)
| | | | | | |-- m: string (nullable = true)
| | | | | | |-- n: array (nullable = true)
| | | | | | | |-- element: struct (containsNull = true)
| | | | | | | | |-- o: string (nullable = true)
| | | | | | |-- p: array (nullable = true)
| | | | | | | |-- element: struct (containsNull = true)
| | | | | | | | |-- q: string (nullable = true)
| | |-- r: boolean (nullable = true)
|-- s: integer (nullable = true)
|-- t: string (nullable = true)
I get the error on the cluster (13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12):
Caused by: grpc_shaded.com.google.protobuf.InvalidProtocolBufferException: Protocol message had too many levels of nesting. May be malicious. Use CodedInputStream.setRecursionLimit() to increase the depth limit.