I use databricks connect v11 and try to use it now for v13. So I do the follwing:
pip install -U "databricks-connect==13.3.3"
Then I run
echo "y
https://adb-xxx.7.azuredatabricks.net
xxxx-2
yyyy
zzz
15001" | databricks-connect configure
Then I run
python3 -m pytest whereby one file uses SparkSession.builder.getOrCreate() which results in the following error:
RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.
EDIT: I also run:
export SPARK_REMOTE="sc://adb-xxxx.7.azuredatabricks.net:15001/;token=xxx-2;x-databricks-clus
ter-id=xxx"
which didn't work either.
In the home directory a valid .databricks-connect file is existing
Obviously, some config step is missing which was not necessary for v11