I'm looking for, with no success, how to read a Azure Synapse table from a SQL-Pool of another workspace using Scala Spark (since it is apparently the only option). I found in https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/synapse-spark-sql-pool-import-export#use-pyspark-with-the-connector very useful samples but not for my scenario.
I would like to know if I can add as option a linked service to the following query?
%%spark
import com.microsoft.spark.sqlanalytics.utils.Constants
import org.apache.spark.sql.SqlAnalyticsConnector._
val df = spark.read.option(Constants.SERVER, "<SYNAPSE_WORKSPACE_NAME>.sql.azuresynapse.net").synapsesql("<DATABASE_NAME>.<DB_SCHEMA>.<DB_TABLE_NAME>")
Something like this:
.option(Constants.LINKED_SERVICE, "LINKED_SERVICE_NAME")
I couldnt find any documentation about this.
My problem is that I'm using private networks, both synapse workspaces are connected trough a private endpoint, they work well since I can connect to the SQL-Pool using a pipeline. My goal is to achieve the same using a Spark Notebook.
For some reason, I'm getting the error:
Error: com.microsoft.spark.sqlanalytics.exception.SQLAnalyticsConnectorException: com.microsoft.sqlserver.jdbc.SQLServerException: Data exfiltration to 'DATALAKE_NAME.dfs.core.windows.net' is blocked. Add destination to allowed list for data exfiltration and try again.
but I dont understand why it's telling me that "data exfiltration to DATALAKE is blocked? Im trying to reach the SQL-Pool, not the datalake, in any case I just created a private endpoint to the datalake too and still it's not working.
Does anyone know if it is possible?