We have Azure HDI cluster ( linux worker nodes ) with primary storage account linked to ADLS gen2 storage. We use user manage identity (umi) to connect hdi cluster to it's primary storage. Everything works fine cluster successfully runs and creates some system data in a storage. We can see it. So umi authentication works fine in this context.
One thing I still can't clear, can I use the same umi to access the storage directly from within worker node:
- ssh worker_node
hdfs dfs -ls abfss://$adls-gen2-url
$adls-gen2-url is an URL of ADLS gen2 storage.
My assumption, I cannot - because workers' linux OS has nothing to do with umi used to communicate cluster itself and ADLS gen2 storage. So we have to authenticate explicitly when runs commands directly from cluster nodes.