We have Azure HDI cluster ( linux worker nodes ) with primary storage account linked to ADLS gen2 storage. We use user manage identity (umi) to connect hdi cluster to it's primary storage. Everything works fine cluster successfully runs and creates some system data in a storage. We can see it. So umi authentication works fine in this context.

One thing I still can't clear, can I use the same umi to access the storage directly from within worker node:

  1. ssh worker_node
  2. hdfs dfs -ls abfss://$adls-gen2-url

Where $adls-gen2-url is an URL of ADLS gen2 storage.

My assumption, I cannot - because workers' linux OS has nothing to do with umi used to communicate cluster itself and ADLS gen2 storage. So we have to authenticate explicitly when runs commands directly from cluster nodes.

1 Answers


You can access the storage directly from the worker node.

Example: Default filesystem: hdfs dfs -ls abfss://[email protected]/

Another filesystem within the data lake storage gen2: hdfs dfs -ls abfss://[email protected]/

enter image description here

For more details, refer "Use the Azure Data Lake Storage Gen2 with Azure HDInsight".

Hope this helps.