I have a HDP cluster as well as the Azure Storage Emulator installed on my machine and would like to access the Azure Storage Emulator from my hadoop job. Obviously this is not default HDFS for my Hadoop cluster. I know that if I had HDInsight emulator I was able to access that because it automatically set the configuration. I wonder to know what configurations I need to set in my hadoop cluster in order to access the azure storage emulator?
When I run 'hdfs dfs -ls wasb://127.0.0.1:10000/' I get the following error message
org.apache.hadoop.fs.azure.AzureException: Unable to access container $root in account 127.0.0.1:10000 using anonymous credentials, and no credentials found for them in the configuration.
Z.
You need use a different wasb:// syntax for the storage emulator. See the following for details:
https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/#addressing
NOTE:
The syntax for addressing the files on the storage emulator (running on HDInsight emulator) is wasb://@storageemulator.
Regards, Jason