404:Not Found error when running Jupyter Notebooks on Microsoft Azure HDInsight 3.5

Question

I have deployed a HDInsight 3.5 Spark (2.0) cluster on Microsoft Azure with the standard configurations (Location = US East, Head Nodes = D12 v2 (x2), Worker Nodes = D4 v2 (x4)). When the cluster is running, I click Jupyter Notebook in "Cluster Dashboards" quick link. After entering my username and password in the prompt, I receive a page giving me a 404: Not Found page. More specific the the following (json) message is shown:

{ "status": 404, "message": "Not Found" }

(This is in Google Chrome. In Internet page a similar, but differently looking, page 404 error page is shown.)

Cloud somebody help me with this error?

Answer 1

There's an undergoing issue preventing from Jupyter to work as expected out of the box on HDInsight at the moment. You can still use other deployment methods until this is resolved.

Edit: For example, you can run Jupyter locally and connect to the HDI cluster in Azure. See https://github.com/jupyter-incubator/sparkmagic/blob/master/README.md#installation and https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-jupyter-notebook-install-locally

Update: Issue was fixed

Answer 2

Running the following via SSH will solve the issue:

echo "c.NotebookApp.token = ''" | sudo tee --append /var/lib/.jupyter/jupyter_notebook_config.py > /dev/null
sudo pkill -f jupyter-notebook

It fixes the config file and then kills the jupyter process. Ambari will then take a minute or so to notice that Jupyter is no longer running and restart the service, which will pick up the new config.

These commands are being run on live clusters at the moment to fix the bug.