How to set the Jupyterlab notebooks default kernel with Papermill in prepare_only mode

Question

Is there a way to set the default kernel in JupyterLab in Azure ML notebooks, in prepare_only mode? I know it's only a few clicks away to change it on the top right - but I need to do this many times a day, and I often forget. Having the right default would be very convenient.

I saw this solution: Change Default kernel in jupyter notebook which suggest to add a line like

c.MultiKernelManager.default_kernel_name = 'py38' 

in the Jupyter config.

For JupyterLab in Azure ML, I can edit the Jupyter config file with:

sudo nano /home/azureuser/.jupyter/jupyter_notebook_config.py

But this has no effect. Update: /home/azureuser/.jupyter/jupyter_notebook_config.py and /home/azureuser/.jupyter/jupyter_server_config.py gets overwritten anyway every time I restart Jupyterlab or restart my compute...

We create those notebooks using papermill, so I tried to set the kernel explicitly like this:

papermill.execute_notebook(
    ... ,
    prepare_only=True,
    kernel_name='py38'
    )

But that had no effect either, when using prepare_only mode.

How can I set the kernel then? Any idea how else the default kernel could be set in prepare_only mode?

Answer 1

To set default kernel in Jupyter in Azure ML Studio, Follow the steps below:-

I created a new kernel with name py38 using the code below:-

pip install ipykernel
python -m ipykernel install --user --name py38
cd ~/.local/share/jupyter/kernels
Add the code below in kernel.json
{
    "argv": [
        "/anaconda/envs/py38/bin/python",
        "-m",
        "ipykernel_launcher",
        "-f",
        "{connection_file}"
    ],
    "display_name": "Python 3.8",
    "language": "python",
    "default": true
}

py38 kernel got created successfully like below:-

I created one jupyter_notebook_config.py by navigating to .jupyter directory, and typed this code. If you do not have jupyter directory in your Azure ML studio you can create a new one :-

Code 1 :-

mkdir .jupyter [if jupyter dir does not exist]
cd ~
cd  .jupyter  
echo  "c.KernelManager.default_kernel_name = 'py38'" > jupyter_notebook_config.py
sudo nano jupyter_notebook_config.py

Code 2:-

cd ~
cd .jupyter
ls
sudo nano jupyter_notebook_config.py
Add "c.KernelManager.default_kernel_name = 'py38'"  in the jupyter_notebook_config.py file 
or Add 
c.Spawner.default_url = '/lab?reset&kernel_name=py38'

I got permission denied error while creating the jupyter_notebook_config.py file with the code above like below:-

Error:-

bash: jupyter_notebook_config.py: Permission denied

In order to resolve this error, I ran the command below:-

sudo chown -R $USER:$USER ~/.jupyter

Output:- jupyter_notebook_config.py file got created with "c.KernelManager.default_kernel_name = 'py38'" config like below:-

Output of Code 2:-

Now, I set the Kernel to py38 and when I restarted the cluster or connected to jupyter notebook py38 was selected by default and did not change everytime, I wanted to connect to my Azure ML notebook. Refer below:-

py38 got selected by default after restart and whenever I wanted to connect to Jupyter notebook.

You can manually set the kernel in the notebook information as you are using Papermill in prepare only mode. You can accomplish this by including the following metadata in the notebook:

{
  "kernelspec": {
    "name": "py38",
    "display_name": "Python 3.8"
  }
}

The command shown below can be used to add this metadata to the notebook:

import papermill

input_path = "Untitled.ipynb"
output_path = "Untitled2.ipynb"

papermill.execute_notebook(
    input_path,
    output_path,
    prepare_only=True,
    kernel_name='py38',
    metadata={
        "kernelspec": {
            "name": "py38",
            "display_name": "Python 3.8"
        }
    }
)

Additionally, Papermill's prepare only mode only gets the notebook ready for execution, not actually runs it. As a result, using kernel name in execute notebook with prepare only=True will have no effect when manually setting the kernel.