Issues in upgrading ML libraries using AWS Sagemaker Notebook's Lifecycle configurations

Question

I use the following script to automate upgrading of my libraries.

My script (Start Notebook):

#!/bin/bash

set -e

echo 'Before:'
echo $PATH

export PATH=/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:$PATH

echo 'After:'
echo $PATH

echo `pwd`

source /home/ec2-user/anaconda3/bin/activate tensorflow_p36

pip install pandas --upgrade

pip install tensorflow-gpu --upgrade

Error:

I get the following error, how can I point to the correct location(/home/ec2-user) of keras instead of /root

cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory 

Full Logs:

Before:
/sbin:/bin:/usr/sbin:/usr/bin
After:
/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/sbin:/bin:/usr/sbin:/usr/bin
/home/ec2-user
cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory

Without lifecycle configuration:

All the commands in the above script works.

Actual keras.json file is exising, under /home/ec2-user when I remove the lifecycle configuration with the following value.

sh-4.2$ cat .keras/keras.json
{
    "backend": "tensorflow"
}

Answer 1

Looks like pip install in your case executed "outside" of virtualenv

try to change from:

source /home/ec2-user/anaconda3/bin/activate tensorflow_p36

to:

source /home/ec2-user/anaconda3/bin/activate tensorflow_p36 && pip install pandas tensorflow-gpu --upgrade

and delete redundant lines