Automate Python application installation

Question

I have a Java application that should interact with Python agents installed on remote machines. My main application has a SSH access to these remote machines. To make user's live simpler, I want to automate process of agent installation, so user can just push button and main application connects, install and run application remotely.

I expect that:

• remotes will be different UNIX for now (Windows then);
• SSH user may not be root;
• Python is installed on the remote machine;
• Remote machine may not have an Internet connection.

My Python agents use virtualenv and pip, have number of dependencies. I've made a script to download virtualenv and dependencies as tar.gz / zip:

#!/usr/bin/env bash

# Should be the same in dist.sh and install_dist.sh.
VIRTUALENV_VERSION=13.0.3
VIRTUALENV_BASE_URL=https://pypi.python.org/packages/source/v/virtualenv

echo "Recreating dist directory..."

if [ -d "dist" ]; then
  rm -rf dist
fi
mkdir -p dist

echo "Copying agent sources..."

cp -r src dist
cp requirements.txt dist
cp agent.sh dist
cp install_dist.sh dist


echo "Downloading virtualenv..."

curl -o dist/virtualenv-$VIRTUALENV_VERSION.tar.gz -O $VIRTUALENV_BASE_URL/virtualenv-$VIRTUALENV_VERSION.tar.gz

echo "Downloading requirements..."

mkdir dist/libs
./env/bin/pip install --download="dist/libs" --no-binary :all: -r requirements.txt

echo "Packing dist directory..."
tar cvzf dist.tar.gz dist

When installation starts, my main app scp all this archive to the remote machine, installs virtualenv and requirements, copy all required scripts:

#!/usr/bin/env bash

# Runs remotely.

SCRIPT_DIR="`dirname \"$0\"`"
SCRIPT_DIR="`( cd \"$SCRIPT_DIR\" && pwd )`"
HOME_DIR="`( cd \"$SCRIPT_DIR/..\" && pwd )`"

VIRTUALENV_VERSION=13.0.3

echo "Installing virtualenv..."

tar xzf $SCRIPT_DIR/virtualenv-$VIRTUALENV_VERSION.tar.gz --directory $SCRIPT_DIR
python $SCRIPT_DIR/virtualenv-$VIRTUALENV_VERSION/virtualenv.py $HOME_DIR/env
if [ $? -ne 0 ]
then
    echo "Unable to create virtualenv."
    exit 1
fi

$HOME_DIR/env/bin/pip install $SCRIPT_DIR/virtualenv-$VIRTUALENV_VERSION.tar.gz
if [ $? -ne 0 ]
then
    echo "Unable to install virtualenv."
    exit 1
fi

echo "Installing requirements..."

$HOME_DIR/env/bin/pip install --no-index --find-links="$SCRIPT_DIR/libs" -r $SCRIPT_DIR/requirements.txt
if [ $? -ne 0 ]
then
    echo "Unable to install requirements."
    exit 1
fi

echo "Copying agent sources..."

cp -r $SCRIPT_DIR/src $HOME_DIR
if [ $? -ne 0 ]
then
    echo "Unable to copy agent sources."
    exit 1
fi
cp -r $SCRIPT_DIR/agent.sh $HOME_DIR
if [ $? -ne 0 ]
then
    echo "Unable to copy agent.sh."
    exit 1
fi
cp -r $SCRIPT_DIR/requirements.txt $HOME_DIR

echo "Cleaning installation files..."
rm -rf $HOME_DIR/dist.tar.gz
rm -rf $SCRIPT_DIR

I faced a problem with building some of the dependencies remotely - for example, they may require gcc or other libraries that should be installed with sudo manually... may be, I should use pre-compiled wheels where possible, providing them per each target system? Or may be you see some better way to implement this task?

Answer 1

Maybe deliver fully packed applications ? Have a look at Freeze and py2exe.

Also, if you planned on delivering for windows environment, notice that compiling require a lots and is quite annoying, prefer sending libraries pre-compiled with your app. For linux environment, well, it mostly depend of the environment so you probably gonna stick with building depedencies.

Note: From your comment:

providing distribution for each target system requires building them on each target system

Notice that you can do cross-compilation which allow you do build for different system (even Windows) without the need to have a running machine with this environment.
I would go ahead with building with cross compile. Worst case scenario is very little hosts will be annoying and won't be able to use your binaries. In this scenario just solve the problem case by case.

Best luck :)