Install hadoopy in google composer

Question

I am using google composer.

How can we install hadoopy in google composer environment.

This page has steps for installing hadoopy in Linux machine

Github Clone

git clone https://github.com/bwhite/hadoopy.git
cd hadoopy
sudo python setup.py install

PIP installation

sudo pip install -e git+https://github.com/bwhite/hadoopy#egg=hadoopy

I am not getting how to install this in Google Composer Environment

Answer 1

The package seems to be available on Python Package Index. So you can install it via the web UI (your composer environment > Pypi packages tab > edit > add <your_dependency>==<version>). You can also do that via command-line :

gcloud composer environments update ENVIRONMENT-NAME \
    --update-pypi-packages-from-file requirements.txt \
    --location LOCATION

requirements.txt should then contain something like :

hadoopy==0.6.0

Note that installing the dependency can be quite long (Composer needs to rebuild Docker images with the included package).

If you want to add the dependency directly from sources on Github, you'll have to follow the steps described in the docs : Installing a local Python library.