How can I install python modules for spark on BigInsights on cloud enterprise

Question

I need to install python modules such as nltk so that they are available to spark on yarn on BigInsights 4.2 Enterprise.

How can I install python modules, e.g. Using pip?

Note that the current python version is 2.6.6 and root access is not available to uses on BigInsights on cloud.

Answer 1

See this answer: https://stackoverflow.com/a/41206890/1033422

It shows how Anaconda python can be installed and made available to pyspark jobs running on yarn. It also shows how python modules can be installed using the pip available with anaconda.