I want to run Python script in OOzie workflow with special libraries. But, I want to run this script without installing these special libraries to Hadoop nodes. I tried to run with virtualenv but failed. Does anyone know how to run this Python script without installing any libraries in server-side?
Thanks :)
why not run the job as a pyspark job and pass the required dependencies via "
--py-files" parameter? A little annoying but doesn't require any special libraries.You said "No installed libraries" so this is not what you asked for... but If you have vitrualenv installed on all nodes, then you can use pyspark to deploy the environment. If you combine that with an Oozie spark job you'd have what I think you where looking to do.