I understand you can use a batch custom activity with custom VM image with preloaded python libraries to run your python programs through ADF.
Can we also use a plain linux VM image and load the python libraries as part of the execution of the custom activity? I understand this might increase the execution time since libraries have to be installed each time, but I want to find out how it is done. Explanation/example is appreciated.
Thanks!
You can use Azure Batch start task to install pre defined libraries efficiently. All jobs submitted via Custom Activity against the same pool will have access to them.
Alternatively, you can install dependent libraries by executing a shell scripts against Linux pool nodes which would setup the libraries and then trigger the python code.