Activating a Conda environment in Ansible playbook

Question

I am trying to run a list of tasks (here running airflow but it could be anything really) that require to be executed in a existing Conda environment.

I would like to do these tasks:

- name: activate conda environment
 # does not work, just for the sake of understanding
 command: source activate my_conda_env

- name: initialize the database
  command: airflow initdb

- name: start the web server
  command: 'airflow webserver -p {{ airflow_webserver_port }}'

- name: start the scheduler
  command: airflow scheduler

Of course, this does not work as each task is independent and the conda environment activation in the first task is ignored by the following tasks.

I guess the issue would be the same if using a python virtualenv instead of conda.

How can I achieve each task being run in the Conda environment?

Answer 1

Each of your commands will be executed in a different process.

source command, on the other hand, is used for reading the environment variables into the current process only (and its children), so it will apply only to the activate conda environment task.

What you can try to do is:

- name: initialize the database
  shell: source /full/path/to/conda/activate my_conda_env && airflow initdb
  args:
    executable: /bin/bash

- name: start the web server
  shell: 'source /full/path/to/conda/activate my_conda_env && airflow webserver -p {{ airflow_webserver_port }}'
  args:
    executable: /bin/bash

- name: start the scheduler
  shell: source /full/path/to/conda/activate my_conda_env && airflow scheduler
  args:
    executable: /bin/bash

Before, check what's the full path to activate on the target machine with which activate (you need to do it before any environment is sourced). If Conda was installed in a user's space, you should use the same user for the Ansible connection.

Answer 2

Was looking out for something similar. Found a neater solution than having multiple actions:

- name: Run commands in conda environment
  shell: source activate my_conda_env && airflow {{ item }}
  with_items:
    - initdb
    - webserver -p {{ airflow_webserver_port }}
    - scheduler