Ansible play hung freezes when python interpreter not found

Question

This issue is reproducible if anyone wants to try.

Ansible hangs when target host "10.9.88.205" does not have python installed.

I have successful ssh connectivity from ansible to target host shown below below:

ansible-host# ssh [email protected]

Below is my playbook:

- name: Play 2- Configure Source nodes
  hosts: all
  user: root
  ignore_errors: yes
  gather_facts: false
  tasks:

   - name: Get HTTPD userid on server
     raw: id mwweb || id webadm || ls -ld /web

   - name: Get OHS userid on server
     raw: id mwweb

Below is how i run my playbook

[ansibleuser@ansiblehost checkssh]$ ansible-playbook -i 10.9.88.205, -f 5 testpython.yml -vvvv
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/ansibleuser/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed 10.9.88.205, inventory source with host_list plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: testpython.yml ***********************************************************************************************************************************************
Positional arguments: testpython.yml
become_method: sudo
inventory: (u'10.9.88.205,',)
forks: 5
tags: (u'all',)
verbosity: 4
connection: smart
timeout: 10
1 plays in testpython.yml

PLAY [Play 2- Configure Source nodes] **********************************************************************************************************************************
META: ran handlers

TASK [Get HTTPD userid on server] **************************************************************************************************************************************
task path: /app/axmw/Ansible/playbook/checkssh/testpython.yml:24
<10.9.88.205> ESTABLISH SSH CONNECTION FOR USER: root
<10.9.88.205> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/app/ssh_keys/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ControlPath=/home/ansibleuser/.ansible/cp/6abdc12511 -tt 10.9.88.205 'id mwweb || id webadm || ls -ld /web'

It just hangs after printing TASK [Get HTTPD userid on server] and never completes.

This behavior is experienced with all IPs where python is not installed and is reachable through ssh.

Update: Below investigation is post debug suggestion by user @larsks

Below is my ansible version.

ansible --version
ansible 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

I then tried the ssh command from the ansible output manually and it also freezes. See output of ssh below:

[ansibleuser@ansiblehostcheckssh]$ ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/app/ssh_keys/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ControlPath=/home/ansibleuser/.ansible/cp/6abdc12511 -tt 10.9.88.205 'id mwweb || id webadm || ls -ld /web'
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 70199
debug3: mux_client_request_session: session request sent

<Freezes Hangs after this>

I further investigated and figured out if i remove -o ControlPath=/home/ansibleuser/.ansible/cp/6abdc12511 -tt from the ssh that ansible constructed the ssh works fine and does not hang.

Thus, the below manually constructed ssh works fine.

ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/app/ssh_keys/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no 10.9.88.205 'id mwweb || id webadm || ls -ld /web'

Can you please suggest how can i get ansible not to hang and proceed to the next task i.e

Get OHS userid on server?