Ansible Inventory dynamic Host dependency

Question

I want to run a specific command on an ansible group. However this command (device upgrade) makes this device unavailable, also it makes child devices unavailable as well.

Is it somehow possible in Ansible to define a depedency list, ie. device topology and run commands in that order, and more importantly, do not fail because another parent device become unavailable?

Practically: R1 -> SW1 -> SW2 : Upgrade should start from SW2, and then SW1. R1 can't be updated until at least SW1 is up, because uplink is received from SW1, so it should wait for SW1.

Ps.: I have read all other threads on this, I need to do this on multiple group of devices, so creating specific playbooks with hard coded groups is not that ideal.

Answer 1

There are at least two relatively simple ways to achieve that. One would look a bit kludgy but it's both fast and safe, another is more elegant, but contains some caveats.

The most straightforward one would be to organize your inventory so that the hierarchy is reflected in the group names. If we consider the deepness of the dependency limited, we can simply have a number of plays that corresponds to the maximum deepness (see the note on the hardcoding below). If we define level_1 as the highest one (R1 in your example), our inventory will be "open-ended" so we can refine as many levels as we want. We can also skip some levels and/or make them empty. Of course, you can use more meaningful names instead of level numbers:

# inventory.yaml
---
devices:
  vars:
    ansible_python_interpreter: auto_silent
    ansible_connection: local
  children:
    level_1:
      hosts:
        R[1:3]:
    level_2:
      hosts:
        SW[1:15]:
    level_3:
    level_4:
    level_5:
      hosts:
        SW[20:30]:

# playbook.yaml
---
- name: Upgrade the devices of level 6
  hosts: level_6
  gather_facts: false # just for testing
  tasks:
    - name: Upgrading the device
      debug:
        var: inventory_hostname

# other levels go here

- name: Upgrade the devices of level 1
  hosts: level_1
  gather_facts: false # just for testing
  tasks:
    - name: Upgrade the device
      debug:
        var: inventory_hostname

Before commenting on hardcoding: yes, but this makes perfect sense as Ansible runs the tasks within the group in parallel, and the plays are executed sequentially. If the hierarchy is defined and it's limited, I'd recommend this solution.

However, we can try a different approach. Ansible offers several options to order the execution based on inventory. Note that while there is an option to use the order in which the hosts are defined, it's not guaranteed in general. It usually works for simple cases, though, but using sorted or reverse_sorted order that relies on host names in more reliable. So, instead of defing generic levels, we can define the sets of the dependent devices:

# inventory.yaml
---
devices:
  vars:
    ansible_python_interpreter: auto_silent
    ansible_connection: local
  children:
    set_1:
      hosts:
        R1:
        SW[1:15]:
        SW[25:30]:
    set_2:
      hosts:
        R2:
        SW[1:4]:
        SW[20:24]:
    set_3:
      hosts:
        R3:
        SW[150:155]:
        SW[50:60]:
        SW[75:80]:

# playbook.yaml
---
- name: Upgrade the devices
  hosts: devices
  gather_facts: false # just for testing
  order: reverse_inventory
  tasks:
    - name: Upgrading the device
      debug:
        var: inventory_hostname

# ...

- name: Upgrade the devices of level 1
  hosts: level_1
  gather_facts: false # just for testing
  tasks:
    - name: Upgrading the device
      debug:
        var: inventory_hostname

Of course, these examples are only intended to show the options of ordering, and in real life wait_for will most likely be required. Besides, you should be aware of some nuances:

• when running the separate plays on the separate groups you're safe;
• in case of specifying the order, you should remember that Ansible still runs the tasks against the batches of hosts in parallel. The default batch size is 5. If the defined batch size (let's assume 10) is greater than the number of the hosts in a pseudo group (e.g. 6 for SW[150:155]), Ansible will add some of the subsequent hosts to the batch so the "dependency" order may be violated.

An optimal solution could be a combination of these approaches, where the inventory would contain a number of upper-level groups to control the devices like SW2, SW3 and so on in parallel, and the bottom-level would be upgrades one by one.

In theory, it is also possible to define the hierarchy level as a group/host variable as well. But one will need to implement an extremely complex logic to control the execution flow, and the built-in parallelism features could be (and most likely - will be) lost. It's not possible to define the batch size as an inventory variable and we can't add the plays to the playbook dynamically. serial can be set from hostvars but since we don't have the inventory_hostname at that point it's useless in this case, not to say we have to calculate the batch size first. I tried to implement it and I wouldn't recommend to repeat the attempt :)