When developing Ansible plugins, how should you use module_utils imports?

Question

When developing custom Ansible modules in an Ansible collection, you often want to share code between multiple modules.

If I have the following directory structure in my collection:

collection/
  ...
  plugins/
    modules/
      module_one.py
      module_two.py
    module_utils/
      utils.py
  ...

What is the correct way to import the shared code between all the different modules in your collection/plugins/modules directory? In this case module_one.py and module_two.py.

Should I use the short path?

from ansible.module_utils.utils import helper_function

Or the full path?

from ansible.module_utils.namespace.collection.plugins.module_utils.utils import helper_function

Using and developing module utilities says the following:

The ansible.module_utils namespace is not a plain Python package: it is constructed dynamically for each task invocation, by extracting imports and resolving those matching the namespace against a search path derived from the active configuration.

To reduce the maintenance burden in a collection or in local modules, you can extract duplicated code into one or more module utilities and import them into your modules. For example, if you have your own custom modules that import a my_shared_code library, you can place that into a ./module_utils/my_shared_code.py file like this:

from ansible.module_utils.my_shared_code import MySharedCodeClient

When you run ansible-playbook, Ansible will merge any files in your local module_utils directories into the ansible.module_utils namespace in the order defined by the Ansible search path.

This seems to imply that you use the short version above.

Looking at popular Open Source module code, like autoscaling_group.py, I see the following

from ansible_collections.amazon.aws.plugins.module_utils.botocore import is_boto3_error_code

This is using the long form.

Additionally, both seem to fail when testing your code directly via python based on the information at Verifying your module code locally

python plugins/modules/module_one.py /tmp/args.json
Traceback (most recent call last):
  File ".../ansible/collections/namespace/collection/plugins/modules/module_one.py", line 69, in <module>
    from ansible.module_utils.utils import helper_function
ModuleNotFoundError: No module named 'ansible.module_utils.utils'

• What is the correct format to import shared code?
• What is the correct way to test Ansible modules directly using Python?

Answer 1

One way to get around the problem of executing your modules directly via Python is to symlink your code to the directory Python is looking in.

For example if I have an import like

from ansible_collections.namespace.collection.plugins.module_utils.utils import helper_function

and I'm using a virtual environment in my project at ./venv with Python 3.10, you can execute from the root of your collection directory:

# Create directory
mkdir \
  --parents \
  $(pwd)/venv/lib/python3.10/site-packages/ansible_collections/namespace/collection

# Create symlink
ln \
  --symbolic \
  $(pwd)/plugins \
  $(pwd)/venv/lib/python3.10/site-packages/ansible_collections/namespace/collection/

Then imports will find your module_utils directory.

This doesn't feel like the ideal approach for solving this issue, though.