Process Substitution in Ansible for Path-based Parameters

Question

Many Ansible modules are designed to accept file paths as a parameter, the but lack the possibility to supply the contents of the file directly. In cases where the input data actually comes from something other than a file, this forces one to create a temporary file somewhere on disk, write the intended parameter value into it and then supply the path of this temporary file to the Ansible module.

For illustration purposes a real life example: the java_cert Ansible module takes the parameter pkcs12_path for the path to a PKCS12 keystore containing a keypair to be imported into a given Java keystore. Now say for example this data is retrieved through a Vault lookup, so in order to be able to supply the module with the path it demands, we must write the Vault lookup result into a temporary file, use the file's path as the parameter and then handle the secure deletion of the temporary file, seeing as the data is likely confidential.

When a situation such as this arises within the context of Shell/bash scripting, namely a command line tool's flag only supporting interaction with a file, the magic of process substitution (e.g. --file=<(echo $FILE_CONTENTS)) allows for the tool's input and output data to be linked with other commands by transparently providing a named pipe that acts as if it were a (mostly) normal file on disk.

Within Ansible, is there any comparable mechanism to replace file-based parameters with more flexible constructs that allow for the usage of data from variables or other commands? If there is no built-in method to achieve this, are there maybe 3rd-party solutions that allow for it, or that simplify workflows like the one I described? For example something like a custom lookup plugin which is supplied with the file content data and then handles, transparently and in the background, the file management (i.e. creation, writing the data, and ultimately deletion) and provides the temporary path as its return value, without the user necessarily ever having to know it.

Exemplary usage of such a plugin could be:

    ...
    pkcs_path: "{{ lookup('as_file', '-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY----- ') }}"
    ...

with the plugin then creating a file under e.g. /tmp/as_file.sg7N3bX containing the textual key from the second parameter and returning this file path as the lookup result. I am however unsure how exactly the continued management of the file (especially the timely deletion of sensitive data) could be realized in such a context.

Answer 1

Disclaimer:

• I am (obviously!) the author of the below collection which was created as a reaction to the above question
• The lookup plugin was not thoroughly tested and might fail with particular modules.

---

Since this was a pretty good idea and nothing existed, I decided to give it a try. This all ended up in a collection now called thoteam.var_as_file which is available in a github repo. I won't paste all files in this answer as they are all available in the mentioned repo with a full README documentation to install, test and use.

The global idea was the following:

• Create a lookup plugin responsible for pushing new temporary files with a given content and returning a path to use them.
• Clean-up the created files at the end of playbook run. For this step, I created a callback plugin which launches the cleanup action listening to v2_playbook_on_stats events.

I still have some concerns about concurrency (files yet to be cleaned are stored in a static json file on disk) and reliability (not sure that the stats stage happens in all situation, especially on crashes). I'm also not entirely sure using a callback for this is a good practice / best choice.

Meanwhile this was quite fun to code and it does the job. I will see if this work is used by other and might very well enhance all this in the next weeks (and if you have PRs to fix the already know issues, I'm happy to accept them).

Once installed and the callback plugin enabled (see https://github.com/ansible-ThoTeam/thoteam.var_as_file#installing-the-collection), the lookup can be used anywhere to get a file path containing the passed content. For example:

- name: Get a filename with the given content for later use
  ansible.builtin.set_fact:
    my_tmp_file: "{{ lookup('thoteam.var_as_file.var_as_file', some_variable) }}"
    
- name: Use in place in a module where a file is mandatory and you have the content in a var
  community.general.java_cert:
    pkcs12_path: "{{ lookup('thoteam.var_as_file.var_as_file', pkcs12_store_from_vault) }}"
    cert_alias: default
    keystore_path: /path/to/my/keystore.jks
    keystore_pass: changeit
    keystore_create: yes
    state: present

These are the relevant parts of the two plugin files. I removed the ansible documentation vars (for conciseness) which you can find in the git repo directly if your wish.

plugins/lookup/var_as_file.py

from ansible.errors import AnsibleError
from ansible.plugins.lookup import LookupBase
from ansible.module_utils.common.text.converters import to_native
from ansible_collections.thoteam.var_as_file.plugins.module_utils.var_as_file import VAR_AS_FILE_TRACK_FILE
from hashlib import sha256
import tempfile
import json
import os

def _hash_content(content):
    """
    Returns the hex digest of the sha256 sum of content
    """
    return sha256(content.encode()).hexdigest()

class LookupModule(LookupBase):

    created_files = dict()

    def _load_created(self):
        if os.path.exists(VAR_AS_FILE_TRACK_FILE):
            with open(VAR_AS_FILE_TRACK_FILE, 'r') as jfp:
                self.created_files = json.load(jfp)

    def _store_created(self):
        """
        serialize the created files as json in tracking file
        """

        with open(VAR_AS_FILE_TRACK_FILE, 'w') as jfp:
            json.dump(self.created_files, jfp)

    def run(self, terms, variables=None, **kwargs):

        '''
        terms contains the content to be written to the temporary file
        '''
        try:
            self._load_created()

            ret = []
            for content in terms:
                content_sig = _hash_content(content)
                file_exists = False

                # Check if file was already create for this content and check it.
                if content_sig in self.created_files:
                    if os.path.exists(self.created_files[content_sig]):
                        with open(self.created_files[content_sig], 'r') as efh:
                            if content_sig == _hash_content(efh.read()):
                                file_exists = True
                                ret.append(self.created_files[content_sig])
                            else:
                                os.remove(self.created_files[content_sig])

                # Create / Replace the file
                if not file_exists:
                    temp_handle, temp_path = tempfile.mkstemp(text=True)
                    with os.fdopen(temp_handle, 'a') as temp_file:
                        temp_file.write(content)
                        self.created_files[content_sig] = temp_path
                        ret.append(temp_path)

            self._store_created()

            return ret

        except Exception as e:
            raise AnsibleError(to_native(repr(e)))

plugins/callback/clean_var_as_file.py

from ansible.plugins.callback import CallbackBase
from ansible_collections.thoteam.var_as_file.plugins.module_utils.var_as_file import VAR_AS_FILE_TRACK_FILE
from ansible.module_utils.common.text.converters import to_native
from ansible.errors import AnsibleError
import os
import json

def _make_clean():
    """Clean all files listed in VAR_AS_FILE_TRACK_FILE"""
    try:
        with open(VAR_AS_FILE_TRACK_FILE, 'r') as jfp:
            files = json.load(jfp)
            for f in files.values():
                os.remove(f)
        os.remove(VAR_AS_FILE_TRACK_FILE)
    except Exception as e:
        raise AnsibleError(to_native(repr(e)))

class CallbackModule(CallbackBase):
    ''' This Ansible callback plugin cleans-up files created by the thoteam.var_as_file.var_as_file lookup '''
    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = 'utility'
    CALLBACK_NAME = 'thoteam.var_as_file.clean_var_as_file'

    CALLBACK_NEEDS_WHITELIST = False
    # This one doesn't work for a collection plugin
    # Needs to be enabled anyway in ansible.cfg callbacks_enabled option
    CALLBACK_NEEDS_ENABLED = False

    def v2_playbook_on_stats(self, stats):
        _make_clean()

I'll be happy to get any feedback if your give it a try.