how to handle import of files using Jinja that don't exist

Question

In Salt Pillars I need to load the contents of a file into a Jinja variable. If the file doesn't exist Jinja throws an error (TemplateNotFound:...)

I have some folders (Company A, Company B, etc) in my Pillars that may or may not contain a file called Default.sls. If the file exists - it will be loaded into a variable. The problem arises when the file doesn't exist - in this case Jinja throws a TemplateNotFound exception. My Pillar folder looks something like this:

pillar
|
|-Default.sls
|
|-Company A
| |-Default.sls  <-- This could be missing
| |-init.sls
| |-prod.sls
| |-test.sls
| |-etc...
|
|-Company B
| |-Default.sls  <-- This could be missing
| |-init.sls
| |-prod.sls
| |-test.sls
| |-etc...
|
|-Company C
|...

I have not found a good way to solve this. I have been using the following code (info: tpldir contains the name of the current foldername the sls file (in this case init.sls) is located in):

{{Company x}}/init.sls
----------------------
{% from tpldir + '/default.sls' import companyDefaultX with context %}
xx:
  companyDefault: {{ companyDefaultX | yaml }}

In order to check if the file exists - I have been using:

salt['file.file_exists']('/srv/pillar/' + tpldir + "/default.sls")

This works fine as long as file_roots defined in Salt master config is setup as default. We have now moved to using gitfs - now the above use of file.file_exists obviously does not work since the files no longer is located on the salt master in /srv/pillar/...!

The reason for the above code is that it is a way of making a Tiering between default- and companyDefault configurations. First default (Defalt.sls) is read, then {{Company X}}/Default.sls is merged into/onto default. Thirdly environment (prod.sls, test.sls, etc) is merged with the new default...

Bonus question: How can I loop over files in a folder without using:

salt['file.readdir']('/srv/pillar/' + tpldir)

As this is the same problem (/srv/pillar)...

Answer 1

I have found a solution to the above. Only way is to use include in combination with "ignore missing":

{%- load_yaml as customerDefaultX %}
{% include tpldir + '/default.sls' ignore missing with context %}
{%- endload %}

The customerDefaultX variable will now contain the content of the {{Company X}}/default.sls file OR None if it doesn't exist...

Original bonus question is still open though ;-)

Answer 2

A hack that works for me is in the use of cp.cache_(dir|file) Salt function. Within .sls file(note that it has to be SLS to get slspath or tpldir working) I can use then:

{%- set include_dir = opts['id'] | replace('.', '_') %}
{%- if salt['cp.cache_dir']("salt://{0}/{1}".format(slspath, include_dir)) %}
include:
  - .{{ include_dir }}
{%- endif %}

On a minion that function will try to ensure that the given salt://.. path is expandable and that the (dir|file) is downloaded and cached locally. On a success that returns the actual location of the cached entity, but we are interested only in the confirmation that such an identity does exist.