I have two separate Kubernetes clusters that will be used for a staging and a production environment, respectively. I want to have YAML manifests for the Kubernetes API objects I will be submitting to each cluster, but some of the objects will have slightly different configurations between the two environments.
As a made up but illustrative example, imagine running an internal Docker registry on each cluster, one using S3 as the storage back end and one using the GCS back end. The registry container can accept these configuration values as environment variables or read from a file, both of which Kubernetes manifests support, but how should I populate these values for each environment?
Essentially what I want is a way to have a manifest that looks something like this, where the $() syntax is variable interpolation that would happen on the server when the manifest is submitted:
---
apiVersion: v1
kind: Pod
metadata:
name: foo
spec:
containers:
- name: foo
image: foo
env:
- name: bar
value: $(etcdctl get /path/to/bar)
I could write templates that use placeholders for the values and then process the template, pulling in real values from some external source, to produce the final manifest that is actually submitted to the cluster. However, I thought I'd ask first in case a tool that does this already exists, or there is some Kubernetes-blessed way of doing this that I'm not aware of. It certainly seems like something that many people will need to do in order to use Kubernetes.
Other ideas I've had include using some combination of etcd, confd, and scripts on the host nodes, but this starts to get into the realm of host configuration management which I want to avoid at pretty much all costs. I'm using CoreOS and the hosts are provisioned entirely through coreos-cloudinit. In other words, nothing is manipulated on the host system that is not defined at the time the node is created, so traditional configuration management tools like Ansible are out.
I'm aware of Kubernetes's secrets system, but some of these variable values could be quite large, and most of them are not secrets.
This may work for you: ktmpl is a tool for processing Kubernetes manifest templates. It is a very simple client-side implementation of the Templates + Parameterization proposal.
https://github.com/InQuicker/ktmpl