How to make self updating pipeline in concourse

Question

I would like to make a pipeline that as first step checks its own configuration and updates itself if needed.

What tool / API should I use for this? Is there a docker image that has this installed for the correct concourse version? What is the advised way to authenticate in concourse from such task?

Answer 1

naively, it'd be a task which gets the repo the pipeline is committed to, and does a fly set-pipeline to update the configuration. However there are a few gotchas here:

• fly binary. you'll want fly executable to be available to your container which runs this task, and it should be same version of fly as the concourse that's being targeted. Probably that means you should download it directly via curl from the host.
• authenticating with the concourse server. you'll need to provide credentials for fly to use -- probably via parameters.
• parameter updates. if new parameters become needed, you'll need to use some kind of single source for all the parameters that need to be set, and use --load-vars-from rather than just --var. My group uses Lastpass notes with a bunch of variables saved in them and download via the lpass tool, but that gets hard if you use 2FA or similar.
• moving the server. You will need the external address of the concourse to be injected as a parameter as well, if you want to be resilient to it changing.

Answer 2

Regarding the previous answer suggesting the Fly binary, see the Fly resource.

However, having a similar request, I am going to try with the Pipeline resource. It seems more specific and has var injection solved directly through parameters.

I still have to try it out, but it seems to me that it would be more efficient to have a single pipeline which updates all pipelines, and not having to insert this job in all of your pipelines.

Also, a specific pipeline should not be concerned with itself, just the source code it builds (or whatever it does). If you want to start a pipeline if its config file changed, this could be done by modifying a triggering resource, e.g. pushing an empty "pipeline changed" commit