How a resource from a container be available to another container if each task runs in its own container?

Question

I keep hearing that each task runs in its own container and in that case say If have a Job1 that produces some resource as an output which Job2 wants to read as input (a typical case of pipelining Jobs) so in that case I wonder How a resource from a container be available to another container if each task runs in it’s own container?

Also if a single Job has multiple tasks then does each task runs in its own new container?

Answer 1

Each task does indeed run in its own container. The trick here is that Concourse uses an internal package called baggageclaim for artifact persistence beyond the lifecycle of individual task containers.

If you specify that a task has an output, then the directory named for (or mapped to) that output will be mounted as a persistent volume rather than an ephemeral container-internal folder. When you write to that directory, it will live in the baggage claim's internal workspace available to be mounted to other containers (which is what happens when you specify that a later task in the same build should be using that output as an input.

get steps produce these volumes in the same sense; this is why when you do a get on a particular resource, the downloaded bits for the resource are available to any of the later steps in the job if you name it as an input.