We are considering moving Jenkins to Kubernetes (GKE) and using Jenkins Kubernetes plugin to run job in pods. However, our build infra uses ccache for build speed optimization and we keep ccache files on the jenkins agents (every build reads and writes to local ccache folder on each agent).
Per my understanding Jenkins agent pods in k8s are ephemeral, meaning they are for a single use only. I am struggling to understand how ccache set up in going to work. Here is what I am thinking:
- Use hostpath volume in the pod templates to have agent pod mount a path on a node. I am afraid that is not going to work because based on GKE documentation hostpath doesn't allow ReadWriteMany, meaning multiple pods won't be able to write to ccache mount at the same time.
- Use NFS based volume - I know for a fact that won't work well as it's going to be slow and will have locking problems.
- Somehow configure pod templates with 'podRetention' and 'idleMinutes' option to have Pods to be re-used. I think this is doable, but IMO this defeats the purpose of using Kubernetes.
- Use StateFull Jenkins Agents - again I think this is doable but defeats the purpose of using Kubernetes.
I found https://issues.jenkins.io/browse/JENKINS-42422 which might be related, but I don't think there is a solution.
Any other ideas on how this can be implemented? Not being able to use ccache is a deal breaker for us as it will make build turnaround time way worse compared to static agents.
Being stateful does not defeat the purpose of Kubernetes. Stateful apps, are necessary, for example, if you are running a DB in your cluster. I would suggest that you use glusterfs/ceph/nfs/EFS(AWS) and use readWriteMany for ccache.
This blog has more details:
https://blog.hiya.com/kubernetes-base-jenkins-stateful-agents/