I'm learning about semaphores in the Argo project workflows to avoid concurrent workflows using the same resource.
My use case is that I have several external resources which only one workflow can use one at a time. So far so good, but sometimes the resource needs some maintenance and during that period I don't want to Argo to start any workflow.
I guess I have two options:
- I tested manually setting the semaphore value in the configMap to the value 0, but Argo started one workflow anyway.
- I can start a workflow that runs forever, until it is deleted, claiming the synchronization lock, but that requires some extra overhead to have workflows running that don't do anything.
So I wonder how it is supposed to work if I set the semaphore value to 0, I think it should not start the workflow then since it says 0. Anyone have any info about this?
This is the steps I carried out:
- First I apply my configmap with kubectl -f.
- I then submit some workflows and since they all use the same semaphore Argo will start one and the rest will be executed in order one at a time.
- I then change value of the semapore with kubectl edit configMap
- Submit new job which then Argo will execute.
Perhaps Argo will not reload the configMap when I update the configMap through kubectl edit? I would like to update the configmap programatically in the future but used kubectl edit now for testing.
Quick fix: after applying the ConfigMap change, cycle the workflow-controller pod. That will force it to reload semaphore state.
I couldn't reproduce your exact issue. After using
kubectl editto set the semaphore to0, any newly submitted workflows remainedPending.I did encounter an issue where using
kubectl editto bump up the semaphore limit did not automatically kick off any of thePendingworkflows. Cycling the workflow controller pod allowed the workflows to start running again.Besides using the quick fix, I'd recommend submitting an issue. Synchronization is a newer feature, and it's possible it's not 100% robust yet.