The machineSet im deploying has the keyword CLUSTERID injected in several places of the manifest. Kyverno will mutate each instance of the word for the actual cluster name. if i deploy it "manually" using helm install it works great, but when deployed by ArgoCD it gets created > mutated > Out of Sync > Deleted > re-created in an endless loop.
i have tried removing all the jsonPointers in order to ignore the whole api-resoource and also add Kyverno as filedManager. none worked.
ignoreDifferences:
- group: "*"
kind: "*"
managedFieldsManagers:
- kyverno
This is the result of running helm install kyverno-machineset -f ../../myvalues.yml ./ocp-machineset --dry-run --debug (to show where is CLUSTERID injected by the helm template)
# Source: ocp-machineset/templates/machineset.yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
creationTimestamp: null
labels:
machine.openshift.io/cluster-api-cluster: CLUSTERID
name: CLUSTERID-kyverno-worker
namespace: openshift-machine-api
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: CLUSTERID
machine.openshift.io/cluster-api-machineset: CLUSTERID-kyverno-worker
template:
metadata:
creationTimestamp: null
labels:
machine.openshift.io/cluster-api-cluster: CLUSTERID
machine.openshift.io/cluster-api-machine-role: kyverno
machine.openshift.io/cluster-api-machine-type: kyverno
machine.openshift.io/cluster-api-machineset: CLUSTERID-kyverno-worker
spec:
metadata:
labels:
node-role.kubernetes.io/kyverno: ""
taints:
- key: node-role.kubernetes.io/kyverno
effect: NoSchedule
providerSpec:
value:
apiVersion: vsphereprovider.openshift.io/v1beta1
credentialsSecret:
name: vsphere-secret
diskGiB: 200
kind: VSphereMachineProviderSpec
memoryMiB: 76000
metadata:
creationTimestamp: null
network:
devices:
- networkName: LAB-NET
numCPUs: 16
numCoresPerSocket: 16
snapshot: ""
template: CLUSTERID-rhcos
userDataSecret:
name: worker-user-data
workspace:
datacenter: lab
datastore: vsphere_openshift
folder: /folder/to/datastore
resourcePool: /folder/to/resourcepool
server: vcenter.lab.example.com
this is my ArgoCD app.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: app-worker-nodes-kyverno-test
labels:
clusterName: lab1
owner: "John Doe"
spec:
project: project
source:
chart: "ocp-machineset"
repoURL: 'https://repo.example.com'
targetRevision: ^0.1.1-0
helm:
values: |-
cluster: "CLUSTERID"
vmware:
datacenter: "lab"
datastore: "vsphere_openshift"
folder: " /folder/to/datastore"
resourcePool: "/folder/to/resourcepool"
server: "vcenter.lab.example.com"
networkName: "LAB-NET"
machineset:
name: kyverno
storage: 200
cpu: 16
memory: 76000
replicas: 1
isDefaultWorker: false
destination:
server: 'https://lab.cluster.example.com'
namespace: openshift-machine-api
syncPolicy:
automated:
prune: false
selfHeal: true
syncOptions:
- Replace=true
ignoreDifferences:
- group: machine.openshift.io
kind: MachineSet
namespace: openshift-machine-api
jsonPointers:
- /metadata/name
- /metadata/labels/machine.openshift.io~1cluster-api-cluster
- /spec/selector/matchLabels/machine.openshift.io~1cluster-api-cluster
- /spec/selector/matchLabels/machine.openshift.io~1cluster-api-machineset
- /spec/template/metadata/labels/machine.openshift.io~1cluster-api-cluster
- /spec/template/metadata/labels/machine.openshift.io~1cluster-api-machineset
- /spec/template/spec/providerSpec/value/template
As you can see this makes ArgoCD think the resource deployed to the cluster doesn't exist in the source manifest and should be deleted (trash bin) and the one defined in the manifest (CLUSTERID) is missing (ghost)
ArgoCD should ignore the differences in the fields listed under ignoreDifferences and keep the resrouces as Sync & Healthy.