I have 5 nodes in a cluster (Test). I have labeled them to the following:
Nodes:
- namespace=A
- namespace=A
- namespace=B
- namespace=B
- namespace=C
I applied taints and tolerations, nodeAffinity and podAntiAffinity. Our nodes is auto scale enabled. However, our nodes weren't scaling up, all the pods are going in 1 node. I have read in this link Kubernetes: Evenly distribute the replicas across the cluster, that using podAntiAffinity, node Affinity, taints and tolerations do not guarantee this requirement. Our requirement is, 1 pod should be deployed evenly on nodes and should scale up accordingly.
What am I missing?
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: prometheus
labels:
app: prometheus
spec:
serviceName: "prometheus"
selector:
matchLabels:
name: prometheus
template:
metadata:
labels:
name: prometheus
app: prometheus
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: namespace
operator: In
values:
- A
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- prometheus
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- name: prometheus
image: quay.io/prometheus/prometheus:v2.6.0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
terminationGracePeriodSeconds: 30
tolerations:
- key: namespace
operator: Equal
value: A
Did you try
nodeSelector
which will schedule the pod on the node that you attached the label to?