Require that k8s pods have a certain label before they run on certain nodes

Question

Say I have some special nodes in my cluster, and I want to be able to identify with a pod label all pods running on these nodes.

Taints & tolerations are close to this, but afaik tolerations aren’t labels, they’re their own thing, and can’t be referenced in places where you expect a label selector, right? Is there any way to require all pods running on these nodes to identify themselves in a way that can then be queried via label selectors?

Answer 1

There are two parts of problem to solve:
1. Ensure that pods with specific labels are scheduled to specific set of nodes.
This can be done by using custom scheduler or a plugin for default scheduler (more here, here, here, here and here).

2. Prevent other schedulers to schedule pods to that nodes
This can by achieved by using nodeAffinity or node taints (more here and here)

A complete solution may require additional apiserver admission controller to set all those properties to a pod that suppose to have a specific label only.
Unfortunately, there is no easy built-in solution as of yet. Hopefuly the above will point you in the right direction.