I'm finding a lack of examples in the docs with a specific configuration I have in one of my clusters.
Basically my pods are running with hostNetwork: true enabled, which means that they share a port mapping with all of the pods with that flag set. This is due to the unique type of application that I am running in these pods and must be enabled.
The problem arises when I try to run a prometheus_client (python) in these pods. The prometheus client has to pick a port at random and try to bind to it. (If it picks a hard-coded one, it will only succeed for the first pod on the node and the second will fail as the port will already be taken) So if we start up the prometheus client in the pod on a random port, I'm not sure how to tell grafana-agent how to scrape it. Afaik grafana-agent config expect a port to be declared in the annotation with this part of the config:
- source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
Can I dynamically update my annotation from within the application? I think that is not a good practice in k8s?
Is there some way in k8s to declare a routing to a specific port for a specific pod?
Any and all help appreciated.