I have a complex product that runs like this.
- A parent Java process which expose an HTTP service.
- The parent process starts worker subprocesses (new JVM) and manage the lifecycle of them.
- Worker processes can't have HTTP server (for security and many other reasons), they communicate with other components through TCP connections. A parent process can manage many worker processes for a very long period of time.
- We need to use Prometheus to scrape metrics.
- Both parent and worker processes have micrometer observation instruments, but only parent exposed an HTTP server.
- We understand the limitations of Prometheus push gateway so we decided not to use it.
Question 1: How do we config Prometheus to scrape metrics from the workers when there is no HTTP.
Question 2: If Q1 is not doable, is there a way that we can send all metrics from workers to parent and let the parent to export these metrics? How should this work? Is there existing Micrometer tools to send a whole registry to another process through TCP? Once the parent received the registry from worker, how does the parent merge it with its own metrics?
I tried to find some examples but everything online shows that each JVM should have its own HTTP endpoint for scraping.