Implementing node discovery on Kubernetes

Question

We have a java application (microservice) that would be deployed on kubernetes using k8s deployment and one of the requirements is that each pod needs to know all the other pods running that application. So, in other words, we would need a way by which application layer retrieve all the pods that are part of the k8s deployment.

One custom implementation could be to have a persistant data store like database and have each of the pods send a hearbeat entry to indicate its liveness and manage the entries by listening to application lifecycle events

I know that any clustering technology satisfies this requirement. But I am looking for a minimal readymade library that does only this node discovery (and/or) management in kubernetes ?

Answer 1

One custom implementation could be to have a persistant data store like database and have each of the pods send a hearbeat entry to indicate its liveness and manage the entries by listening to application lifecycle events

I would suggest here to use the liveness probe and readiness probe for the POD whenever pod get join respective DNS entry will be auto-updated rather checking the liveness using custom solution.

to getting the list of all PODs IP you can use the Kubernetes service with type headless. you have to use spec.clusterIP as None.

Example service YAMl :

apiVersion: v1
kind: Service
metadata:
  name: mongodb-headless-service
  namespace: infrastructure
spec:
  type: NodePort
  clusterIP: None
  selector:
    app: mongodb
  ports:
  - name: mongodb
    port: 27017
    targetPort: 27017
    protocol: TCP
  selector:
    app: mongodb

from your custom solution code you have send request to kuberneets service which type if headless : <servicename>.<namespace>.svc.cluster.local.

code example for mongo :

System.Net.IPAddress[] ipAddresses = Dns.GetHostAddresses("mongodb-headless-service");
string connectionString = "";
foreach(IPAddress in ipAddresses)
{
  if(connectionString = "")
    connectionString = "mongodb://";
  else
    connectionString += ",";
  connectionString += $"{IPAddress.ToString()}:27017";
}
connectionString += "/database";
var client = new MongoClient(connectionString);

extra document : https://medium.com/swlh/discovering-running-pods-by-using-dns-and-headless-services-in-kubernetes-7002a50747f4

Answer 2

You can use a headless service. This service returns the IP addresses of the selected pods at a dns lookup instead of load balancing the requests.

First create the service. The important thing is to set the spec.clusterIP to None.

apiVersion: v1
kind: Service
metadata:
  name: headless-service
spec:
  clusterIP: None
  selector:
    ...
  ports:
    - ...

Afterwards you can get your IPs in java e.g. via:

InetAddress address = InetAddress.getByName("headless-service"); 
System.out.println(address.getHostAddress());