Spring Cloud Kubernetes: Timeout waiting for informers cache to be ready

Question

I'm attempting to take advantage of Spring Cloud Kubernetes in my Spring Boot-based microservice, namely autoconfiguration and service discovery.

However, I get an error message (Timeout waiting for informers cache to be ready, is the kubernetes service up?) during the initialization, as well as high verbosity, both of which I am unable to find information online to help me pinpoint the cause. Error message and excessive log messages are detailed below. Apologies for the lack of context, but I'm also puzzled!

The application is running within the default service account, which has all the permissions specified in the documentation:

$ kubectl describe serviceaccount default
Name:                default
Namespace:           joaomlneto
Labels:              <none>
Annotations:         <none>
Image pull secrets:  devspace-auth-rg-nl-ams-scw-cloud
Mountable secrets:   default-token-2sxvc
Tokens:              default-token-2sxvc
Events:              <none>

$ kubectl describe rolebinding namespace-reader-binding
Name:         namespace-reader-binding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  Role
  Name:  namespace-reader
Subjects:
  Kind            Name     Namespace
  ----            ----     ---------
  ServiceAccount  default  joaomlneto

$ kubectl describe role namespace-reader
Name:         namespace-reader
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources              Non-Resource URLs  Resource Names  Verbs
  ---------              -----------------  --------------  -----
  configmaps             []                 []              [list watch get]
  endpoints              []                 []              [list watch get]
  pods                   []                 []              [list watch get]
  secrets                []                 []              [list watch get]
  services               []                 []              [list watch get]
  configmaps.apps        []                 []              [list watch get]
  endpoints.apps         []                 []              [list watch get]
  pods.apps              []                 []              [list watch get]
  secrets.apps           []                 []              [list watch get]
  services.apps          []                 []              [list watch get]
  configmaps.extensions  []                 []              [list watch get]
  endpoints.extensions   []                 []              [list watch get]
  pods.extensions        []                 []              [list watch get]
  secrets.extensions     []                 []              [list watch get]
  services.extensions    []                 []              [list watch get]

I'm using Spring Boot 2.4; Spring Cloud Kubernetes 2.0.2. My pom.xml:

<properties>
    <java.version>11</java.version>
    <spring-cloud.version>2020.0.2</spring-cloud.version>
    <spring-cloud-kubernetes.version>2.0.2</spring-cloud-kubernetes.version>
    <graphql-java-kickstart.version>11.0.0</graphql-java-kickstart.version>
    <lombok.version>1.18.20</lombok.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-security</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-kubernetes-client-all</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-loadbalancer</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-openfeign</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>${lombok.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.graphql-java-kickstart</groupId>
        <artifactId>graphql-spring-boot-starter</artifactId>
        <version>${graphql-java-kickstart.version}</version>
    </dependency>
    <dependency>
        <groupId>com.graphql-java-kickstart</groupId>
        <artifactId>graphql-spring-boot-starter-test</artifactId>
        <version>${graphql-java-kickstart.version}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-kubernetes-dependencies</artifactId>
            <version>${spring-cloud-kubernetes.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <excludeDevtools>false</excludeDevtools>
            </configuration>
        </plugin>
    </plugins>
</build>

<repositories>
    <repository>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>Maven Central</name>
        <url>https://repo1.maven.org/maven2</url>
    </repository>
    <repository>
        <id>supersecret</id>
        <name>A Super Secret Repo</name>
        <url>https://example.com/1</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

<distributionManagement>
    <repository>
        <id>supersecret2</id>
        <name>Another Super Secret Repo</name>
        <url>https://example.com/2</url>
    </repository>
</distributionManagement>

However, a minute after the application starts successfully, I get the following error:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kubernetesInformerDiscoveryClient' defined in class path resource [org/springframework/cloud/kubernetes/client/discovery/KubernetesDiscoveryClientAutoConfiguration$KubernetesInformerDiscoveryConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Timeout waiting for informers cache to be ready, is the kubernetes service up?
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.5.jar:5.3.5]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.5.jar:5.3.5]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:769) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1313) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1302) ~[spring-boot-2.4.4.jar:2.4.4]
    at myorg.myapp.Application.main(Application.java:9) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.4.4.jar:2.4.4]
Caused by: java.lang.IllegalStateException: Timeout waiting for informers cache to be ready, is the kubernetes service up?
    at org.springframework.cloud.kubernetes.client.discovery.KubernetesInformerDiscoveryClient.afterPropertiesSet(KubernetesInformerDiscoveryClient.java:221) ~[spring-cloud-kubernetes-client-discovery-2.0.2.jar:2.0.2]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1845) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782) ~[spring-beans-5.3.5.jar:5.3.5]
    ... 22 common frames omitted

I also get the following messages logged every second (both before and after the error, ad infinitum):

2021-04-18 16:19:12.585  INFO 224 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2021-04-18 16:19:12.935  INFO 224 --- [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...
2021-04-18 16:19:12.950  INFO 224 --- [els.V1Service-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Service#Start listing and watching...

Answer 1

I managed to get rid of the error messages by using the Fabric8 client instead of the official client. This simple change was enough to get rid of my problems.

This was done by changing the dependency in pom.xml from spring-cloud-starter-kubernetes-client-all to spring-cloud-starter-kubernetes-fabric8-all.

As this doesn't seem like expected behavior, I opened a bug report on the Spring Cloud Kubernetes project on GitHub.

---

EDIT Jan/2022: The bug report has been marked as solved, although some people still experience the issue. I am still using Fabric8 and haven't ran into any issues. Haven't looked back.

Answer 2

You need to grant neccessary permissions to your spring app, with dependency spring-cloud-starter-kubernetes-fabric8-all that you're using, you will need to grant these permissions: "configmaps", "pods", "services", "endpoints", "secrets".

Below is an example from Spring Cloud document :

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: YOUR-NAME-SPACE
  name: namespace-reader
rules:
  - apiGroups: ["", "extensions", "apps"]
    resources: ["configmaps", "pods", "services", "endpoints", "secrets"]
    verbs: ["get", "list", "watch"]

---

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: namespace-reader-binding
  namespace: YOUR-NAME-SPACE
subjects:
- kind: ServiceAccount
  name: default
  apiGroup: ""
roleRef:
  kind: Role
  name: namespace-reader
  apiGroup: ""

Link : https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/index.html

Answer 3

You haven't posted your yaml configuration and how you applied it, so I cannot give guidance what's exactly wrong in your case. However, a possible cause for your issue could be a wrong token for the service account.

Reading the Security Configurations Inside Kubernetes part of the docs, we see the following quote:

For Kubernetes (1.3+), the namespace is made available to the pod as part of the service account secret and is automatically detected by the client

This means that if the service account's secret that the pod uses has for any reason misconfigured namespace, the kubernetes client will silently ignore the error and continue to try to list services/endpoints in a wrong namespace to which access is actually not granted.

You can list secrets in the namespace joaomlneto with:

kubectl get secrets --namespace joaomlneto

You can then check the namespace of the default service account token for the same namespace with:

kubectl get secret/default-token-..... --template='{{.data.namespace}}' --namespace joaomlneto

The base64-encoded output should decode to joaomlneto. If not, you will have issues with permissions. To fix it, you can delete and then recreate the service account, properly configuring it this time.