I have a ksqldbcluster with 2 nodes. Sometimes i might receive an alert from grafana(which i monitor the cluster) that some queries are in error state. When i check the show queries i see that a number of queries have 1 PERSISTENT running and 1 ERROR. I restart first the second ksqdb docker instance. Sometimes is fixed sometimes is not. If is not i restart the first ksqldb docker instance and then is fixed. Any ideas?
version: '3' services: ksqldb-server: container_name: ksqldb-server hostname: ksqldb.xxxxxxx network_mode: host image: confluentinc/ksqldb-server:latest environment: KSQL_HOST_NAME: ksqldb.xxxxxxxx KSQL_BOOTSTRAP_SERVERS: xxxxxx:9093 KSQL_SECURITY_PROTOCOL: SSL KSQL_LISTENERS: http://xxxxxx:8088,https://xxxxxx:8443 KSQL_KSQL_SCHEMA_REGISTRY_URL: http://xxxxxxxx-as.net:8081,http://xxxxxx:8082 KSQL_SSL_CLIENT_KEY_STORE_LOCATION: /etc/kafka/secrets/ksqldb.keystore.jks KSQL_SSL_CLIENT_KEY_STORE_PASSWORD: xxxxxx KSQL_SSL_CLIENT_TRUST_STORE_LOCATION: /etc/kafka/secrets/ksqldb.keystore.jks KSQL_SSL_CLIENT_TRUST_STORE_PASSWORD: xxxxxxx KSQL_SERVER_ID: id1 KSQL_STREAMS_NUM_STANDBY_REPLICAS: 1 # Corrected property name KSQL_LOG4J_ROOT_LOGLEVEL: "ERROR" KSQL_QUERY_PULL_ENABLE_STANDBY_READS: true KSQL_HEARTBEAT_ENABLE: true KSQL_QUERY_PULL_THREAD_POOL_SIZE: "30" KSQL_LAG_REPORTING_ENABLE: true KSQL_INTERNAL_LISTENER: http://xxxxxxx:8080 KSQL_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/ksqldb.truststore.jks KSQL_SSL_TRUSTSTORE_PASSWORD: xxxxxx KSQL_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/ksqldb.keystore.jks KSQL_SSL_KEYSTORE_PASSWORD: xxxxxxx KSQL_JMX_OPTS: -Dcom.sun.management.jmxremote -javaagent:/etc/kafka/secrets/jmx_prometheus_javaagent-0.17.2.jar=4000:/etc/kafka/secrets/ksql.yaml ports: - 8088:8088 - 4000:4000 - 8443:8443 volumes: - /ksqldata/data:/var/lib/ksqldb-server/data - /ksqldata/newssl/:/etc/kafka/secrets/
ksqldb-cli: image: confluentinc/ksqldb-cli:latest container_name: ksqldb-cli entrypoint: /bin/sh tty: true # Corrected indentation
telegraf: image: telegraf restart: always volumes: - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro - /var/run/docker.sock:/var/run/docker.sock ~
When restarting ksqldb instance the issue is resolved. Maybe i am missing something in my parameters or how to fix the error?