ActiveMQ Artemis management console fails to load due to Jolokia endpoint timeout. No error logs. Where to start debugging?

Question

I am using the activemq-artemis-operator to deploy an activemq-artemis cluster.

When port-forwarding to the management console the UI does not load consistently. More often there is one endpoint which times out and all the UI fails to load.

Having limited knowledge with the stack I don't know where to start to debug this.

The payload always seems to be this:

{"type":"exec","mbean":"hawtio:type=security,area=jmx,name=ArtemisJMXSecurity","operation":"canInvoke(java.lang.String, java.lang.String)","arguments":["org.apache.activemq.artemis:broker=\"amq-broker\"","THIS_CHANGES"]}

Also logging and in and out again yields different results after every login. Sometimes the Artemis doesn't even show up and images are missing.

Would really appreciate some help to debug if possible.

EDIT: Some more context.

login.config

activemq {
  org.apache.activemq.artemis.spi.core.security.jaas.PropertiesLoginModule required
    reload=true
    org.apache.activemq.jaas.properties.user="artemis-users.properties"
    org.apache.activemq.jaas.properties.role="artemis-roles.properties";
};

log4j2.properties

# Log4J 2 configuration
# Monitor config file every X seconds for updates
monitorInterval = 5
rootLogger = INFO, console
logger.activemq.name=org.apache.activemq
logger.activemq.level=WARN
logger.jolokia.name=org.jolokia
logger.jolokia.level=DEBUG
logger.artemis_server.name=org.apache.activemq.artemis.core.server
logger.artemis_server.level=WARN
logger.artemis_journal.name=org.apache.activemq.artemis.journal
logger.artemis_journal.level=WARN
logger.artemis_utils.name=org.apache.activemq.artemis.utils
logger.artemis_utils.level=WARN
logger.critical_analyzer.name=org.apache.activemq.artemis.utils.critical
logger.critical_analyzer.level=WARN
# Audit loggers: to enable change levels from OFF to INFO
logger.audit_base = OFF, audit_log_file
logger.audit_base.name = org.apache.activemq.audit.base
logger.audit_base.additivity = false
logger.audit_resource = OFF, audit_log_file
logger.audit_resource.name = org.apache.activemq.audit.resource
logger.audit_resource.additivity = false
logger.audit_message = OFF, audit_log_file
logger.audit_message.name = org.apache.activemq.audit.message
logger.audit_message.additivity = false
# Jetty logger levels
logger.jetty.name=org.eclipse.jetty
logger.jetty.level=WARN
# Quorum related logger levels
logger.curator.name=org.apache.curator
logger.curator.level=WARN
logger.zookeeper.name=org.apache.zookeeper
logger.zookeeper.level=WARN
# Console appender
appender.console.type=Console
appender.console.name=console
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d %-5level [%logger] %msg%n
# Log file appender
appender.log_file.type = RollingFile
appender.log_file.name = log_file
appender.log_file.fileName = ${sys:artemis.instance}/log/artemis.log
appender.log_file.filePattern = ${sys:artemis.instance}/log/artemis.log.%d{yyyy-MM-dd}
appender.log_file.layout.type = PatternLayout
appender.log_file.layout.pattern = %d %-5level [%logger] %msg%n
appender.log_file.policies.type = Policies
appender.log_file.policies.cron.type = CronTriggeringPolicy
appender.log_file.policies.cron.schedule = 0 0 0 * * ?
appender.log_file.policies.cron.evaluateOnStartup = true
# Audit log file appender
appender.audit_log_file.type = RollingFile
appender.audit_log_file.name = audit_log_file
appender.audit_log_file.fileName = ${sys:artemis.instance}/log/audit.log
appender.audit_log_file.filePattern = ${sys:artemis.instance}/log/audit.log.%d{yyyy-MM-dd}
appender.audit_log_file.layout.type = PatternLayout
appender.audit_log_file.layout.pattern = %d [AUDIT](%t) %msg%n
appender.audit_log_file.policies.type = Policies
appender.audit_log_file.policies.cron.type = CronTriggeringPolicy

bootstrap.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<broker xmlns="http://activemq.apache.org/schema">
  <jaas-security domain="activemq"/>
  <server configuration="file:/home/jboss/amq-broker/etc/broker.xml"/>
  <web customizer="org.eclipse.jetty.server.ForwardedRequestCustomizer" path="web" rootRedirectLocation="console">
      <binding uri="http://0.0.0.0:8161">
          <app url="activemq-branding" war="activemq-branding.war"/>
          <app url="artemis-plugin" war="artemis-plugin.war"/>
          <app url="console" war="console.war"/>
          <app url="metrics" war="metrics.war"/>
      </binding>
  </web>
</broker>

And last, but not least: artemis-cr.yaml