Configuration for ActiveMQ Artemis cluster on many Docker containers?

Question

I have no idea what I did wrong. I'm trying to make right configuration of ActiveMQ Artemis HA cluster on 2 docker containers, but I don't see in logs anything about "backup server" and so on. It seems to not work. I tried to make a simple Java client for the cluster, but it doesn't connect to it. Please don't write "have you read a documentation" - because yes, I've read it. From my perspective it seems like there is no connection between master and slave, but I don't know why. I've configured docker environment with data:

Dockerfile:

FROM ubuntu:latest
LABEL maintainer="Bernard"

RUN mkdir /artemis
WORKDIR /artemis

RUN apt-get update
RUN apt-get -y install wget
RUN apt-get -y install openjdk-11-jdk
RUN java --version

RUN wget -O "artemis.tar" "https://www.apache.org/dyn/closer.cgi?filename=activemq/activemq-artemis/2.18.0/apache-artemis-2.18.0-bin.tar.gz&action=download"
RUN tar -xvf ./artemis.tar; \
ln -s /artemis/apache-artemis-2.18.0/ ./current

RUN /artemis/current/bin/artemis create --user admin --password admin --http-host 0.0.0.0 --require-login --relax-jolokia bernard

EXPOSE 8161 \
9404 \
61616 \
61617 \
5445 \
5672 \
1883 \
61613

ENTRYPOINT ["/artemis/bernard/bin/artemis", "run"]

docker-compose.yml:

version: '3'

services:
  myartemis1:
    build: artemis/
    volumes:
      - "/e/sandbox/artemis_volume/artemis1/etc:/artemis/bernard/etc"
    networks:
      artemis-cluster-network:
        ipv4_address: 172.20.0.100
    ports:
      - 8161:8161
      - 9404:9404
      - 9876:9876
      - 61616:61616
      - 5445:5445
      - 5672:5672
      - 1883:1883
      - 61613:61613
      - 5432:5432
  myartemis2:
    build: artemis/
    volumes:
      - "/e/sandbox/artemis_volume/artemis2/etc:/artemis/bernard/etc"
    networks:
      artemis-cluster-network:
        ipv4_address: 172.20.0.101
    ports:
      - 8162:8161
      - 9405:9404
      - 9877:9876
      - 61618:61616
      - 61617:61617
      - 5446:5445
      - 5673:5672
      - 1884:1883
      - 61614:61613
      - 5433:5432

volumes:
  artemis1:
  artemis2:

networks:
  artemis-cluster-network:
    ipam:
      config:
        - subnet: 172.20.0.0/16

master broker.xml:

<?xml version='1.0'?>
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xi="http://www.w3.org/2001/XInclude" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
    <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core ">
        <bindings-directory>data/bindings</bindings-directory>
        <journal-directory>data/journal</journal-directory>
        <large-messages-directory>data/largemessages</large-messages-directory>
        <paging-directory>data/paging</paging-directory>
        <connectors>
            <connector name="netty-connector">tcp://localhost:61616</connector>
            <connector name="server1-connector">tcp://172.20.0.101:61617</connector>
        </connectors>
        <acceptors>
            <acceptor name="netty-acceptor">tcp://localhost:61616</acceptor>
        </acceptors>
        <ha-policy>
            <replication>
                <master>
                    <group-name>mynode</group-name>
                    <check-for-live-server>true</check-for-live-server>
                    <cluster-name>my-cluster</cluster-name>
                </master>
            </replication>
        </ha-policy>
        <broadcast-groups>
            <broadcast-group name="my-broadcast-group">
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <broadcast-period>2000</broadcast-period>
                <connector-ref>netty-connector</connector-ref>
            </broadcast-group>
        </broadcast-groups>
        <discovery-groups>
            <discovery-group name="my-discovery-group">
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <refresh-timeout>10000</refresh-timeout>
            </discovery-group>
        </discovery-groups>
        <cluster-connections>
            <cluster-connection name="my-cluster">
                <address>jms</address>
                <connector-ref>netty-connector</connector-ref>
                <check-period>1000</check-period>
                <connection-ttl>5000</connection-ttl>
                <min-large-message-size>50000</min-large-message-size>
                <call-timeout>5000</call-timeout>
                <retry-interval>500</retry-interval>
                <retry-interval-multiplier>1.0</retry-interval-multiplier>
                <max-retry-interval>5000</max-retry-interval>
                <initial-connect-attempts>-1</initial-connect-attempts>
                <reconnect-attempts>-1</reconnect-attempts>
                <use-duplicate-detection>true</use-duplicate-detection>
                <message-load-balancing>ON_DEMAND</message-load-balancing>
                <max-hops>1</max-hops>
                <confirmation-window-size>32000</confirmation-window-size>
                <call-failover-timeout>30000</call-failover-timeout>
                <notification-interval>1000</notification-interval>
                <notification-attempts>2</notification-attempts>
                <discovery-group-ref discovery-group-name="my-discovery-group"/>
            </cluster-connection>
        </cluster-connections>
        <security-settings>
            <security-setting match="#">
                <permission type="createNonDurableQueue" roles="guest"/>
                <permission type="deleteNonDurableQueue" roles="guest"/>
                <permission type="createDurableQueue" roles="guest"/>
                <permission type="deleteDurableQueue" roles="guest"/>
                <permission type="createAddress" roles="guest"/>
                <permission type="deleteAddress" roles="guest"/>
                <permission type="consume" roles="guest"/>
                <permission type="browse" roles="guest"/>
                <permission type="send" roles="guest"/>
                <permission type="manage" roles="guest"/>
            </security-setting>
        </security-settings>
        <addresses>
            <address name="DLQ">
                <anycast>
                    <queue name="DLQ"/>
                </anycast>
            </address>
            <address name="ExpiryQueue">
                <anycast>
                    <queue name="ExpiryQueue"/>
                </anycast>
            </address>
            <address name="exampleQueue">
                <anycast>
                    <queue name="exampleQueue"/>
                </anycast>
            </address>
        </addresses>
        <address-settings>
            <address-setting match="#">
                <redistribution-delay>0</redistribution-delay>
            </address-setting>
        </address-settings>
    </core>
</configuration>

slave broker.xml:

<?xml version='1.0'?>
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xi="http://www.w3.org/2001/XInclude" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
    <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core ">
        <paging-directory>data/paging</paging-directory>
        <bindings-directory>data/bindings</bindings-directory>
        <journal-directory>data/journal</journal-directory>
        <large-messages-directory>data/large-messages</large-messages-directory>
        <connectors>
            <connector name="netty-connector">tcp://localhost:61617</connector>
            <connector name="server0-connector">tcp://172.20.0.100:61616</connector>
        </connectors>
        <acceptors>
            <acceptor name="netty-acceptor">tcp://localhost:61617</acceptor>
        </acceptors>
        <ha-policy>
            <replication>
                <slave>
                    <group-name>mynode</group-name>
                    <cluster-name>my-cluster</cluster-name>
                </slave>
            </replication>
        </ha-policy>
        <broadcast-groups>
            <broadcast-group name="my-broadcast-group">
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <broadcast-period>2000</broadcast-period>
                <connector-ref>netty-connector</connector-ref>
            </broadcast-group>
        </broadcast-groups>
        <discovery-groups>
            <discovery-group name="my-discovery-group">
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <refresh-timeout>10000</refresh-timeout>
            </discovery-group>
        </discovery-groups>
        <cluster-connections>
            <cluster-connection name="my-cluster">
                <address/>
                <connector-ref>netty-connector</connector-ref>
                <check-period>1000</check-period>
                <connection-ttl>5000</connection-ttl>
                <min-large-message-size>50000</min-large-message-size>
                <call-timeout>5000</call-timeout>
                <retry-interval>500</retry-interval>
                <retry-interval-multiplier>1.0</retry-interval-multiplier>
                <max-retry-interval>5000</max-retry-interval>
                <initial-connect-attempts>-1</initial-connect-attempts>
                <reconnect-attempts>-1</reconnect-attempts>
                <use-duplicate-detection>true</use-duplicate-detection>
                <message-load-balancing>ON_DEMAND</message-load-balancing>
                <max-hops>1</max-hops>
                <confirmation-window-size>32000</confirmation-window-size>
                <call-failover-timeout>30000</call-failover-timeout>
                <notification-interval>1000</notification-interval>
                <notification-attempts>2</notification-attempts>
                <discovery-group-ref discovery-group-name="my-discovery-group"/>
            </cluster-connection>
        </cluster-connections>
        <security-settings>
            <security-setting match="#">
                <permission type="createNonDurableQueue" roles="guest"/>
                <permission type="deleteNonDurableQueue" roles="guest"/>
                <permission type="createDurableQueue" roles="guest"/>
                <permission type="deleteDurableQueue" roles="guest"/>
                <permission type="createAddress" roles="guest"/>
                <permission type="deleteAddress" roles="guest"/>
                <permission type="consume" roles="guest"/>
                <permission type="browse" roles="guest"/>
                <permission type="send" roles="guest"/>
                <permission type="manage" roles="guest"/>
            </security-setting>
        </security-settings>
        <addresses>
            <address name="DLQ">
                <anycast>
                    <queue name="DLQ"/>
                </anycast>
            </address>
            <address name="ExpiryQueue">
                <anycast>
                    <queue name="ExpiryQueue"/>
                </anycast>
            </address>
            <address name="exampleQueue">
                <anycast>
                    <queue name="exampleQueue"/>
                </anycast>
            </address>
        </addresses>
        <address-settings>
            <address-setting match="#">
                <redistribution-delay>0</redistribution-delay>
            </address-setting>
        </address-settings>
    </core>
</configuration>

snippet of Java client:

final String groupAddress = "231.7.7.7";
final int groupPort = 9876;
DiscoveryGroupConfiguration discoveryGroupConfiguration = new DiscoveryGroupConfiguration();
UDPBroadcastEndpointFactory udpBroadcastEndpointFactory =
        new UDPBroadcastEndpointFactory().setGroupAddress(groupAddress)
        .setGroupPort(groupPort);
discoveryGroupConfiguration.setBroadcastEndpointFactory(udpBroadcastEndpointFactory);
ConnectionFactory jmsConnectionFactory =
        ActiveMQJMSClient.createConnectionFactoryWithHA(discoveryGroupConfiguration, JMSFactoryType.CF);
Connection jmsConnection1 = jmsConnectionFactory.createConnection();
Connection jmsConnection2 = jmsConnectionFactory.createConnection();

Answer 1

The netty-acceptor acceptor must use the container IP address to allow external connections, i.e. the master acceptor should be:

<acceptor name="netty-acceptor">tcp://172.20.0.100:61616</acceptor>

and the slave acceptor should be:

<acceptor name="netty-acceptor">tcp://172.20.0.101:61616</acceptor>

The netty-connector must use the container IP address so other broker instances are able to connect to it when it is discovered, i.e. the master connector should be:

<connector name="netty-connector">tcp://172.20.0.100:61617</connector>

and the slave connector should be:

<connector name="netty-connector">tcp://172.20.0.100:61617</connector>

Docker Compose makes deploying microservice applications very easy but it has some limitations for a production environment. I would take a look to the open source ArtemisCloud.io project, it is a collection of container images that provide a way to deploy the Apache ActiveMQ Artemis Broker on Kubernetes.