docker swarm macvlan network issues

Question

The Goal

---

Run a single haproxy container in the swarm with a static IP on my LAN that will be recreated if a node fails. In essence, it would be akin to a VIP pointing to haproxy but without requiring an external load balancer outside of the swarm.

---

What works - local macvlan

Example: Creating the macvlan networks on two nodes, running the container on node1, stopping the container, then running the container on node2. The haproxy container is created with the same static IP and is accessible on the LAN.

What doesn't work - swarm macvlan

Example: Setting the macvlan network scope to swarm, then deploying the stack. Container is not visible to the network.

---

Files used in example

docker-compose.yml:

version: "3.2"
networks:
  vlan0:
    external: true
services:
  haproxy:
    image: haproxy:2.3.2-alpine
    container_name: haproxy
    volumes:
      - ./data:/usr/local/etc/haproxy:ro
    environment:
      - TZ=America/Los_Angeles
    restart: always
    networks:
      vlan0:
        ipv4_address: 192.168.0.201

localnet.sh (script to stop stack / remove network / recreate network as local / run container local):

#!/bin/bash
docker service rm haproxy_haproxy
docker-compose down
docker network rm vlan0
docker network create -o parent=eth0 --subnet 192.168.0.0/24 --gateway 192.168.0.1 --driver macvlan --scope local vlan0
docker-compose up

swarmnet.sh (script to remove container and network / recreate network as swarm / run as swarm stack):

#!/bin/bash
docker service rm haproxy_haproxy
docker-compose down
docker network rm vlan0
docker network create -o parent=eth0 --subnet 192.168.0.0/24 --gateway 192.168.0.1 --driver macvlan --scope swarm vlan0
docker stack deploy -c docker-compose.yml haproxy

Is it possible to run a single container with a static macvlan IP in swarm mode?

Any help is appreciated.

Edit: I have been able to get macvlan addresses working, but docker swarm does not obey the ipv4_address field to static the container. I understand the reason for this (replicas, etc. not going on the same IP) but in this scenario, it will not happen due to it being a single container. I've found this issue discussed here: https://forums.docker.com/t/docker-swarm-1-13-static-ips-for-containers/28060/

Answer 1

I've figured out how to bind a single static ip address (with a major caveat below).

Method

To do this, create a config only network on each host with a single number ip range (32 bit mask). I've created a script here to auto-generate the commands.

    #!/bin/bash 
    echo 
    echo VIP swarm command generator
    echo 
    defint=$(ip route get 8.8.8.8 | head -n1 | awk '{print $5}')
    hostip=$(ip addr show dev $defint | grep "inet" | awk 'NR==1{print $2}' | cut -d'/' -f 1)
    hostsub=$(ip route | grep "src $hostip" | awk '{print $1}')
    hostgate=$(ip route show | grep default | awk '{print $3}')
    read -p "Subnet (default "$hostsub"): " sub
    sub=${sub:=$hostsub}
    read -p 'Gateway (default '$hostgate'): ' gate
    gate=${gate:=$hostgate}
    read -p 'Vip address: ' ip
    last=`echo $ip | cut -d . -f 4`
    begin=`echo $ip | cut -d"." -f1-3`
    read -p 'Ethernet interface (default '$defint'): ' eth
    eth=${eth:=$defint}
    read -p 'Docker Network Name: (default vip-'$last'): ' name
    name=${name:="vip-"$last}
    echo -e "Build the network on each node, \033[0;33mmake sure the physical parent interface (parent=) is set properly on each node if different)\033[0m:"
    echo -e "        \033[44mdocker network create --config-only --subnet $sub --gateway $gate --ip-range $ip/32 -o parent=$eth $name\033[0m"
    echo
    echo "Then create the swarm network from any manager node:"
    echo -e "        \033[44mdocker network create -d macvlan --scope swarm --config-from $name swarm-$name\033[0m"

Example output:

VIP swarm command generator

Subnet (default 192.168.0.0/24):
Gateway (default 192.168.0.1):
Vip address: 192.168.0.201
Ethernet interface (default eth0):
Docker Network Name: (default vip-201):
Build the network on each node, make sure the physical parent interface (parent=) is set properly on each node if different):
        docker network create --config-only --subnet 192.168.0.0/24 --gateway 192.168.0.1 --ip-range 192.168.0.201/32 -o parent=eth0 vip-201

Then create the swarm network from any manager node:
        docker network create -d macvlan --scope swarm --config-from vip-201 swarm-vip-201

So, on each node, I build the network config

docker network create --config-only --subnet 192.168.0.0/24 --gateway 192.168.0.1 --ip-range 192.168.0.201/32 -o parent=eth0 vip-201

Then I build the swarm interface

docker network create -d macvlan --scope swarm --config-from vip-201 swarm-vip-201

And in the applicable docker-compose.yaml file, add the appropriate lines:

networks:
  swarm-vip-201:
    external: true
services:
  haproxy:
    ...
    networks:
      swarm-vip-201:

Results

My container will now always be available at that single static IP address. If the node fails, it will restart on another node with the same IP, in essence creating a high-availability VIP for the swarm.

Caveat (Swarm macvlan bug?)

Docker swarm will throw errors trying to bind multiple macvlans to the same parent interface, like below:

"network NETNAME is already using parent interface eth0"

Described here:

https://github.com/moby/libnetwork/issues/2384

https://github.com/moby/libnetwork/issues/1743

Conclusion

It is promising that it is actually possible to create a static macvlan IP in the swarm, but until the bug is fixed with binding multiple networks to a single parent interface, it is very limited.

If anyone knows how to reliably fix the problem above, or has a better method, please post.