Neutron Router is not replying arp requests

Question

Neutron Router can ping the instance but Instance can not ping the router or external network. The instance is on provider network via DHCP outside nova.

Following is the ping from router to instance

[root@rishi2 ~(keystone_admin)]# ip netns exec qrouter-c038097c-62bf-4eba-851c-f0ac060801bb ping 192.168.70.8
PING 192.168.70.8 (192.168.70.8) 56(84) bytes of data.
64 bytes from 192.168.70.8: icmp_seq=1 ttl=64 time=2.47 ms
64 bytes from 192.168.70.8: icmp_seq=2 ttl=64 time=0.340 ms

Following is the arp and ip route table

[root@rishi2 ~(keystone_admin)]# ip netns exec qrouter-c038097c-62bf-4eba-851c-f0ac060801bb ip r
default via 172.16.2.1 dev qg-b1e2ea82-67
10.10.10.0/24 dev qr-ee12b390-25  proto kernel  scope link  src 10.10.10.1
10.10.20.0/24 dev qr-88d204dd-45  proto kernel  scope link  src 10.10.20.1
172.16.2.0/24 dev qg-b1e2ea82-67  proto kernel  scope link  src 172.16.2.224
192.168.70.0/24 dev qg-b1e2ea82-67  scope link
192.168.70.0/24 dev qr-7f4d988f-bf  proto kernel  scope link  src 192.168.70.1
192.168.80.0/24 dev qg-b1e2ea82-67  scope link
192.168.80.0/24 dev qr-f44f0b41-53  proto kernel  scope link  src 192.168.80.1



[root@rishi2 ~(keystone_admin)]# ip netns exec qrouter-c038097c-62bf-4eba-851c-f0ac060801bb ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
7: qr-88d204dd-45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:5f:fa:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.10.20.1/24 brd 10.10.20.255 scope global qr-88d204dd-45
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe5f:fae0/64 scope link
       valid_lft forever preferred_lft forever
8: qr-ee12b390-25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:c4:24:99 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.1/24 brd 10.10.10.255 scope global qr-ee12b390-25
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec4:2499/64 scope link
       valid_lft forever preferred_lft forever
9: qg-b1e2ea82-67: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:6a:82:3b brd ff:ff:ff:ff:ff:ff
    inet 172.16.2.224/24 brd 172.16.2.255 scope global qg-b1e2ea82-67
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe6a:823b/64 scope link
       valid_lft forever preferred_lft forever
13: qr-f44f0b41-53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:22:1d:b1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.80.1/24 brd 192.168.80.255 scope global qr-f44f0b41-53
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe22:1db1/64 scope link
       valid_lft forever preferred_lft forever
14: qr-7f4d988f-bf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:73:f5:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.70.1/24 brd 192.168.70.255 scope global qr-7f4d988f-bf
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe73:f593/64 scope link
       valid_lft forever preferred_lft forever
[root@rishi2 ~(keystone_admin)]#

Any ideas why neutron router is not responding to arp?

[root@rishi2 ~(keystone_admin)]# ip netns exec qrouter-c038097c-62bf-4eba-851c-f0ac060801bb tcpdump -n -i qr-f44f0b41-53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qr-f44f0b41-53, link-type EN10MB (Ethernet), capture size 65535 bytes
02:24:28.322541 ARP, Request who-has 192.168.70.1 tell 192.168.70.8, length 42
02:24:29.317836 ARP, Request who-has 192.168.70.1 tell 192.168.70.8, length 42
02:24:29.679404 IP 172.16.2.66 > 224.0.0.22: igmp v3 report, 1 group record(s)
02:24:30.317832 ARP, Request who-has 192.168.70.1 tell 192.168.70.8, length 42
02:24:31.715735 IP 172.16.2.151.54985 > 224.0.0.252.hostmon: UDP, length 24
02:24:31.715722 IP 172.16.2.151.netbios-ns > 172.16.2.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
02:24:32.135843 IP 172.16.2.151.54985 > 224.0.0.252.hostmon: UDP, length 24
02:24:32.320279 ARP, Request who-has 192.168.70.1 tell 192.168.70.8, length 42
02:24:32.479140 IP 172.16.2.151.netbios-ns > 172.16.2.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
02:24:33.276089 IP 172.16.2.151.netbios-ns > 172.16.2.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
02:24:33.319842 ARP, Request who-has 192.168.70.1 tell 192.168.70.8, length 42

Answer 1

Neutron router cannot route between the subnets of same network. When the subnets were separated and kept on different networks - router worked correctly.

Root cause may still not be clear from this.

Answer 2

In my case, i disabled DHCP agent. I don't really know how my instance received an IP address.

I think routers uses DHCP agent to respond to ARP requests. Maybe i missconfigured something ?

I'm quite new to Openstack.

Re-enable DHCP agent solved my problem and routers are now able to respond to external ARP requests.