Multicast routing debugging
Setting up multicast reception on Fedora 34 requires using
firewall-cmd to allow multicast traffic through firewalld via rich
rules. The following is a description of a debugging session (or two)
that helped identify the problem.
The basic problem was that multicast traffic was not received by a userland program on Fedora 34 x86_64 machine. All of the tests below use the excellent mcjoin multicast test program.
1. Checking iptables
My first hypothesis was that multicast was blocked by the standard
firewall. As I was familiar only with ipchains and iptables, I
verified first that ipchains was not in use. From the iptables
man page:
This iptables is very similar to ipchains by Rusty Russell. The main difference is that the chains INPUT and OUTPUT are only traversed for packets coming into the local host and originating from the local host respectively. Hence every packet only passes through one of the three chains (except loopback traffic, which involves both INPUT and OUTPUT chains); previously a forwarded packet would pass through all three.
Trusting the quote above, the simplest way to test whether the INPUT and OUTPUT chains inhibit multicast reception is to use loopback. Done as follows:
sudo ifconfig lo multicast # loopback normally does not allow multicast sudo route add -host 224.12.12.12 dev lo # cargo-cult, but let's leave it here for now
With the setup above, the route table and the lo interface are
ready for multicast testing:
[ravi@rcvr ~]$ ifconfig lo
lo: flags=4169<UP,LOOPBACK,RUNNING,MULTICAST> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 28 bytes 2986 (2.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 28 bytes 2986 (2.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[ravi@rcvr ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.100.66.1 0.0.0.0 UG 100 0 0 enp4s0
10.100.66.0 0.0.0.0 255.255.255.0 U 100 0 0 enp4s0
192.168.211.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
224.12.12.12 0.0.0.0 255.255.255.255 UH 0 0 0 lo
Multicast published to 224.12.12.12 is received appropriately by
the receiver bound to lo:
mcjoin -s 224.12.12.12 -p 1234 -i lo # sender mcjoin -j 224.12.12.12 -i lo # receiver
The next step is to remove the multicast address from the route table:
sudo route del -host 224.12.12.12 dev lo
which results in
[ravi@rcvr ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 10.100.66.1 0.0.0.0 UG 100 0 0 enp4s0 10.100.66.0 0.0.0.0 255.255.255.0 U 100 0 0 enp4s0 192.168.211.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
Multicast transmission and reception work as expected again.
My conclusion is that iptables does not block multicast. In case
I am mistaken, here are the iptables rules:
ravi@rcvr ~]$ for table in filter nat; do echo "------ $table ------"; sudo iptables -t $table -L -n; done ------ filter ------ Chain INPUT (policy ACCEPT) target prot opt source destination LIBVIRT_INP all -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT) target prot opt source destination LIBVIRT_FWX all -- 0.0.0.0/0 0.0.0.0/0 LIBVIRT_FWI all -- 0.0.0.0/0 0.0.0.0/0 LIBVIRT_FWO all -- 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT) target prot opt source destination LIBVIRT_OUT all -- 0.0.0.0/0 0.0.0.0/0 Chain LIBVIRT_FWI (1 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 192.168.211.0/24 ctstate RELATED,ESTABLISHED REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable Chain LIBVIRT_FWO (1 references) target prot opt source destination ACCEPT all -- 192.168.211.0/24 0.0.0.0/0 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable Chain LIBVIRT_FWX (1 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 Chain LIBVIRT_INP (1 references) target prot opt source destination ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67 Chain LIBVIRT_OUT (1 references) target prot opt source destination ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:68 ------ nat ------ Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination LIBVIRT_PRT all -- 0.0.0.0/0 0.0.0.0/0 Chain LIBVIRT_PRT (1 references) target prot opt source destination RETURN all -- 192.168.211.0/24 224.0.0.0/24 RETURN all -- 192.168.211.0/24 255.255.255.255 MASQUERADE tcp -- 192.168.211.0/24 !192.168.211.0/24 masq ports: 1024-65535 MASQUERADE udp -- 192.168.211.0/24 !192.168.211.0/24 masq ports: 1024-65535 MASQUERADE all -- 192.168.211.0/24 !192.168.211.0/24
All of the libvirt and 192.168.211.0 stuff above is due to the
presence of a virtual machine used for testing. In the end, those
pieces turned out to be irrelevant, but I left them in to show
acutal debugging values.
2. Sending multicast from a separate host
| Sender: sndr | Receiver: rcvr | |
|---|---|---|
| IP address | 10.100.66.68 | 10.100.66.67 |
| Interface | enp1s0 | enp4s0 |
| Kernel version | 5.14.16-201.fc34.x86_64 | 5.15.11-100.fc34.x86_64 |
- Both hosts can ping each other.
- Traffic between them does not have any VLAN tagging, since they are on the same VLAN.
- Each of them has a wireless interface without an assigned IP address or an associated access point.
- Sender sends multicast on 224.17.17.17 port 1234.
Focusing on the receiver:
[ravi@rcvr ~]$ sudo sysctl -ar '\.rp_filter'
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.enp4s0.rp_filter = 2
net.ipv4.conf.lo.rp_filter = 2
net.ipv4.conf.virbr0.rp_filter = 2
net.ipv4.conf.wlp3s0.rp_filter = 2
Packets are sent from the transmitter at 1pps:
./mcjoin -f 1000 -s -t 4 224.17.17.17
The receiver does not see the data in userland both with and without the useless multicast route:
sudo route add -host 224.17.17.17 dev enp4s0 # for testing sudo route del -host 224.17.17.17 dev enp4s0 # remove testing
Packets are visible on the interface, regardless of whether there has been a multicast join, since they are on the same VLAN and the switch floods those packets:
[ravi@rcvr ~]$ sudo tcpdump -i enp4s0 -nn -e -vv -c 5 multicast and not broadcast
dropped privs to tcpdump
tcpdump: listening on enp4s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:24:36.838722 48:9a:e3:30:5c:74 > 01:00:5e:11:11:11, ethertype IPv4 (0x0800), length 142: (tos 0x0, ttl 4, id 25384, offset 0, flags [DF], proto UDP (17), length 128)
10.100.66.68.36682 > 224.17.17.17.1234: [udp sum ok] UDP, length 100
09:24:37.839498 48:9a:e3:30:5c:74 > 01:00:5e:11:11:11, ethertype IPv4 (0x0800), length 142: (tos 0x0, ttl 4, id 25701, offset 0, flags [DF], proto UDP (17), length 128)
10.100.66.68.36682 > 224.17.17.17.1234: [udp sum ok] UDP, length 100
09:24:38.840155 48:9a:e3:30:5c:74 > 01:00:5e:11:11:11, ethertype IPv4 (0x0800), length 142: (tos 0x0, ttl 4, id 25798, offset 0, flags [DF], proto UDP (17), length 128)
10.100.66.68.36682 > 224.17.17.17.1234: [udp sum ok] UDP, length 100
09:24:39.840319 48:9a:e3:30:5c:74 > 01:00:5e:11:11:11, ethertype IPv4 (0x0800), length 142: (tos 0x0, ttl 4, id 26349, offset 0, flags [DF], proto UDP (17), length 128)
10.100.66.68.36682 > 224.17.17.17.1234: [udp sum ok] UDP, length 100
09:24:40.840963 48:9a:e3:30:5c:74 > 01:00:5e:11:11:11, ethertype IPv4 (0x0800), length 142: (tos 0x0, ttl 4, id 27016, offset 0, flags [DF], proto UDP (17), length 128)
10.100.66.68.36682 > 224.17.17.17.1234: [udp sum ok] UDP, length 100
5 packets captured
5 packets received by filter
0 packets dropped by kernel
Note that the TTL is 4 at the receiver, which should be more than sufficient.
When the receiver is running, packets do appear:
[ravi@rcvr ~]$ netstat -gn -4
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.100.66.67:42580 10.100.66.68:22 ESTABLISHED
[... other irrelevant connections redacted ...]
udp 0 0 10.100.66.67:68 10.100.66.1:67 ESTABLISHED
IPv6/IPv4 Group Memberships
Interface RefCnt Group
--------------- ------ ---------------------
lo 1 224.0.0.251
lo 1 224.0.0.1
enp4s0 1 224.17.17.17
enp4s0 1 224.0.0.251
enp4s0 1 224.0.0.252
enp4s0 1 224.0.0.1
virbr0 1 224.0.0.251
virbr0 1 224.0.0.1
virbr0 1 224.0.0.106
[ravi@rcvr ~]$ netstat -sgu; sleep 10; netstat -sgu
IcmpMsg:
InType3: 22
OutType3: 22
Udp:
2552 packets received
0 packets to unknown port received
0 packet receive errors
2203 packets sent
0 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InMcastPkts: 754
OutMcastPkts: 1105
InBcastPkts: 2
OutBcastPkts: 4
InOctets: 3587966
OutOctets: 757004
InMcastOctets: 83946
OutMcastOctets: 127091
InBcastOctets: 4202
OutBcastOctets: 4242
InNoECTPkts: 5968
MPTcpExt:
IcmpMsg:
InType3: 22
OutType3: 22
Udp:
2552 packets received
0 packets to unknown port received
0 packet receive errors
2203 packets sent
0 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InMcastPkts: 764
OutMcastPkts: 1105
InBcastPkts: 2
OutBcastPkts: 4
InOctets: 3594625
OutOctets: 757719
InMcastOctets: 85226
OutMcastOctets: 127091
InBcastOctets: 4202
OutBcastOctets: 4242
InNoECTPkts: 5990
MPTcpExt:
[ravi@rcvr ~]$ ip maddr
1: lo
inet 224.0.0.251
inet 224.0.0.1
inet6 ff02::fb
inet6 ff02::1
inet6 ff01::1
2: enp4s0
link 01:00:5e:00:00:01
link 33:33:00:00:00:01
link 33:33:00:00:00:fb
link 33:33:ff:13:a0:aa
link 33:33:00:01:00:03
link 01:00:5e:00:00:fc
link 01:00:5e:00:00:fb
link 01:00:5e:11:11:11
inet 224.17.17.17
inet 224.0.0.251
inet 224.0.0.252
inet 224.0.0.1
inet6 ff02::1:3
inet6 ff02::1:ff13:a0aa
inet6 ff02::fb
inet6 ff02::1
inet6 ff01::1
3: wlp3s0
inet6 ff02::1
inet6 ff01::1
4: virbr0
link 01:00:5e:00:00:6a
link 33:33:00:00:00:6a
link 01:00:5e:00:00:01
link 01:00:5e:00:00:fb
inet 224.0.0.251
inet 224.0.0.1
inet 224.0.0.106
inet6 ff02::6a
inet6 ff02::1
inet6 ff01::1
Since packets are sent at the rate of 1 packet per second from the
sender, the number of packets matches expectations in netstat
-sgu.
Repeating the same experiment above with the following changes yields the same results.
[ravi@rcvr ~]$ sudo sysctl -w net.ipv4.conf.enp4s0.rp_filter=0
net.ipv4.conf.enp4s0.rp_filter = 0
[ravi@rcvr ~]$ sudo sysctl -w net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.rp_filter = 0
[ravi@rcvr ~]$ sudo sysctl -w net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.all.rp_filter = 0
[ravi@rcvr ~]$ sudo sysctl -ar '\.rp_filter'
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.enp4s0.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 2
net.ipv4.conf.virbr0.rp_filter = 2
net.ipv4.conf.wlp3s0.rp_filter = 2
3. Solution
At this point, I started looking through any documentation for firewalls, as there was nothing else I could see. The man page for firewalld hinted vaguely at a debugging facility:
–set-log-denied=value
Add logging rules right before reject and drop rules in the INPUT, FORWARD and OUTPUT chains for the default rules and also final reject and drop rules in zones for the configured link-layer packet type. The possible values are: all, unicast, broadcast, multicast and off. The default setting is off, which disables the logging.
Useful debugging information came from
sudo firewall-cmd --set-log-denied=multicast
whose output could be observed via journalctl. We still need to
figure out the exact set of multicast destinations to allow.
The problem is in netfilter; even though iptables did not return
anything useful, firewalld (using nft) was still setting up
rejections. All multicast except mDNS was blocked. The solution
was to allow multicast via firewalld:
sudo firewall-cmd --add-rich-rule='rule family=ipv4 destination address="224.0.0.0/4" accept'
sudo firewall-cmd --add-protocol=igmp
sudo firewall-cmd --reload
It's annoying that there was no obvious way to query nftables
rules directly without dropping into the interactive nft shell,
but that's probably only a lack of knowledge on my part. As I was
testing this during some VyOS SSDP/DLNA debugging, isolating this
as a problem with the local host also took additional time.
