This command was used to set bits in 0x3f00 (bits 8-13) of inbound packets to 0x200 (000010):
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/32 mark set=0x200/0x3f00"
With the iptables backend, we get --set-xmark 0x200/0x3f00:
$ sudo iptables -t mangle --list-rules
...
-A PRE_public_allow -s 192.0.2.0/32 -j MARK --set-xmark 0x200/0x3f00
...
This means:
meta mark = meta mark & ~0x00003f00 ^ 0x00000200
which is equivalent to:
meta mark = meta mark & 0xffffc0ff ^ 0x00000200
This AND's the meta mark with the inverse of our mask, discarding the bits in 0x3f00, then XOR's that value with 0x200. This effectively sets only bits 0x3f00 to 0x200, while leaving all bits outside of 0x3f00 intact.
The AND with the mask inverse is also confirmed by the iptables-extensions(8) manual:
--set-xmark value[/mask]
Zeroes out the bits given by mask and XORs value into the packet
mark ("nfmark"). If mask is omitted, 0xFFFFFFFF is assumed.
However, with the nftables backend, we get the following rule:
$ sudo nft list ruleset
...
chain mangle_PRE_public_allow {
ip saddr 192.0.2.0 meta mark set meta mark & 0x00003f00 ^ 0x00000200
}
...
Which means:
meta mark = meta mark & 0x00003f00 ^ 0x00000200
This AND's the meta mark with the mask (not its inverse), keeping only bits 0x3f00, then XOR's that value with 0x200. This effectively toggles (not sets) bits 0x200 and sets all bits outside of 0x3f00 to zero.
In the event that we're starting with a meta mark of all zeroes, both implementations of the rule have the same effect of setting the overall meta mark value to 0x200, and any routing rules matching fwmark 0x200/0x3f00 will work by coincidence, but that's about it. Chaining these sorts of rules to set different bits in a meta mark won't work the same way between iptables and nftables backends.
For example, when using the iptables backend:
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/32 mark set=0x200/0x3f00"
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/32 mark set=0x20/0xf0"
$ sudo iptables -t mangle -A PRE_public_allow -s 192.0.2.0/32 -j LOG
$ sudo iptables -t mangle --list-rules
...
-A PRE_public_allow -s 192.0.2.0/32 -j MARK --set-xmark 0x200/0x3f00
-A PRE_public_allow -s 192.0.2.0/32 -j MARK --set-xmark 0x20/0xf0
-A PRE_public_allow -s 192.0.2.0/32 -j LOG
...
On another machine:
$ sudo ip address add dev br-trunk.16 192.0.2.0/32
$ ping -nc1 -I 192.0.2.0 -b 255.255.255.255
Back on the test machine (note the MARK=0x220, which is desired):
$ sudo dmesg -tw
...
IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:5a:8f:75:c1:61:89:08:00 SRC=192.0.2.0 DST=255.255.255.255 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=61423 SEQ=1 MARK=0x220
Versus the nftables backend:
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/32 mark set=0x200/0x3f00"
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/32 mark set=0x20/0xf0"
$ sudo nft add rule inet firewalld mangle_PRE_public_allow ip saddr 192.0.2.0 log
$ sudo nft list ruleset
...
chain mangle_PRE_public_allow {
ip saddr 192.0.2.0 meta mark set meta mark & 0x00003f00 ^ 0x00000200
ip saddr 192.0.2.0 meta mark set meta mark & 0x000000f0 ^ 0x00000020
ip saddr 192.0.2.0 log
}
...
On another machine:
$ sudo ip address add dev br-trunk.16 192.0.2.0/32
$ ping -nc1 -I 192.0.2.0 -b 255.255.255.255
Back on the test machine (note the MARK=0x20, where 0x200 was discarded by the second rule):
$ sudo dmesg -tw
...
IN=enp16s0 OUT= MAC=ff:ff:ff:ff:ff:ff:5a:8f:75:c1:61:89:08:00 SRC=192.0.2.0 DST=255.255.255.255 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=61424 SEQ=1 MARK=0x20
Again note that I said the bits were being toggled, meaning if you add a second mark set=0x20/0xf0 rule and a logging rule, you will see that the two mark set=0x20/0xf0 rules cancel each other out, producing a meta mark of 0 in the second log rule output:
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/31 mark set=0x20/0xf0"
$ sudo nft add rule inet firewalld mangle_PRE_public_allow ip saddr 192.0.2.0 log
$ sudo nft list ruleset
...
chain mangle_PRE_public_allow {
ip saddr 192.0.2.0 meta mark set meta mark & 0x00003f00 ^ 0x00000200
ip saddr 192.0.2.0 meta mark set meta mark & 0x000000f0 ^ 0x00000020
ip saddr 192.0.2.0 log
ip saddr 192.0.2.0/31 meta mark set meta mark & 0x000000f0 ^ 0x00000020
ip saddr 192.0.2.0 log
}
On another machine:
$ ping -nc1 -I 192.0.2.0 -b 255.255.255.255
Back on the test machine (note the lack of MARK=0x20 on the second log):
$ sudo dmesg -tw
...
IN=enp16s0 OUT= MAC=ff:ff:ff:ff:ff:ff:5a:8f:75:c1:61:89:08:00 SRC=192.0.2.0 DST=255.255.255.255 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=61434 SEQ=1 MARK=0x20
IN=enp16s0 OUT= MAC=ff:ff:ff:ff:ff:ff:5a:8f:75:c1:61:89:08:00 SRC=192.0.2.0 DST=255.255.255.255 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=61434 SEQ=1
The workaround is to invert the mask on the firewalld rules when FirewallBackend=nftables:
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/32 mark set=0x200/$((0x3f00^0xffffffff))"
$ sudo firewall-cmd --add-rich-rule="rule family=ipv4 source address=192.0.2.0/32 mark set=0x20/$((0xf0^0xffffffff))"
$ sudo nft add rule inet firewalld mangle_PRE_public_allow ip saddr 192.0.2.0 log
$ sudo nft list ruleset
...
chain mangle_PRE_public_allow {
ip saddr 192.0.2.0 meta mark set meta mark & 0xffffc2ff | 0x00000200
ip saddr 192.0.2.0 meta mark set meta mark & 0xffffff2f | 0x00000020
ip saddr 192.0.2.0 log
}
...
On another machine:
$ sudo ip address add dev br-trunk.16 192.0.2.0/32
$ ping -nc1 -I 192.0.2.0 -b 255.255.255.255
Back on the test machine (note the MARK=0x220, which is desired):
$ sudo dmesg -tw
...
IN=enp16s0 OUT= MAC=ff:ff:ff:ff:ff:ff:5a:8f:75:c1:61:89:08:00 SRC=192.0.2.0 DST=255.255.255.255 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=61426 SEQ=1 MARK=0x220
This command was used to set bits in
0x3f00(bits 8-13) of inbound packets to0x200(000010):With the iptables backend, we get
--set-xmark 0x200/0x3f00:This means:
which is equivalent to:
This AND's the meta mark with the inverse of our mask, discarding the bits in
0x3f00, then XOR's that value with0x200. This effectively sets only bits0x3f00to0x200, while leaving all bits outside of0x3f00intact.The AND with the mask inverse is also confirmed by the
iptables-extensions(8)manual:However, with the nftables backend, we get the following rule:
Which means:
This AND's the meta mark with the mask (not its inverse), keeping only bits
0x3f00, then XOR's that value with0x200. This effectively toggles (not sets) bits0x200and sets all bits outside of0x3f00to zero.In the event that we're starting with a meta mark of all zeroes, both implementations of the rule have the same effect of setting the overall meta mark value to
0x200, and any routing rules matchingfwmark 0x200/0x3f00will work by coincidence, but that's about it. Chaining these sorts of rules to set different bits in a meta mark won't work the same way between iptables and nftables backends.For example, when using the iptables backend:
On another machine:
Back on the test machine (note the
MARK=0x220, which is desired):Versus the nftables backend:
On another machine:
Back on the test machine (note the
MARK=0x20, where0x200was discarded by the second rule):Again note that I said the bits were being toggled, meaning if you add a second
mark set=0x20/0xf0rule and a logging rule, you will see that the twomark set=0x20/0xf0rules cancel each other out, producing a meta mark of 0 in the second log rule output:On another machine:
Back on the test machine (note the lack of
MARK=0x20on the second log):The workaround is to invert the mask on the firewalld rules when
FirewallBackend=nftables:On another machine:
Back on the test machine (note the
MARK=0x220, which is desired):