Suricata, Jumbo Frames, and netmap
One of my FreeBSD machines sits between the Internet and the rest of my network.
I wanted Suricata to monitor traffic arriving on four of the machine's interfaces and complain when something looks wrong.
I started with one interface, mce1, and planned to add the other three after capture worked.
The capture interfaces are backed by Mellanox ConnectX adapters handled by the mlx5en driver, and they use a jumbo-frame MTU of 9000 bytes.
On FreeBSD, Suricata has two realistic capture methods: pcap and netmap. With pcap, packets pass from the network card through the normal FreeBSD network stack, and BPF copies them into capture buffers for Suricata. Native netmap can reduce that copying and system-call overhead by exposing packet rings directly to Suricata. I expected netmap to be the faster solution.
When I started Suricata, it stopped while registering mce1:
opening devname netmap:mce1/R failed: Invalid argument
I immediately fired up kdump to see what was happening.
That was an EINVAL from the NIOCCTRL ioctl on /dev/netmap.
The /R suffix means "RX rings only", which is what Suricata requests in intrusion detection system (IDS) mode.
In IDS mode, Suricata observes packets and raises alerts, but unlike an intrusion prevention system (IPS), it does not block them.
This post is the war story of getting from that unhelpful errno to the real problem.
Adjusting Suricata blindfolded
My first capture interface is mce1, which belongs to the mlx5en driver.
Grep the FreeBSD source tree for netmap support in sys/dev/mlx5/ and you will find, well, nothing.
There is no native netmap adapter there.
That does not make netmap refuse to work.
It falls back to the emulated adapter, also called the generic adapter, which intercepts if_input and if_transmit and copies packets between mbufs and netmap buffers.
It works with drivers that do not implement native netmap, at the price of a copy per received packet.
Two defaults matter here:
dev.netmap.generic_rings=1 # TX and RX rings created by each generic adapter
dev.netmap.generic_ringsize=1024 # slots per ring
Suricata can assign one capture thread to each RX ring, so using multiple rings spreads packet capture across multiple threads.
I started with eight rings and 4096 slots per ring:
sysctl dev.netmap.generic_rings=8
sysctl dev.netmap.generic_ringsize=4096
netmap reports useful errors through dmesg
Every error path in the netmap code prints the real reason via nm_prerr(), which wraps the kernel's printf().
netmap returns a single errno, but for debugging we need to look into dmesg.
dmesg | grep -iE "netmap|nm_" | tail
Jumbo frames
I restarted Suricata and got the same EINVAL.
This time I finally read dmesg.
error: netmap buf size (2048) < device MTU (9000)
Before registering the interface, netmap_buf_size_validate() checks that one netmap buffer can hold an MTU-sized packet.
The default buffer was only 2048 bytes, so it could not hold a packet from an interface with an MTU of 9000 bytes.
A 9216-byte buffer is large enough and leaves a little wiggle room.
sysctl dev.netmap.buf_size=9216
ENOMEM
The next attempt returned ENOMEM.
The requested number of buffers in the global pool was still the default:
sysctl dev.netmap.buf_num
dev.netmap.buf_num: 163840
That value comes from NETMAP_BUF_MAX_NUM, whose source comment calls it suitable for a "large machine".
At 2048 bytes per buffer, the pool occupies 320 MiB.
At 9216 bytes per buffer, the same request becomes about 1.4 GiB.
By then, the machine had been running for a while, and netmap could not allocate the memory.
The ordinary sysctls describe the requested count and size, while the *_curr_* values describe what the allocator actually created.
They are worth checking after the first registration attempt because the allocator can end up with fewer objects than requested.
sysctl dev.netmap.buf_num dev.netmap.buf_size
sysctl dev.netmap.buf_curr_num dev.netmap.buf_curr_size
I counted both directions for one interface, added some margin, and reduced the request:
8 rings × 4096 slots × 2 directions = 65536 buffers
sysctl dev.netmap.buf_num=73728
Ring too large
The next run produced a different error:
netmap_ring request size 65792 too large
Buffers are not the only objects in netmap's shared memory. The ring descriptors and their slot arrays come from a separate ring pool.
On this build, a netmap_ring has a 256-byte header and every netmap_slot occupies 16 bytes.
A 4096-slot ring therefore needs 256 + 4096 × 16 = 65792 bytes.
The default ring-pool object size is nine 4096-byte pages, or 36864 bytes. That fits at most 2288 slots, so a 4096-slot ring needs a larger object.
sysctl dev.netmap.ring_size=69632 # 17 pages, enough for a 4096-slot ring
The ring-size change finally let Suricata register mce1.
Four interfaces
With one interface working, I scaled the setup to four Mellanox interfaces, all using jumbo frames with an MTU of 9000 bytes.
no more netmap_buf objects
no more buffers after 4094 of 4096
I initially read this as netmap being two buffers short of completing the entire configuration. That interpretation was completely wrong.
On this attempt, Suricata had already registered earlier interfaces before a later one ran out of buffers. All four generic adapters used the same global netmap pool, so every successful registration reduced what remained for the next interface. Suricata opened each interface in RX-only mode, so only the RX rings consumed buffers from the pool.
The global allocator reserves buffer indices 0 and 1 for internal use in netmap_mem_init_bitmaps().
The pool was consumed in these steps:
# Start with the pool and remove the two reserved buffers
73728 - 2 = 73726
# Allocate eight 4096-slot rings to each of the first two interfaces
73726 - (2 × 8 × 4096) = 8190
# Allocate the first 4096-slot ring to the third interface
8190 - 4096 = 4094
The next ring also needed 4096 buffers, but only 4094 remained. Netmap allocates buffers one ring at a time, so the calculation stops at the ring that failed instead of counting the third interface as complete. The error therefore described a two-buffer shortage for that ring, not for the complete configuration.
For identical RX-only interfaces with no extra buffers, the minimum is:
buf_num ≥ 2 + interfaces × generic_rings × generic_ringsize
The final configuration
The four RX-only interfaces needed at least:
2 + 4 interfaces × 8 RX rings × 4096 slots = 131074 buffers
With 9216-byte buffers, that is about 1.125 GiB before the smaller netmap pools and allocator overhead. Possible, but I did not need rings that large for an adapter that copies every packet anyway.
I reduced both the ring count and the number of slots:
# /etc/sysctl.conf
dev.netmap.buf_size=9216
dev.netmap.buf_num=98304 # 32770 required for these RX rings, with generous headroom
dev.netmap.generic_rings=4
dev.netmap.generic_ringsize=2048 # a 33024-byte ring fits the default 36864-byte object
Four interfaces with four RX rings and 2048 slots need only 32770 buffers from the global pool:
2 reserved buffers + (4 interfaces × 4 RX rings × 2048 slots) = 32770 buffers
The configured 98304-buffer pool occupies 864 MiB and leaves 65534 buffers beyond those rings for other allocations sharing the pool.
Victory
netmap's defaults make sense for 1500-byte MTUs, not four jumbo capture interfaces sharing one pool. Generic netmap is still an emulated path rather than native access to mlx5en's rings, so I should probably check whether it is actually faster than pcap mode.
And just do not forget to read dmesg.