os-frr : OSPF routes reported as installed in the Zebra RIB (O>*) but missing from the kernel FIB after a reboot; temporarily fixed by restarting the frr daemon.

[thibaud-meriel]

Important notices

• I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• I have searched the existing issues, open and closed, and I'm convinced that mine is new.
• The title contains the plugin to which this issue belongs

Describe the bug

After every OPNsense reboot, OSPF converges with no anomaly observed and the Zebra RIB is correctly populated with 74 entries, among them prefixes learned through OSPF and flagged as installed in the FIB (O>*). Nevertheless, the kernel routing table only contains the Directly Attached routes, not a single prefix reported as installed in the FIB shows up in it. A restart of the service (tested in two steps, stop then start) systematically corrects the anomaly encountered, roughly doubling the number of routes (33 before restarting the frr daemon, 62 after). No configuration change is involved, and this phenomenon is reproducible at every OPNsense reboot.

The following CLI commands return the results shown below:

A) After the OPNsense reboot and before restarting the frr daemon:

< route get 0.0.0.0 > = "route: route has not been found".

< vtysh -c "show ip route" | grep 0.0.0.0> = "O>* 0.0.0.0/0 [110/100] via 10.9.26.1, vtnet8, weight 1, 00:00:01".

The Zebra dataplane subsystem reports the detailed elements below.
Information relating to the running dataplane plugins that provide updates to the FIB. The local kernel plugin is present :

< vtysh -c "show zebra dplane providers" > =
"dataplane Incoming Queue from Zebra: 0.
Zebra dataplane providers :
Kernel (1): in: 129, q:0, q_max: 41, out: 129, q: 0, q_max: 41
dataplane Outgoing Queue to Zebra: 0."

Statistics relating to the updates and events passing through the dataplane subsystem :

< vtysh -c "show zebra dplane detailed" | head -22 > =
"Zebra dataplane :
Route updates : 115
Route updates errors : 0
Nexthop updates : 20
Other errors : 0
Route update queue limit : 200
Route update queue depth: 0
Route update queue max: 41
Dplane update yields : 0
LSP updates : 0
LSP update errors : 0
PW updates : 0
PW update errors : 0
Intf addr updates : 0
Intf addr errors : 0
Intf change updates : 0
Intf change errors : 0
EVPN MAC updates : 0
EVPN MAC errors : 0
EVPN neigh updates : 9
EVPN neigh erros : 0".

B) After the OPNsense reboot and after restarting the frr daemon (service frr stop) & (service frr start) :

< route get 0.0.0.0 > =
"route to: default
destination: default
mask: default
gateway: 10.9.26.1
fib: 0
interface vtnet8
flags <UP,GATEWAY,DONE,PROT01>
recvpipe 0
sendpipe 0
ssthresh 0
rtt,msec 0
mtu 1500
weight 1
expire 0".

< vtysh -c "show ip route | grep 0.0.0.0 > = "O>* 0.0.0.0/0 [110/100] via 10.9.26.1, vtnet8, weight 1, 00:06:10".

The Zebra dataplane subsystem reports the detailed elements below.
Information relating to the running data plane plugins that provide updates to the FIB after the frr restart :

< vtysh -c "show zebra dplane providers" > =
"dataplane Incoming Queue from Zebra: 0.
Zebra dataplane providers:
Kernel (1): in: 129, q:0, q_max: 26, out: 129, q: 0, q_max: 26
dataplane Outgoing Queue to Zebra: 0."

Statistics relating to the updates and events passing through the data plane subsystem :

< vtysh -c "show zebra dplane detailed" | head -22 > =
"Zebra dataplane :
Route updates : 115
Route updates errors : 0
Nexthop updates : 20
Nexthop updates errors : 0
Other errors : 0
Route update queue limit : 200
Route update queue depth: 0
Route update queue max: 26
Dplane update yields : 0
LSP updates : 0
LSP update errors : 0
PW updates : 0
PW update errors : 0
Intf addr updates : 0
Intf addr errors : 0
Intf change updates : 0
Intf change errors : 0
EVPN MAC updates : 0
EVPN MAC errors : 0
EVPN neigh updates : 9
EVPN neigh errors : 0".

This does not appear to be the incident described in issue #5422 (#5422), which stemmed from an insufficient maximum size of frr's internal socket buffer, since that one, already encountered previously, was resolved by means of the dynamic tunable kern.ipc.maxsockbuf = 33554432.

To Reproduce

1. Reboot OPNsense.
2. Once the system has rebooted, check that OSPF has converged and that the Zebra RIB is complete (vtysh -c "show ip route"). The OSPF prefixes are present and flagged as installed and selected in the FIB.
3. Compare with the kernel routing table (netstat -r) & (route get 0.0.0.0). Only the directly connected routes are present, and the default route is missing.
4. Restart the frr daemon (service frr stop) & (service frr start) or (service frr restart), then check the kernel routing table again. The OSPF prefixes are now correctly installed.

Expected behavior

When OPNsense reboots, the kernel routing table should be made up of the Directly Attached routes and of the prefixes reported as installed in the FIB by Zebra, and should not require manually restarting the frr daemon in order to remedy this absence from the FIB of the prefixes correctly learned through OSPF. This anomaly is liable to have an impact in production during an unplanned reboot of the OPNsense router/firewall, by lengthening the time needed to restore service since human action is required.

Screenshots

Reboot on console :

Post-reboot phase on console :

Restart frr on console :

Post-reboot phase and restart frr on console :

Relevant log files

frr daemon logs from the OPNsense web console.

Routing -> General -> Log Level Debugging.

Diagnostics -> Log

OPNsense reboot phase :

"2026-08-12T12:01:19 Warning zebra [X55T4-ZADN1][EC 4043309149] Client 25 'ospf' (session id 0) encountered an error and is shutting down.

2026-08-12T12:01:19 Error ospfd [MNSF9-KVB43]_bfd_sess_send: BFD session 10.9.67.1 -> 10.9.67.2 interface vtnet10 VRF default(0) was not uninstalled

2026-08-12T12:01:19 Error ospfd [MNSF9-KVB43]_bfd_sess_send: BFD session 10.9.28.2 -> 10.9.28.1 interface vtnet9 VRF default(0) was not uninstalled

2026-08-12T12:01:19 Error ospfd [MNSF9-KVB43]_bfd_sess_send: BFD session 10.9.26.2 -> 10.9.26.1 interface vtnet8 VRF default(0) was not uninstalled

2026-08-12T12:01:19 Warning zebra [X55T4-ZADN1][EC 4043309149] Client 30 'bfd' (session id 0) encountered an error and is shutting down."

OPNsense post-reboot phase :

2026-08-12T12:02:06 Error zebra [PHQ11-XRDRX][EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS usbus7) failed: Device not configured

2026-08-12T12:02:06 Error zebra [PHQ11-XRDRX][EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS usbus6) failed: Device not configured

2026-08-12T12:02:06 Error zebra [PHQ11-XRDRX][EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS usbus5) failed: Device not configured

2026-08-12T12:02:06 Error zebra [PHQ11-XRDRX][EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS usbus4) failed: Device not configured

2026-08-12T12:02:05 Error zebra [PHQ11-XRDRX][EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS usbus3) failed: Device not configured

2026-08-12T12:02:05 Error zebra [PHQ11-XRDRX][EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS usbus2) failed: Device not configured

2026-08-12T12:02:05 Error zebra [PHQ11-XRDRX][EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS usbus0) failed: Device not configured

OPNsense post-reboot phase and after stopping the daemon (service frr stop) :

2026-08-12T13:01:11 Warning [X55T4-ZADN1][EC 4043309149] Client 25 'ospf' (session id 0) encountered an error and is shutting down.

2026-08-12T13:01:11 Error [MNSF9-KVB43]_bfd_sess_send: BFD session 10.9.67.1 -> 10.9.67.2 interface vtnet10 VRF default(0) was not uninstalled

2026-08-12T13:01:11 Error [MNSF9-KVB43]_bfd_sess_send: BFD session 10.9.28.2 -> 10.9.28.1 interface vtnet9 VRF default(0) was not uninstalled

2026-08-12T13:01:11 Error [MNSF9-KVB43]_bfd_sess_send: BFD session 10.9.26.2-> 10.9.26.1 interface vtnet8 VRF default(0) was not uninstalled

2026-08-12T13:01:11 Warning [X55T4-ZADN1][EC 4043309149] Client 30 'bfd' (session id 0) encountered an error and is shutting down.

No new entry when frr starts.

Additional context

• A single area 0.0.0.0 made up of 3 ASBR routers (only one currently online, two offline for miscellaneous maintenance) and 5 other routers.
• CARP cost demotion (carp-demote).
• Virtualized environment on a type 1 hypervisor, Proxmox V9.2.3.

Environment

OS & Plugin :

FreeBSD 15.1-RELEASE-p1
OPNsense 26.7.1_1
FRRouting 10.6.1

< vtysh -c "show version" > =
"FRRouting 10.6.1 (fw-if.secteur-02.it.opti-info.internal) on FreeBSD ( 15. 1-RELEAS
E-p1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
' -- enable-user=frr' ' -- enable-group=frr' ' -- enable-vty-group=frrvty' ' -- enab
le-vtysh' ' -- disable-doc-html' '-- sysconfdir=/usr/local/etc/frr' ' -- localstatedir=/var/run/frr' ' -- disable-nhrpd' ' -- disable-pathd'
-- disable-ospfclient' ' -- disable-pimd' ' -- disable-pbrd' ' -- with-vtysh-pager=cat ' -- enable-backtrace '--disable-config-rollbacks' '-- disable-datacenter'
'-- enable-fpm' ' -- disable-ldpd' '-- disable-doc' ' -- enable-multipath=64' ' -- without-libpam' ' '-- enable-rpki' '--disable-scripting' ' -- disable-sharpd' ' -- disable-shell-access' ' -- enable-snmp
'--disable-tcmalloc' ' -- prefix=/usr/local' ' -- mandir=/usr/local/share/man' ' -- disable-silent-rules' ' -- infodir=/usr/local/share/info/' ' -- build=amd64-portbld-freebs
d15.1' 'build_alias=amd64-portbld-freebsd15. 1' 'PKG_CONF IG=pkgconf' 'PKG_CONFIG
LIBDIR=/usr/obj/usr/ports/net/frr10/work/.pkgconfig:/usr/local/libdata/pkgconfig
: /usr / local/share/pkgconfig:/usr/libdata/pkgconfig' 'CC=cc' 'LIBS=' 'CPP=cpp' 'C
XX=c++' 'PYTHON=/usr/local/bin/python3. 13'"

VM :

Virtual Disk : QCOW2 / OVMF UEFI.
SCSI Controller : VirtIO SCSI single.
Network Device (13 Network Devices) : VirtIO.
Processors : 2 (1 sockets, 2 cores x86-64-v2-AES.
Memory : 4,00 GiB.

END

Thanks so much for your help.

[Monviech]

Maybe the same kind of problem class as described here?

FRRouting/frr#22418

It feels kinda upstream from what you describe.

[thibaud-meriel]

Hello @Monviech ,

Thank you for your reply and your suggestion. I have therefore reproduced the conditions described by @alexissavin in order to check whether the malfunction reported in Issue #22418 was also affecting my infrastructure.

Procedure :

1. < vtysh -c "show ip route" >. Zebra's RIB shows no anomaly.
2. < netstat -rn >. The kernel routing table is consistent.
3. < ifconfig vtnet8 down >. The interface associated with the default route is shutdown.
4. < vtysh -c "show ip route" >. Zebra's RIB shows no anomaly, the default route is now associated with interface vtnet9, with the third octet at 28 in the class A range.
5. < netstat -rn >. The kernel routing table is consistent with Zebra's RIB.
6. < ifconfig vtnet8 up >. The interface previously associated with the default route is brought back up.
7. < vtysh -c "show ip route" >. Zebra's RIB shows no anomaly, the default route is once again associated with interface vtnet8, with the third octet at 26 in the class A range.
8. < netstat -rn >. The kernel routing table is consistent with Zebra's RIB.

The deactivation of interface vtnet8 was properly taken into account by OSPF, and OSPF reconverged correctly. No "overwriting" of the FRR routes was observed following the loss of an interface.

Enjoy the rest of your day !

Thibaud.

[Monviech]

Did you increase the kern.ipc.maxsockbuf?

https://docs.opnsense.org/manual/dynamic_routing.html

I've heard it can mess with route installation sometimes if its not set to 33554432.

[thibaud-meriel]

@Monviech ,

Yes, before writing this ticket, I had looked into issue #5422 (insufficient maximum internal socket buffer size, resolved by means of the dynamic option kern.ipc.maxsockbuf = 33554432).

Thanks,
Thibaud.

[Monviech]

I don't have any idea right now. Maybe somebody else shares the same symptoms as you.

If you can reproduce it in FreeBSD 15.1 you could also try an upstream report.

[thibaud-meriel]

Thank you for your help @Monviech. I will endeavor to keep you informed on this Issue if I come across any possible leads toward a resolution or the identification of the root cause.