ecmp nh group create failed with error Feature unavailable

[tomvil]

Description

After enabling ECMP in FRR containers starts to crash, in the logs we can see the following error: ecmp nh group create failed with error Feature unavailable

Full log:

2025 Jan 20 14:23:09.639219 sonic ERR syncd#syncd: [none] SAI_API_NEXT_HOP_GROUP:brcm_sai_xgs_nexthop_group_create:113 ecmp nh group create failed with error Feature unavailable (0xfffffff0).
2025 Jan 20 14:23:09.639219 sonic ERR syncd#syncd: [none] SAI_API_NEXT_HOP_GROUP:brcm_sai_create_next_hop_group:173 pd nexthop group create failed failed with error -2.
2025 Jan 20 14:23:09.639219 sonic ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_NOT_SUPPORTED
2025 Jan 20 14:23:09.640108 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_NEXT_HOP_GROUP_ATTR_TYPE: SAI_NEXT_HOP_GROUP_TYPE_DYNAMIC_ORDERED_ECMP
2025 Jan 20 14:23:09.640684 sonic ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED
2025 Jan 20 14:23:09.640684 sonic ERR swss#orchagent: :- addNextHopGroup: Failed to create next hop group 169.254.0.1@Ethernet48,169.254.0.1@Ethernet52, rv:-2
2025 Jan 20 14:23:09.640745 sonic ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_NEXT_HOP_GROUP, status: SAI_STATUS_NOT_SUPPORTED

Steps to reproduce the issue:

1. Configure two BGP neighbors
2. Enable ECMP
3. Containers will start to crash

BGP Config snippet:

router bgp 1
 bgp router-id 10.10.10.10
 no bgp ebgp-requires-policy
 bgp bestpath as-path multipath-relax
 neighbor Ethernet48 interface remote-as external
 neighbor Ethernet52 interface remote-as external

If bgp bestpath as-path multipath-relax is disabled everythings works fine.

Describe the results you received:

After enabling ECMP in BGP configuration containers start to crash and switch becomes unusable

Describe the results you expected:

After enabling ECMP switch continues to operate as expected

Output of show version:

SONiC Software Version: SONiC.202405.0-dirty-20240916.120509
SONiC OS Version: 12
Distribution: Debian 12.7
Kernel: 6.1.0-11-2-amd64
Build commit: ee93bc7bd
Build date: Mon Sep 16 09:11:13 UTC 2024
Built by: netop@netop

Platform: x86_64-accton_as7326_56x-r0
HwSKU: Accton-AS7326-56X
ASIC: broadcom
ASIC Count: 1
Hardware Revision: N/A
Uptime: 14:29:39 up 43 min,  1 user,  load average: 0.70, 0.83, 0.98
Date: Mon 20 Jan 2025 14:29:39

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[tomvil]

After further investigation, we found that setting l3_ecmp_levels=2 in the ASIC configuration resolves the issue, and we are no longer able to reproduce it.

However, I still have a few questions:

• Is it a proper solution?
• Why is it not enabled by default on devices that support it?