Upgrade pyroute2 and improve cli response time

[vivekrnv]

What I did

• Older pyroute2 depends on distutils. Thus upgrade the version to latest to improve import time. A similar issue for natsort is reported here [warm-reboot] natsort import is taking more time with python 3.11 in bookworm sonic-buildimage#17246. However pyroute2 import is still heavy in bookworm and thus every CLI command is slow compared to bullseye.

<Bullseye>
root@msn2700:/home/admin# time python3 -c "import pyroute2"
real	0m0.378s
user	0m0.308s
sys	0m0.060s

<Before upgrade>
root@msn2700:/home/admin# time python3 -c "import pyroute2"
real        0m0.707s
user        0m0.425s
sys        0m0.097s

<After upgrade>
root@msn2700:/home/admin#  time python3 -c "import pyroute2"
real	0m0.511s
user	0m0.433s
sys	0m0.075s

• To fix this, i've delayed the pyroute2 import into the method where it is actually used, this has an improvement of 0.4 sec for all the CLI commands on slower CPU devices

root@msn2700:/home/admin#  time python3 -c "import utilities_common.cli as clicommon"
real	0m0.693s
user	0m0.579s
sys	0m0.109s

root@msn2700/home/admin#  time python3 -c "import utilities_common.cli as clicommon"
real	0m0.363s
user	0m0.271s
sys	0m0.072s

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

[vivekrnv]

@qiluo-msft Requesting waiver to ignore the coverage failure.

I tried adding an UT to cover the multi_asic_get_ip_intf_from_ns and multi_asic_get_ip_intf_addr_from_ns methods (50e311a) .

Test fails with this error:
    def test_masic_get_ip_intf_and_address(self):
        import utilities_common
        with patch("utilities_common.multi_asic.netifaces.interfaces") as mocked_object:
            mocked_object.return_value = ["Ethernet0"]
>           assert utilities_common.multi_asic.multi_asic_get_ip_intf_from_ns('') == ["Ethernet0"]
E           AssertionError: assert ['eth0', 'Ethernet0', 'PortChannel0001', 'Vlan100', 'lo'] == ['Ethernet0']
E             At index 0 diff: 'eth0' != 'Ethernet0'
E             Left contains 4 more items, first extra item: 'Ethernet0'
E             Full diff:
E             - ['Ethernet0']
E             + ['eth0', 'Ethernet0', 'PortChannel0001', 'Vlan100', 'lo']

This is because the monkey patched API is called instead of the actual API

sonic-utilities/tests/mock_tables/mock_single_asic.py

Line 61 in 5eb266a

def mock_single_asic_get_ip_intf_from_ns(namespace):

def mock_single_asic_get_ip_intf_from_ns(namespace):
    interfaces = []
    try:
        interfaces = list(mock_intf_table[namespace].keys())
        if add_unknown_intf:
            interfaces.append("unknownintf")
    except KeyError:
        pass
    return interfaces

multi_asic_util.multi_asic_get_ip_intf_from_ns = mock_single_asic_get_ip_intf_from_ns
multi_asic_util.multi_asic_get_ip_intf_addr_from_ns = mock_single_asic_get_ip_intf_addr_from_ns

Thus i couldn't add a UT to cover the lines i've added. This PR just changes the location of import and so no major functionality change that warrants a UT

[qiluo-msft]

Thanks for the UT effort! Could you further explore if you can just disable multi_asic_get_ip_intf_from_ns monkey patch in your specific testcase? We should not globally enable one function monkey patch, because it will prevent us to UT the the function itself.

---

In reply to: 2330382035

[vivekrnv]

Thanks for the UT effort! Could you further explore if you can just disable multi_asic_get_ip_intf_from_ns monkey patch in your specific testcase? We should not globally enable one function monkey patch, because it will prevent us to UT the the function itself.

In reply to: 2330382035

Hi, i checked but monkey patching is global. once the file is imported and loaded, the original api is lost.

I suppose this can be updated but changing this would also probably require some other Multi Asic UT's to also be updated and so i think this should be taken as a separate item

[bingwang-ms]

@vivekrnv Can you raise a PR to do the cherry-pick to 202405 since automation is not working?

[mssonicbld]

Cherry-pick PR to 202405: #3640