Error on host.isHyperThreadingEnabled - unable to reboot host from XO

[RemiNetEU]

Are you using XOA or XO from the sources?

XO from the sources

Which release channel?

latest

Provide your commit number

78fc8

Describe the bug

After installation of latest XCP-NG patches I can't perform smart reboot. Force reboot works ok.

Error message

{
  "code": "-1",
  "params": [
    "list index out of range",
    "",
    "Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper\n    return func(*args, **kwds)\n  File \"/etc/xapi.d/plugins/hyperthreading.py\", line 17, in get_hyperthreading\n    return json.dumps(int(lines[0]) > 1)\nIndexError: list index out of range\n"
  ],
  "call": {
    "method": "host.call_plugin",
    "params": [
      "OpaqueRef:28e620c7-ca25-270a-456d-9562e156d4ee",
      "hyperthreading.py",
      "get_hyperthreading",
      {}
    ]
  },
  "message": "-1(list index out of range, , Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper\n    return func(*args, **kwds)\n  File \"/etc/xapi.d/plugins/hyperthreading.py\", line 17, in get_hyperthreading\n    return json.dumps(int(lines[0]) > 1)\nIndexError: list index out of range\n)",
  "name": "XapiError",
  "stack": "XapiError: -1(list index out of range, , Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper\n    return func(*args, **kwds)\n  File \"/etc/xapi.d/plugins/hyperthreading.py\", line 17, in get_hyperthreading\n    return json.dumps(int(lines[0]) > 1)\nIndexError: list index out of range\n)\n    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202404231231/packages/xen-api/_XapiError.mjs:16:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202404231231/packages/xen-api/transports/json-rpc.mjs:38:21"
}

To reproduce

1. Go to '...'
2. Click on '...'
3. Scroll down to '...'
4. See error

Expected behavior

No response

Screenshots

No response

Node

18

Hypervisor

XCP-ng 8.3.0

Additional context

No response

[benjamreis]

Hi!

Thanks for the report.
Could you post here the return of the command: xl info threads_per_core on your host where you encountered the issue please?

Thanks

[RemiNetEU]

After force restart all works fine. So far I've installed latest XCP-NG updates on two hosts with same result - unable to perform smart reboot, access console of the VMs or stop VMs.
Here is the result after force restart.

[benjamreis]

When you say all works fine, does this mean that smart reboot as well?
If you still have an host you have not force rebooted yet can you try the same command on this host?

[RemiNetEU]

Yes, after force restart smart reboot works again. What I've also noticed that after installing those XCP-NG updates (beside issues mentioned above), RAM showed full use (0% free) and Hyperthreading as disabled.
I still have two hosts to upgrade but can't do that before 8PM (CET)

[benjamreis]

Okay so when you do, please before force rebooting run:
xl info threads_per_core
and
xe host-call-plugin host-uuid=<uuid of your host> plugin=hyperthreading.py fun=get_hyperthreading and post the results here.

Latest changes bump the Xen version of XCP-ng and I suspect there's something there that perturbate xl info or something, that is fixed after a reboot.
I'll look into it meanwhile.

Thx!

[RemiNetEU]

xl info threads_per_core gives:

[20:07 xcp1 ~]# xl info threads_per_core
libxl: error: libxl_utils.c:815:libxl_cpu_bitmap_alloc: failed to retrieve the maximum number of cpus
libxl: error: libxl_utils.c:815:libxl_cpu_bitmap_alloc: failed to retrieve the maximum number of cpus
libxl: error: libxl_utils.c:815:libxl_cpu_bitmap_alloc: failed to retrieve the maximum number of cpus
libxl: error: libxl.c:361:libxl_get_physinfo: getting physinfo: Permission denied
libxl_physinfo failed.
libxl: error: libxl_sched.c:166:libxl_get_scheduler: getting current scheduler id: Permission denied
get_scheduler sysctl failed.

and xe host-call-plugin host-uuid=<uuid of your host> plugin=hyperthreading.py fn=get_hyperthreading gives

[20:10 xcp1 ~]# xe host-call-plugin host-uuid=23b2e075-1726-4960-ae09-9ee01fa2590e plugin=hyperthreading.py fn=get_hyperthreading
Error code: -1
Error parameters: list index out of range, , Traceback (most recent call last):
  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 119, in wrapper
    return func(*args, **kwds)
  File "/etc/xapi.d/plugins/hyperthreading.py", line 17, in get_hyperthreading
    return json.dumps(int(lines[0]) > 1)
IndexError: list index out of range

[benjamreis]

Perfect! Thank you for the info.

As I suspected something is wrong at Xen level after the upgrade.
Fortunately a force reboot solves everything.
I'll talk to this with the team.

Regards

[julien-f]

This issue should not impact Smart Reboot, can you show me a log related to this problem?

[benjamreis]

Now for 8.3 and more the info is stored directly in XAPI in the host's cpu_info - the plugin is no longer needed and so upgrade to Xen 4.17 is no longer problematic! 🎉