Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Monitoring Agent 1.30 and 1.31 does not work on Alma Linux #1928

Open
photomorre opened this issue Jun 17, 2024 · 10 comments
Open

Azure Monitoring Agent 1.30 and 1.31 does not work on Alma Linux #1928

photomorre opened this issue Jun 17, 2024 · 10 comments

Comments

@photomorre
Copy link

When upgrading to 1.30 or 1.31 the Azure Monitoring Agent fail on Alma Linux 9.3 and 9.4 with the following errors:

May 31 12:21:22 db amacoreagent[5430]: The required instruction sets are not supported by the current CPU.
May 31 12:21:22 db systemd[1]: Started Process Core Dump (PID 5432/UID 0).
May 31 12:21:23 db systemd-coredump[5433]: Resource limits disable core dumping for process 5430 (amacoreagent).
May 31 12:21:23 db systemd-coredump[5433]: Process 5430 (amacoreagent) of user 0 dumped core.
May 31 12:21:23 db systemd[1]: [email protected]: Deactivated successfully.
May 31 12:21:23 db systemd[1]: azuremonitor-coreagent.service: Main process exited, code=dumped, status=6/ABRT
May 31 12:21:23 db systemd[1]: azuremonitor-coreagent.service: Failed with result 'core-dump'.

The issue was resolved by downgrading to 1.29.6.
(Please make it possible to downgrade from Azure Arc/Extension if you continue to wreck newer versions. 😀 )

The servers in questions are Azure Arc-enabled standard minimal Alma Linux 9.3/9.4 running on VMware ESXi in an on-premises datacenter.

@netoben
Copy link

netoben commented Jun 19, 2024

I have the same issue, with the same scenario , Arc-enabled alma linux 9.4 running on ESXi on premise

========
Jun 19 16:24:41 systemd-coredump[17441]: Resource limits disable core dumping for process 17438 (amacoreagent).
Jun 19 16:24:41 systemd-coredump[17441]: [🡕] Process 17438 (amacoreagent) of user 0 dumped core.
░░ Subject: Process 17438 (amacoreagent) dumped core
░░ Defined-By: systemd
░░ Support: https://wiki.almalinux.org/Help-and-Support
░░ Documentation: man:core(5)
░░
░░ Process 17438 (amacoreagent) crashed and dumped core.
░░
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.
Jun 19 16:24:41 systemd[1]: [email protected]: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://wiki.almalinux.org/Help-and-Support
░░
░░ The unit [email protected] has successfully entered the 'dead' state.
Jun 19 16:24:41systemd[1]: azuremonitor-coreagent.service: Main process exited, code=dumped, status=6/ABRT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://wiki.almalinux.org/Help-and-Support
░░
░░ An ExecStart= process belonging to unit azuremonitor-coreagent.service has exited.
░░
░░ The process' exit code is 'dumped' and its exit status is 6.
Jun 19 16:24:41 systemd[1]: azuremonitor-coreagent.service: Failed with result 'core-dump'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://wiki.almalinux.org/Help-and-Support
░░
░░ The unit azuremonitor-coreagent.service has entered the 'failed' state with result 'core-dump'.

@just-oblivious
Copy link

We're seeing the same error message and restart loop with AMA 1.31 running on Debian on VMWare vSphere.

$ journalctl -u azuremonitor-coreagent -o cat
Started Azure Monitor Agent CoreAgent daemon (on systemd).
The required instruction sets are not supported by the current CPU.
azuremonitor-coreagent.service: Main process exited, code=killed, status=6/ABRT
azuremonitor-coreagent.service: Failed with result 'signal'.
azuremonitor-coreagent.service: Scheduled restart job, restart counter is at 264654.
Stopped Azure Monitor Agent CoreAgent daemon (on systemd).
$ dmidecode -t system
System Information
	Manufacturer: VMware, Inc.
	Product Name: VMware Virtual Platform

$ uname -m
x86_64

@jantekb
Copy link

jantekb commented Jul 8, 2024

These new versions are not listed in the documentation at all https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-extension-versions - who can take care of updating that?

@matheusquei
Copy link

When upgrading to 1.30 or 1.31 the Azure Monitoring Agent fail on Alma Linux 9.3 and 9.4 with the following errors:

May 31 12:21:22 db amacoreagent[5430]: The required instruction sets are not supported by the current CPU.
May 31 12:21:22 db systemd[1]: Started Process Core Dump (PID 5432/UID 0).
May 31 12:21:23 db systemd-coredump[5433]: Resource limits disable core dumping for process 5430 (amacoreagent).
May 31 12:21:23 db systemd-coredump[5433]: Process 5430 (amacoreagent) of user 0 dumped core.
May 31 12:21:23 db systemd[1]: [email protected]: Deactivated successfully.
May 31 12:21:23 db systemd[1]: azuremonitor-coreagent.service: Main process exited, code=dumped, status=6/ABRT
May 31 12:21:23 db systemd[1]: azuremonitor-coreagent.service: Failed with result 'core-dump'.

The issue was resolved by downgrading to 1.29.6. (Please make it possible to downgrade from Azure Arc/Extension if you continue to wreck newer versions. 😀 )

The servers in questions are Azure Arc-enabled standard minimal Alma Linux 9.3/9.4 running on VMware ESXi in an on-premises datacenter.

How did you downgrade?

@martinnymand
Copy link

When upgrading to 1.30 or 1.31 the Azure Monitoring Agent fail on Alma Linux 9.3 and 9.4 with the following errors:

May 31 12:21:22 db amacoreagent[5430]: The required instruction sets are not supported by the current CPU.
May 31 12:21:22 db systemd[1]: Started Process Core Dump (PID 5432/UID 0).
May 31 12:21:23 db systemd-coredump[5433]: Resource limits disable core dumping for process 5430 (amacoreagent).
May 31 12:21:23 db systemd-coredump[5433]: Process 5430 (amacoreagent) of user 0 dumped core.
May 31 12:21:23 db systemd[1]: [email protected]: Deactivated successfully.
May 31 12:21:23 db systemd[1]: azuremonitor-coreagent.service: Main process exited, code=dumped, status=6/ABRT
May 31 12:21:23 db systemd[1]: azuremonitor-coreagent.service: Failed with result 'core-dump'.

The issue was resolved by downgrading to 1.29.6. (Please make it possible to downgrade from Azure Arc/Extension if you continue to wreck newer versions. 😀 )
The servers in questions are Azure Arc-enabled standard minimal Alma Linux 9.3/9.4 running on VMware ESXi in an on-premises datacenter.

How did you downgrade?

I am having the same issue on Ubuntu 20.04 and 22.04. You can downgrade using an ARM template - I don't think it is possible using the portal. The resource should look something like this:

{
    "name": "[concat(parameters('vmName'),'/AzureMonitorLinuxAgent')]",
    "type": "Microsoft.HybridCompute/machines/extensions",
    "location": "[parameters('location')]",
    "apiVersion": "2021-05-20",
    "properties": {
        "publisher": "Microsoft.Azure.Monitor",
        "type": "AzureMonitorLinuxAgent",
        "typeHandlerVersion": "1.29",
        "autoUpgradeMinorVersion": false,
        "enableAutomaticUpgrade": false
    }
}

@jonathanbourke
Copy link

I too have the issue on Ubuntu 22:04. Did the arm template resolve it? IF so, would you mind sharing as the code above did not work for me.

@martinnymand
Copy link

Version 1.29 works in the sense, that the agent doesn't repeatedly crash. Version 1.29 has other bugs, but that may not be an issue for you. You can see a complete ARM template here:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "vmName": {
            "type": "string"
        },
        "location": {
            "type": "string"
        }
    },
    "resources": [
        {
            "name": "[concat(parameters('vmName'),'/AzureMonitorLinuxAgent')]",
            "type": "Microsoft.HybridCompute/machines/extensions",
            "location": "[parameters('location')]",
            "apiVersion": "2021-05-20",
            "properties": {
                "publisher": "Microsoft.Azure.Monitor",
                "type": "AzureMonitorLinuxAgent",
                "typeHandlerVersion": "1.29",
                "autoUpgradeMinorVersion": false,
                "enableAutomaticUpgrade": false
            }
        }
    ]
}

Also notice that according to this page there should be a version 1.32.2 out, but it doesn't seem to be available - at least not to me.

@jonathanbourke
Copy link

jonathanbourke commented Aug 6, 2024 via email

@photomorre
Copy link
Author

photomorre commented Aug 20, 2024

How did you downgrade?

az connectedmachine extension update \
  --extension-name AzureMonitorLinuxAgent \
  --machine-name  <my_name> \
  --resource-group <my_rg> \
  --publisher Microsoft.Azure.Monitor \
  --auto-upgrade 0 \
  --auto-upgrade-minor 0 \
  --type AzureMonitorLinuxAgent \
  --type-handler-version 1.29.6

I just tried to upgrade to 1.32.5 but got the following error:

(HCRP400) The requested extension version is either unavailable or incompatible with the current agent. If the extension is available, upgrade azcmagent to 1.40 or later version and try again.
Code: HCRP400

And I am running the latest version of azcmagent:
Command: azcmagent version
Output: azcmagent version 1.45.02769.1675

@martinnymand
Copy link

@photomorre Microsoft released a version 1.32.6 that actually works, so you can upgrade to that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants