Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CPU resource monitor for Linux systems #4888

Merged
merged 23 commits into from
Aug 14, 2024
Merged
Changes from 2 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
efcca1e
Replace free call on linux
vmapetr Jul 8, 2024
c66abe8
Merge branch 'master' into v-mpetrov/us-2190905-improve-resource-monitor
vmapetr Jul 10, 2024
1e59693
Minor fixes
vmapetr Jul 10, 2024
ea9dfae
Rework CPU info for linux
vmapetr Jul 10, 2024
cf1a22f
Fix typos
vmapetr Jul 10, 2024
ef9e60b
Fix issue with memory monitor
vmapetr Jul 11, 2024
f6ec823
Add comments
vmapetr Jul 11, 2024
e657974
Remove task run delegates
vmapetr Jul 12, 2024
5cda622
Fix for macos
vmapetr Jul 13, 2024
3e053ad
Merge branch 'master' into v-mpetrov/us-2190905-improve-resource-monitor
vmapetr Jul 15, 2024
ffb29f7
Expand comments for metrics methods
vmapetr Jul 15, 2024
ffb239b
Remove unnecessary cancellation exits
vmapetr Jul 15, 2024
621c110
Merge branch 'master' into v-mpetrov/us-2190905-improve-resource-monitor
vmapetr Jul 18, 2024
dbd552c
Add FF to disable resource monitor on debug runs
vmapetr Jul 18, 2024
f053b8d
Minor fix
vmapetr Jul 18, 2024
ca67e50
Fix warning
vmapetr Jul 18, 2024
dc2f5a2
Merge branch 'master' into v-mpetrov/us-2190905-improve-resource-monitor
vmapetr Jul 18, 2024
7a583be
Merge branch 'master' into v-mpetrov/us-2190905-improve-resource-monitor
vmapetr Jul 31, 2024
677ddba
Merge branch 'master' into v-mpetrov/us-2190905-improve-resource-monitor
vmapetr Aug 12, 2024
79022e0
Replace DisableResourceMonitorDebugOutput with EnableResourceMonitorD…
vmapetr Aug 12, 2024
f6e0041
Fix typo
vmapetr Aug 13, 2024
ea4e9b4
Update variable name
vmapetr Aug 13, 2024
9c0f83a
Fix typo
vmapetr Aug 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 7 additions & 46 deletions src/Agent.Worker/ResourceMetricsManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -286,65 +286,26 @@ await Task.Run(() =>

if (PlatformUtil.RunningOnLinux)
{
// Some compact Linux distributions like UBI may not have "free" utility installed, or it may have a custom output
// We don't want to break currently existing pipelines with ADO warnings
// so related errors thrown here will be sent to the trace or debug logs by caller methods

using var processInvoker = HostContext.CreateService<IProcessInvoker>();

using var timeoutTokenSource = new CancellationTokenSource();
timeoutTokenSource.CancelAfter(TimeSpan.FromMilliseconds(METRICS_UPDATE_INTERVAL));

using var linkedTokenSource = CancellationTokenSource.CreateLinkedTokenSource(
_context.CancellationToken,
timeoutTokenSource.Token);

processInvoker.OutputDataReceived += delegate (object sender, ProcessDataReceivedEventArgs message)
await Task.Run(() =>
{
if (!message.Data.StartsWith("Mem:"))
{
return;
}

var processInvokerOutputString = message.Data;
var memoryInfoString = processInvokerOutputString.Split(" ", StringSplitOptions.RemoveEmptyEntries);

if (memoryInfoString.Length != 7)
{
throw new Exception("\"free\" utility has non-default output");
}
string memoryInfo = File.ReadAllText("/proc/meminfo");
vmapetr marked this conversation as resolved.
Show resolved Hide resolved
int totalMemory = int.Parse(memoryInfo.Split('\n')[0].Split(" ", StringSplitOptions.RemoveEmptyEntries)[1]);
int freeMemory = int.Parse(memoryInfo.Split('\n')[1].Split(" ", StringSplitOptions.RemoveEmptyEntries)[1]);
vmapetr marked this conversation as resolved.
Show resolved Hide resolved

lock (_memoryInfoLock)
{
_memoryInfo.Updated = DateTime.Now;
_memoryInfo.TotalMemoryMB = long.Parse(memoryInfoString[1]);
_memoryInfo.UsedMemoryMB = long.Parse(memoryInfoString[2]);
_memoryInfo.TotalMemoryMB = totalMemory / 1024;
_memoryInfo.UsedMemoryMB = (totalMemory - freeMemory) / 1024;
}
};

processInvoker.ErrorDataReceived += delegate (object sender, ProcessDataReceivedEventArgs message)
{
Trace.Error($"Error on receiving memory info: {message.Data}");
};

try
{
var filePath = "free";
var arguments = "-m";
await processInvoker.ExecuteAsync(
workingDirectory: string.Empty,
fileName: filePath,
arguments: arguments,
environment: null,
requireExitCodeZero: true,
outputEncoding: null,
killProcessOnCancel: true,
cancellationToken: linkedTokenSource.Token);
}
catch (Win32Exception ex)
{
throw new Exception($"\"free\" utility is unavailable. Exception: {ex.Message}");
}
}, linkedTokenSource.Token);
}

if (PlatformUtil.RunningOnMacOS)
Expand Down
Loading