Inaccurate CPU usage on multi-threaded processes

[transkatgirl]

When a multi-threaded process is putting a low load on many cores, this library will show the CPU usage as 100%.

[soyuka]

ping @simonepri

[simonepri]

Can you better describe what you mean by "heavily multi-threaded"?

Please provide:

1. Your OS
2. Your CPU
3. Your node version
4. If you can, a screen of the real resource utilization vs the one reported by pidusage.
   Thank You!

[transkatgirl]

OS : Linux 4.16.12-1-ARCH (Arch Linux with kernel 4.16.12)
CPU : Ryzen 7 1800x (8 cores, 16 threads)
NodeJS Version : v10.3.0

The screenshot is attached below. CPU usage in the Electron program (bottom right) is being read using the pidusage package, real CPU usage is displayed as 15% in the system's task manager (top left).

[simonepri]

Can you please also provide the output of:

ps -o etime,pid,ppid,pcpu,rss,time

Thank you!

[transkatgirl]

I ran that command with the PID of the process being monitored supplied as an argument (19918), and this is the output :

ELAPSED   PID  PPID %CPU   RSS     TIME
   03:37 19918 19854  240 39956 00:08:41

[simonepri]

Hi @kittyhacker101!
Thank You.
Does the formula 240 / 16 = 15 % return the CPU percentage that you would expect to see?

---

@soyuka the man page does not talk about multi-threaded process but I guess that the percentage returned is up to vcore * 100.

If that is true for all platforms, the correct formula would be:
(pcpu / vcore)
Where vcore is os.cpus().length and pcpu is the value reported by ps.
https://github.com/soyuka/pidusage/blob/master/lib/ps.js#L101

[soyuka]

@simonepri I guess that the formula there is indeed correct! Do you think that we should patch the library to take this into consideration? Thing is, sometimes you're monitoring a single-threated process (ie javascript thread) therefore it should not use the above formula, or results won't reflect the reality...

[simonepri]

I believe that we need this #57
And then we can return something like that:

{
  cpu: pcpu / os.cpus().length,               // Percentage (0 - 100)
  threads: 5,                                 // Number of threads of the process
  memory: 357306368,                          // bytes
  ppid: 312,                                  // PPID
  pid: 727,                                   // PID
  ctime: 867000,                              // ms user + system time
  elapsed: 6650000,                           // ms since the start of the process
  timestamp: 864000000                        // ms since epoch
}

Then for instance if pidusage returns something like that:

{
  cpu: 25,
  threads: 2,
  [...]
}

And our system has 4 vcore.

We can interpret the reading as:
The process is consuming the 25% of all the computing power. (100/400)
Either by consuming 100% on a single vcore or X% and 100-X% on two different vcore.

[simonepri]

@soyuka
Can you better explain the following?

Thing is, sometimes you're monitoring a single-threated process (ie javascript thread) therefore it should not use the above formula, or results won't reflect the reality...

[transkatgirl]

I modified the code in the library, replacing the existing CPU usage formula with the one you specified above, and now it does this (process was using 0% CPU in the system's task manager).

[transkatgirl]

So, I wrote my own snippet to detect CPU usage ( cpu / (os.cpus().length * 100) * 100), and it shows a value much closer to what the system task manager says.

[simonepri]

Yes sorry, distraction error, actually the correct formula should be just:
cpu / os.cpus().length
We will work on this as soon as possible.

I've updated all the comments with the correct formula.

[transkatgirl]

Oh, okay.

[soyuka]

Can you better explain the following?

It's silly after a second though.

It'd be great to have the thread count as an additional information indeed.. Maybe we could partially support them on OSes that work? Anyway thanks for the report @kittyhacker101 and for looking into this @simonepri !

[soyuka]

Available in 2.0.7! Thanks to both of you!

[simonepri]

@kittyhacker101 can you confirm that the bug is solved?

[transkatgirl]

The bug in question is solved, but due to this library's averaging of the CPU usage, it takes a while for it to show the correct value (making it less useful).

[simonepri]

Could you better explain what "a while" means? It should be really fast on linux.

[transkatgirl]

By "a while", I mean it could take 2+ minutes to adjust to the correct CPU usage.

[simonepri]

That's weird.
It should be as fast as your activity monitor.
Under the hood it uses ps so it should be fast and "accurate".
Could you see if the values given by ps matches with the one given by pidusage?

Thank You for the help!

[lengband]

hi, i hava a querstion about cpu in lib/ps.js.
Why is it like this var cpu = parseFloat(line[3].replace(',', '.'), 10) / os.cpus().length in version 2.0.7, but find code var cpu = parseFloat(line[3].replace(',', '.'), 10) in version 2.0.8