CQ2 - Resource usage

[simleo]

How much memory/cpu/disk was used in run?

• Source entity?
• Property?

[simleo]

First thing is knowing what we're going to model. What does each workflow manager already provide? What could be added that's missing, and how hard would be to do that?

• @mr-c cwltool?
• @ieguinoa Galaxy?
• @GlassOfWhiskey Streamflow?
• @inutano Sapporo?
• @rsirvent COMPSs?
• @lrodrin @jmfernandez WfExS?
• Does anyone know about Snakemake and Nextflow?

More generally, we need to know about any information / logging data (and how structured it is) provided by the framework that runs the workflow, not just resource usage: timestamps, user info, containerization, etc. Another useful categorization is what's available by default and what needs to be explicitly activated (e.g., user info in cwltool).

[mr-c]

cwltool tracks peak memory usage; and start & stop tips for jobs & steps
We don't currently track peak disk usage, nor CPU time (but both could be added)

[simleo]

Discussed with @ilveroluca earlier this morning. One source of confusion was my suggestion of memoryRequirements or storageRequirements for the property names, which was not consistent with the question "How much memory/cpu/disk was used in run". I have now removed that suggestion from the issue's description.

That's not to say that such indications are not useful: they are quite useful to those who want reproduce the run, since they allow to plan ahead, but they've got nothing to do with what happened during the run. Such indications don't come from the observation of a single run, but rather from the experience (or -- even better -- statistics) of the author(s) or anyone who's worked with the application in various scenarios. They are part of prospective, not retrospective provenance, so we should expect them to come from the workflow's author / maintainer. Indeed, CWL has ResourceRequirement for this purpose. I've now opened #32 to track this.

Here, instead, we are focusing on resource usage information for the specific run described in the crate, such as the peak memory usage mentioned by Michael. This isn't just useful to enrich the metadata about the run: it might be the only hint available in all cases where requirements as discussed above are not available (which I expect to be the majority: even when they are known, the authors might not take the time to provide them); additionally, with a sufficiently large number of runs, it could be used to get a good estimate of the general requirements.

[simleo]

cwltool tracks peak memory usage

Where is that recorded? Is it available from the CWLProv output?

[jmfernandez]

About Nextflow, when trace option is enabled, detailed statistics about memory and CPU usage is gathered from each step of execution. The monitoring is done from the shell script created to execute the workflow step. As this script is written in bash and it also depends on additional tools for the detailed monitoring, mainly ps, grep and awk, not all the container instances allow this detailed statistics gathering.

This is the usually gathered information for each executed step:

nextflow.trace/v2
realtime=1361
%cpu=1380
rchar=19914333
wchar=791
syscr=1252
syscw=30
read_bytes=0
write_bytes=4096
%mem=4
vmem=524224
rss=69416
peak_vmem=568880
peak_rss=112264
vol_ctxt=6
inv_ctxt=444

[suecharo]

Sapporo is a Workflow Execution System (WES), so it calls workflow engines (e.g., cwltool and Nextflow) internally.
Here, Sapporo and workflow engines are all running as Docker containers.
Because Sapporo do not use Docker API (we wanted to support other than Docker), it was difficult to get detailed resource information...

RO-Crate generated by Sapporo is stored following information (information in the Sapporo container):

• os
• osVersion
• cpuArchitecture
• cpuCount
• totalMemory
• freeDiskSpace
• uid
• git
• inDocker

These information are generated in https://github.com/sapporo-wes/sapporo-service/blob/856196864e8ccda8c71bef12e5dcf7d5becb21e6/sapporo/ro_crate.py#L697 .
https://github.com/sapporo-wes/tonkaz/blob/a1e4c94c439d6a1d3f9947b519177196f77f1c95/tests/example_crate/trimming.json#L470 is an entity in Sapporo-generated RO-Crate.

Also, we add Log (or provenance) of WES (Sapporo).

• cmd (Execution command of workflow engine)
• start_time
• end_time
• run_pid
• state (WES state, e.g., COMPLETE, EXECUTION_ERROR, etc.)
• stdout (from Workflow engines)
• stderr (from Workflow engines)

These information are originally stored in Sapporo as run_dir (https://github.com/sapporo-wes/sapporo-service#run-dir).
https://github.com/sapporo-wes/tonkaz/blob/a1e4c94c439d6a1d3f9947b519177196f77f1c95/tests/example_crate/trimming.json#L632 is an entity in Sapporo-generated RO-Crate.

Furthermore, we collect information about each files (input/output file)

• contentSize
• dateModified
• uid
• gid
• mode
• lineCount
• sha512
• encodingFormat

These information are generated in https://github.com/sapporo-wes/sapporo-service/blob/856196864e8ccda8c71bef12e5dcf7d5becb21e6/sapporo/ro_crate.py#L318 .
https://github.com/sapporo-wes/tonkaz/blob/a1e4c94c439d6a1d3f9947b519177196f77f1c95/tests/example_crate/trimming.json#L243 is an entity in Sapporo-generate RO-Crate.

[simleo]

From @rsirvent:
example of basic statistics with COMPSs: https://workflowhub.eu/workflows/386 -> App_Profile.json

[rsirvent]

Sorry I missed this thread. Let me elaborate a bit more what we provide with COMPSs. We have some ways to gather statistics / understand resource usage:

• Execution statistics profile (--output_profile flag): this has basic statistics on the resources used by the COMPSs application, how many tasks and of which type were executed on each resource, and maximum, average and minimum execution time for each task. An example, as provided in the previous comment: https://workflowhub.eu/workflows/386 -> App_Profile.json
• Memory profiling (--python_memory_profile): you can see memory usage over time in PyCOMPSs applications (more details at: https://compss-doc.readthedocs.io/en/stable/Sections/04_Troubleshooting/03_Memory_Profiling.html)
• Tracing with Paraver (see https://compss-doc.readthedocs.io/en/stable/Sections/05_Tools/03_Tracing.html): we have the option to activate tracing for an application. Then, the runtime records events, counters... down to hardware counters level. The traces can be opened with the Paraver analysis tool, where you can see in a very detailed manner the status of each thread of the application, communications, ... This is very advanced due to the level of detailed information provided, but very useful to achieve very efficient workflows that exploit the underlying infrastructure to its maximum (which is very important in supercomputers).

Also, more general information of the resources (how many cores, memory, etc... they have) can be found in COMPSs xml configuration files (resources.xml) (See: https://compss-doc.readthedocs.io/en/stable/Sections/01_Installation/06_Configuration_files.html).

Hope it helps.

[simleo]

What to represent?

One of the main challenges here is providing guidelines that make sense for a wide variety of operating systems and workflow engines. This means we cannot go too deep into details. For instance, Nextflow tracing provides details such as virtual memory vs resident set, but this distinction might not apply to all systems and / or be made by all workflow engines. cwltool and Arvados, for instance, simply give "max memory used", which is sufficiently general.

The Nextflow tracing example shows that this kind of information can be very detailed; for generality and simplicity, I think we should focus on the most important bits, especially for the first release of the profiles. We could represent the following:

• memory usage: peak memory usage. This should be real memory if possible, so one can get an idea of the RAM requirements e.g. if there is no swap space, but we don't force the representation of a specific OS concept
• CPU usage: this is typically represented as a percentage, with values above 100 meaning that more than one computing core was used (see Nextflow -- Metrics). The actual number of CPUs where the process ran would give additional info (e.g. 100% could mean one CPU fully used or 4 CPUs with 25% usage). CPU type(s) would be another step in providing more complete info.
• GPU usage: same as CPU usage, but for GPUs

How to represent it?

Schema.org has things like memoryRequirements for SoftwareApplication, but these are mininum requirements to run the application. We need to describe actual resource usage and tie it to the actions. I could not find anything for this in the current RO-Crate context, so we probably need new terms. I've searched for ontologies for inspiration, but could not find much (e.g. the WICUS hardware specs seems also focused on application requirements).

The simplest approach is to add the properties directly to the action, for instance:

{
    "@id": "#action-1",
    "@type": "CreateAction",
    "memoryUsage": "8.43GB",
    "cpuUsage": "140.2%",
    "gpuUsage": "70.3%",
    "usedCpus": "2",
    "usedGpus": "1",
    ...
}

With CPU / GPU details:

{
    "@id": "#action-1",
    "@type": "CreateAction",
    "memoryUsage": "8.43GB",
    "cpuUsage": "140.2%",
    "gpuUsage": "70.3%",
    "usedCpus": [{"@id": "#cpu-1"}, {"@id": "#cpu-2"}]
    "usedGpus": {"@id": "#gpu-1"},
    ...
},
{
    "@id": "#cpu-1",
    "@type": "HardwareComponent",
    "model": "FooBar 314Pro",
    ...
},
{
    "@id": "#cpu-1",
    "@type": "HardwareComponent",
    "model": "FooBar XG666",
    ...
}

[kinow]

Autosubmit (to support RO-Crate soon) keeps track of some variables like memory, CPU, disk, that would fit the model discussed so far, I think... but there are other metrics reported that I am not sure if they would fit in the resource usage here (maybe they'd be reported somewhere else in the RO-Crate archive?).

• Energy used in workflow runs
  • when the HPC batch system provides it, like Slurm, we keep track¹² of that to document the energy used for a workflow run(s) (I believe that's used for reporting in research/group reports/etc)
• SYPD, simulated years per day
• number of nodes used in MPI for some tasks

I think a more flexible approach, allowing for custom values to be added would be useful, from what I understood about the topic so far (still getting familiar with RO-Crate, how to implement it, etc, sorry).

-Bruno

Footnotes

1. https://autosubmit.readthedocs.io/en/master/agui/performance/index.html ↩

2. https://www.bsc.es/supportkc/tech-reports/tech-report/ ↩

[simleo]

I think a more flexible approach, allowing for custom values to be added would be useful

Several group members were of the same idea at yesterday's meeting, with doubts expressed on the addition of "fixed" properties that might fit poorly the descriptions given by the various engines / systems. Since this will require substantially more thought, I'm removing this issue from the 0.1 milestone.