About SI00, CPUTime, maxCPUTime and what more

[ernstpijper]

Hi,

(Sorry if this got much longer than I hoped.)

In a previous ticket of mine (#5897) I made the remark that

 I set SI00 to 250 such that there is a 1-to-1 correspondence between the CPUTime in the jdl file and
 the maxCPUTime in the queue definition.

to which Federico replied:

Out of curiosity, why would you like to do that? My normal reasoning is that the larger the queue maxCPUTime,
the better -- considering that you can match one job after another with the filling mode.
At the same time, I am afraid that there is a misunderstanding here, because in fact what you are trying to
achieve is not really possible:

- SI00 is a benchmark per-minute of the queue (normally defined in BDII). This is effectively used ONLY
  for submitting pilots.
- MaxCPUTime in the queue is the number of seconds allowed
- the CPUTime defined in the JDL is in fact the CPUWork (in HepSpec06 seconds) required by the job.
- the real benchmark, used for matching, is anyway calculated on-the-fly by the pilot using [DB12]
 (https://github.com/DIRACGrid/DB12). This is what you see in the matcher log as PilotBenchmark: 15.5

I'd like to continue this discussion in a separate ticket, so as not to pollute the other ticket.

• Are you sure MaxCPUTime is in seconds? I have been told it is in minutes.
• Against what and how is the PilotBenchmark matched?
• The matcher log also mentions CPUTime. Is this value also a result from the DB12 benchmark?

My goal was not to bother users with HepSpec06 seconds. It may be perfect for CERN but I wonder if it will make our users happy. I'm sure that if we would tell them that CPUTime is in HS06 second, we are going to get a lot of questions about how they should calculate this, and in the end we would have to do it for them. I'm wondering how other communities deal with this.

As I understand it, the CPUTime jdl parameter is multiplied with SI00/250 and then compared with the maxCPUTime parameter to find a queue that matches the jdl. Then a pilot job is submitted. I realise other parameters should also match.

In our case I set the maxCPUTime equal to the maximum wall clock time of the queue. If I do this, then if I set the CPUTime just a few seconds above the maxCPUTime of a particular queue, the job wil not end up in that queue anymore. That's what I meant with the 'a 1-to-1 correspondence between CPUTime and maxCPUTime.'

On our local GinA compute cluster, I configured 2 queues: infra and long. The infra has a wall clock time of 30 minutes and can only run pvier VO jobs (it's basically for testing). In the jdl I would then set CPUTime = 1800 (which is 30 minutes). If I do not set the CPUTime, the defaultCPUTime is invoked, which is the equivalent of 96 hours in our case. Then the job won't run as the defaultCPUTime is much higher than the MaxCPUTime. I was thinking I might as well set the maxCPUTime parameter for the infra queue to the equivalent of 96 hours. Then I don't need the CPUTime parameter in the jdl anymore, and can just use the 'infra' tag. Or not even that, as infra only supports pvier jobs, pier jobs automatically go to the infra queue.

In the documentation it says:

maxCPUTime | Maximum time allowed to jobs to run in the queue

Does this mean the maximum time pilot jobs are allowed to run the queue? At least that's what I think it means.

I guess I need some advise on what would be the best way, considering our non-cern users, to configure dirac in such a way that they are not bothered (too much) with things they should not have and do not want to care about.

[aldbr]

Hi,

Sorry, definitions can be inconsistent between the documentation and the code, we have to work on that...

Let's start with some definitions (that has not been applied into the code yet :-)):

• CPUTime: the real time allowed in the queue. unit: real seconds. This is equivalent to the WallClockTime.
  Because an application will not spend the same time to run on different CPU models, this parameter is not sufficient, especially if you are using heterogeneous computing resources.
  Thus, in such context, we generally need to normalize the CPUTime.
• CPUPower: how efficient a given CPU is to compute a certain type of applications. unit: HS06 by default.
• CPUWork: the amount of work allowed in the queue. unit: HS06 seconds (CPUTime * CPUPower).

Are you sure MaxCPUTime is in seconds? I have been told it is in minutes.

The code and the documentation are not very clear about this, MaxCPUTime in the queue is expected to be in minutes, as you said, and corresponds to the maximum number of minutes a job can run in an allocation associated to a given queue.
MaxCPUTime, along with SI00, are used to compute the "CPUTime" of the queue, which is actually expected to be in HS06.seconds (defined as CPUWork in the above definitions): the maximum number of HS06.seconds a job can run in an allocation associated to a given queue.
1 HS06 is approximately defined as 250 SI00 units.

MaxCPUTime, SI00 should not be needed if you directly provide the queue parameters with "CPUTime" (in HS06.seconds).
This "CPUTime" parameter is mainly used for submitting pilots.

Against what and how is the PilotBenchmark matched?

The PilotBenchmark provides the CPUPower value.
DB12 is the default PilotBenchmark and provides an estimated value in HS06 units.
It has been primarily tailored for Monte Carlo simulation applications in the context of LHCb and might not be adapted to your use case.

The CPUPower is computed before the Pilot starts to fetch a first job.
To get the CPUWork left in an allocation, the pilot gets the CPUTime left by interrogating the batch system (if possible) and compute CPUTime left * CPUPower.
The operation is performed before fetching any job: the Matcher compares the "CPUTime" (actually CPUWork) required by a job to the CPUWork left in the allocation.

If the batch system is not recognized, then the "CPUTime" (actually CPUWork) defined in the queue parameters is used.

The matcher log also mentions CPUTime. Is this value also a result from the DB12 benchmark?

The "CPUTime" mentioned in the Matcher log is actually the CPUWork left in the allocation (computed by the pilot before calling the Matcher and defined as CPUTime left * CPUPower).

As I understand it, the CPUTime jdl parameter is multiplied with SI00/250 and then compared with the maxCPUTime parameter to find a queue that matches the jdl. Then a pilot job is submitted. I realise other parameters should also match.

From what I understand, the "CPUTime" jdl (defined in the job) is actually CPUWork and is not multiplied with SI00/250 (again the code and documentation are not crystal clear).

In our case I set the maxCPUTime equal to the maximum wall clock time of the queue. If I do this, then if I set the CPUTime just a few seconds above the maxCPUTime of a particular queue, the job will not end up in that queue anymore. That's what I meant with the 'a 1-to-1 correspondence between CPUTime and maxCPUTime.'

In this case, what is the value of the SI00 parameter? For instance, if you set SI00 to 250 and MaxCPUTime to 30, then the "CPUTime" (CPUWork) of the queue should be computed as: 250/250 * 30 * 60 = 1800 HS06.s.
Then if the "CPUTime" (CPUWork) of the job is defined as 1900 HS06.s, the Site Director will not submit any pilot for this job in your queue. I guess it is expected.

On our local GinA compute cluster, I configured 2 queues: infra and long. The infra has a wall clock time of 30 minutes and can only run pvier VO jobs (it's basically for testing). In the jdl I would then set CPUTime = 1800 (which is 30 minutes). If I do not set the CPUTime, the defaultCPUTime is invoked, which is the equivalent of 96 hours in our case. Then the job won't run as the defaultCPUTime is much higher than the MaxCPUTime. I was thinking I might as well set the maxCPUTime parameter for the infra queue to the equivalent of 96 hours. Then I don't need the CPUTime parameter in the jdl anymore, and can just use the 'infra' tag. Or not even that, as infra only supports pvier jobs, pier jobs automatically go to the infra queue.

Using the defaultCPUTime of the jobs is probably the best option to not bother your users with such a complex task as you said.
I suggest:

• disabling the Pilot fillingModeFlag option to process one job per Pilot (because we can't rely on the time values in this case).
  You can set ExtraPilotOptions as -o FillingModeFlag=False (not entirely sure it still works) or --MaxCycles=1 in the SiteDirector CS section.
• setting a high "CPUTime" value in the infra queue parameters (more than the defaultCPUTime of the jdl): so more than 345600 (96h).
• setting VO = pvier in the infra queue parameters so that only pvier jobs are executed in the queue.

Let me know if you need further explanations.