Strange tag behaviour

[ernstpijper]

Dirac version 7.3.17 (python 2), diracos2 client 7.3.17

Hi,

I'm experiencing strange behaviour involving tags. I have defined a 1 core infra queue with 2 tags: infra and surf, like this:

          infra-1core
          {
            CEQueueName = nordugrid-slurm-infra
            MaxWaitingJobs = 100
            MaxTotalJobs = 1000
            SI00 = 250
            maxCPUTime = 30
            Tag = infra
            Tag += surf
            VO = pvier
          }

(I set SI00 to 250 such that there is a 1-to-1 correspondence between the CPUTime in the jdl file and the maxCPUTime in the queue definition. This works very nice and I think it's very user friendly.)

I have the following user job:

      [
       JobName = "pvier_1core";
       Executable = "testJob.sh";
       StdOutput = "StdOut";
       StdError = "StdErr";
       InputSandbox = {"testJob.sh"};
       OutputSandbox = {"StdOut","StdErr"};
       CPUTime = 1000;
       Site = "GRID.SURF.nl";
      Tags = " infra";
     ]

When I submit this job, dirac submits a pilot job to the above queue. At this point there are no other user jobs, just the one I just submitted. Then the trouble starts. When the pilot jobs runs, it checks for user jobs that match the requirements, which should be the user job I just submitted and that resulted in this pilot job to be submitted. Unfortunately, the Matcher process says that there are no matching jobs. In the Matcher log i see this:

    2022-02-22 18:54:21 UTC WorkloadManagement/Matcher/[gsiftp://arc01.gina.surfsara.nl:2811/jobs/7ciMDm1Spg0nCifV3nVLvLKmABFKDmABFKDmZwFKDmABFKDm3OKUGn]Matcher INFO: Resource description for matching {
    CPUTime:               27078
    DIRACVersion:          v7r3p17
    GridCE:                arc01.gina.surfsara.nl
    MaxRAM:                488952
    NumberOfProcessors:    1
    OwnerGroup:            pvier_pilot,pvier_user
    PilotBenchmark:        15.5
    PilotInfoReportedFlag: False
    PilotReference:        gsiftp://arc01.gina.surfsara.nl:2811/jobs/7ciMDm1Spg0nCifV3nVLvLKmABFKDmABFKDmZwFKDmABFKDm3OKUGn
    Platform:              Linux_x86_64_glibc-2.17
    ReleaseVersion:        v7r3p17
    Setup:                 DIRAC-Production
    Site:                  GRID.SURF.nl
    Tag:                    infra,surf
    }
    2022-02-22 18:54:21 UTC WorkloadManagement/Matcher/[gsiftp://arc01.gina.surfsara.nl:2811/jobs/7ciMDm1Spg0nCifV3nVLvLKmABFKDmABFKDmZwFKDmABFKDm3OKUGn]Matcher INFO: No TQ matches requirements
    2022-02-22 18:54:21 UTC WorkloadManagement/Matcher/[gsiftp://arc01.gina.surfsara.nl:2811/jobs/7ciMDm1Spg0nCifV3nVLvLKmABFKDmABFKDmZwFKDmABFKDm3OKUGn]Matcher INFO: No match found

Notice the indentation in the 'Tag' line with respect to the other lines.

To understand the problem I did some tests:

1. submit a user job without tags in the jdl file --> user job completes
2. submit a user job where I define only the 'surf' tag in the jdl file. The 'surf' tag is the 2nd tag in the queue definition --> user job completes
3. only define 1 tag in the queue definition, then submit a user job with only that tag in the jdl file --> user job completes (no indentation in the Tag line in the Matcher log)
4. I also tried dirac version 7.2.40 --> same behaviour as for 7.3.17
5. When I add the MaxRAM attribute, the tags in the Tag line in the Matcher log switch around and it becomes

       Site:                 GRID.SURF.nl
       Tag:                  surf, infra

Notice that the indentation is gone but there is still extra space in front of 'infra'.
6. I also tried 7.2.27. I now see the following error message in the Matcher log:

     2022-02-22 12:25:41 UTC WorkloadManagement/Matcher NOTICE: Returning response ([::ffff:145.100.48.88]:43626)[pvier_pilot:r2d2] (0.00 secs) ERROR: Server error while serving requestJob: 'str' object has no attribute 'extend'
     2022-02-22 12:26:31 UTC WorkloadManagement/Matcher NOTICE: Executing action ([::ffff:145.100.48.88]:43800)[pvier_pilot:r2d2] RPC/requestJob(<masked>)
     2022-02-22 12:26:31 UTC WorkloadManagement/Matcher ERROR: Uncaught exception when serving RPC Function requestJob
     Traceback (most recent call last):
       File "/opt/dirac/pro/DIRAC/Core/DISET/RequestHandler.py", line 291, in __RPCCallFunction
         uReturnValue = oMethod(*args)
       File "/opt/dirac/pro/DIRAC/WorkloadManagementSystem/Service/MatcherHandler.py", line 103, in export_requestJob
         result = matcher.selectJob(resourceDescription, credDict)
       File "/opt/dirac/pro/DIRAC/WorkloadManagementSystem/Client/Matcher.py", line 81, in selectJob
         resourceDict = self._getResourceDict(resourceDescription, credDict)
       File "/opt/dirac/pro/DIRAC/WorkloadManagementSystem/Client/Matcher.py", line 165, in _getResourceDict
         resourceDict = self._processResourceDescription(resourceDescription)
       File "/opt/dirac/pro/DIRAC/WorkloadManagementSystem/Client/Matcher.py", line 221, in _processResourceDescription
         resourceDict.setdefault("Tag", []).extend(paramTags)
     AttributeError: 'str' object has no attribute 'extend'
     2022-02-22 12:26:31 UTC WorkloadManagement/Matcher NOTICE: Returning response ([::ffff:145.100.48.88]:43800)[pvier_pilot:r2d2] (0.00 secs) ERROR: Server error while serving requestJob: 'str' object has no attribute 'extend'

I also have another 7.2.27 dirac server that does not have this problem. The fact that I have this problem for 3 different versions, and that another server that does not show this problem, would suggest that my setup has a bug somewhere. I just can't figure out what it might be.

Any help is welcome.

[fstagni]

Hi,

Dirac version 7.3.17 (python 2), diracos2 client 7.3.17

For my understanding, you mean py2 server with py3 client, right? In this case anyway what's important is the Pilot version.

I set SI00 to 250 such that there is a 1-to-1 correspondence between the CPUTime in the jdl file and the maxCPUTime in the queue definition.

Out of curiosity, why would you like to do that? My normal reasoning is that the larger the queue maxCPUTime, the better -- considering that you can match one job after another with the filling mode.
At the same time, I am afraid that there is a misunderstanding here, because in fact what you are trying to achieve is not really possible:

• SI00 is a benchmark per-minute of the queue (normally defined in BDII). This is effectively used ONLY for submitting pilots.
• MaxCPUTime in the queue is the number of seconds allowed
• the CPUTime defined in the JDL is in fact the CPUWork (in HepSpec06 seconds) required by the job.
• the real benchmark, used for matching, is anyway calculated on-the-fly by the pilot using DB12. This is what you see in the matcher log as PilotBenchmark: 15.5

The bug you are reporting for v7r2p27 was also reported in #5826 (and fixed).

Regarding this specific issue: seems like you have a good "debug" setup, can you add a few prints in https://github.com/DIRACGrid/DIRAC/blob/rel-v7r3/src/DIRAC/WorkloadManagementSystem/Client/Matcher.py#L190-L194 and let me know?

[ernstpijper]

Hi Andre,

Yes I tried that yesterday. That works for the regular tags but not for the required tags. That's a different part of the code.

[andresailer]

Just a couple lines below

DIRAC/src/DIRAC/WorkloadManagementSystem/Client/Matcher.py

Line 196 in dfd07fa

resourceDescription["RequiredTag"].split(",")

-  resourceDescription["RequiredTag"].split(",") 
+  list({tag.strip("\"' ") for tag in resourceDescription["RequiredTag"].strip("[]").split(",")})

[ernstpijper]

Ah yes now it works for both the regular and required tags. Thanks.

It seems weird that nobody has a run into this bug before. Apparently nobody uses more than 1 tag or multicore jobs. Or is it me and have I just 'dirty hacked' my way out of it?

[fstagni]

Please check #5904

[andresailer]

I don't know how many instances use tags, but consider that this code has undergone many changes recently, both on the pilot and server side. So maybe you are just the early adopter that got hit. But thanks for debugging the issue! Would you make a PR with the little changes? For rel-v7r2 if possible.