Dirac API supporting JDL submission

[simon-mazenoux]

Dear all,

With the introduction of Workflow in DIRAC, the process of submitting directly a JDL string has quite diverged compared to using the Job API. Moreover, the possibility to submit directly a JDL file limits the possibility to (maybe) move away from this format. It also seems like a security hazard. Are VOs using this way of submitting jobs ? Do we want to maintain the possibility to submit a job direclty as a JDL string ? Is there a use case for this ?

[marianne013]

We (GridPP) do see occasional heavy use of JDLs, often associated with new users, or one off workflows. VOs outside of the big WLCG VOs and the big Astro groups are typically much less centrally organised, so even the "How do VOs use this" is not a meaningful question for some of our users. The ones that do have centralised processing of data tend to use the API.

We advise users that for any production type processing they should use the python API. But other languages are in use, so you cannot assume "everyone knows python" - using dirac-wms-job-submit as the grid equivalent of qsub without ever touching any python code is not unheard of.

Finally: If there are security issues, please submit a fix, allusions in DIRAC discussions are not helpful.

[simon-mazenoux]

Thank you for your feedback. I now understand the point of still having the JDL input format still available.

When I said that it was a potential security hazard, I meant that the user can specify fields in the JDL that he shouldn't put. Some of those fields are checked (for example in the Owner is overwritten systematically), but some of there are not (JobPath for example).

What do you think about the idea to parse the JDL file into a Job object in the Dirac class (which is used by the dirac-wms-job-submit script)? This way, we could assure that all the fields are correctly specified. This would also mean that now all the job submitted will now have a jobDescription XML file.

Also, this would greatly simplify the checking of some variables. For example, instead of checking in the check method of the JobManifest class if the CPUTime in within some range, we could simply test this in the setCPUTime method of the Job class.

[marianne013]

I think converting tthe JDL to an xml file at the first opportunity is probably the way forward, bonus points for user friendly error messages if the conversion fails - so the user can at least attempt to fix their jdl.