CWLViewer fails to render a workflow, possibly due to commas in the name

[kinow]

Hi,

I copied this workflow from a project at CERN (credits to them, I never worked with this, just from communication with another researchers from there): https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon/-/blob/master/tests/example_workflows/complex_workflow_with_filtering/AnaProd_example_workflows_job_tuple%2Csplit%2Cfilter_B0.cwl?ref_type=heads

As the domain is not accepted by CWLViewer, I copied it to a GitHub repo (no guarantees this link works in +1 year 😬 ): https://github.com/kinow/test-cwl/blob/master/AnaProd_example_workflows_job_tuple%2Csplit%2Cfilter_B0.cwl

Importing it, I get a blank graph.

[kinow]

The workflow is created in the database:

And I can also confirm 2 image files, one png and one svg were created, but they are really empty (they are in /tmp).

The debugger shows this as the dot file produced.

digraph workflow {
  graph [
    bgcolor = "#eeeeee"
    color = "black"
    fontsize = "10"
    labeljust = "left"
    clusterrank = "local"
    ranksep = "0.22"
    nodesep = "0.05"
  ]
  node [
    fontname = "Helvetica"
    fontsize = "10"
    fontcolor = "black"
    shape = "record"
    height = "0"
    width = "0"
    color = "black"
    fillcolor = "lightgoldenrodyellow"
    style = "filled"
  ];
  edge [
    fontname="Helvetica"
    fontsize="8"
    fontcolor="black"
    color="black"
    arrowsize="0.7"
  ];
  subgraph cluster_inputs {
    rank = "same";
    style = "dashed";
    label = "Workflow Inputs";
  }
  subgraph cluster_outputs {
    rank = "same";
    style = "dashed";
    labelloc = "b";
    label = "Workflow Outputs";
  }
}

[kinow]

Interesting. During the creation of the queue_workflow, the visualization dot is produced correctly 🤔

The dot:

digraph workflow {
  graph [
    bgcolor = "#eeeeee"
    color = "black"
    fontsize = "10"
    labeljust = "left"
    clusterrank = "local"
    ranksep = "0.22"
    nodesep = "0.05"
  ]
  node [
    fontname = "Helvetica"
    fontsize = "10"
    fontcolor = "black"
    shape = "record"
    height = "0"
    width = "0"
    color = "black"
    fillcolor = "lightgoldenrodyellow"
    style = "filled"
  ];
  edge [
    fontname="Helvetica"
    fontsize="8"
    fontcolor="black"
    color="black"
    arrowsize="0.7"
  ];
  subgraph cluster_inputs {
    rank = "same";
    style = "dashed";
    label = "Workflow Inputs";
    "output-prefix" [fillcolor="#94DDF4"];
    "input-data" [fillcolor="#94DDF4"];
  }
  subgraph cluster_outputs {
    rank = "same";
    style = "dashed";
    label = "Workflow Outputs";
    "PIPIPI_ROOT" [fillcolor="#94DDF4",label="Output Data: PIPIPI.ROOT";];
    "KKPI_ROOT" [fillcolor="#94DDF4",label="Output Data: KKPI.ROOT";];
    "B0_ROOT" [fillcolor="#94DDF4",label="Output Data: B0.ROOT";];
    "KMPIPI_ROOT" [fillcolor="#94DDF4",label="Output Data: KMPIPI.ROOT";];
    "KPPIPI_ROOT" [fillcolor="#94DDF4",label="Output Data: KPPIPI.ROOT";];
    "KPKPPI_ROOT" [fillcolor="#94DDF4",label="Output Data: KPKPPI.ROOT";];
    "BS0_ROOT" [fillcolor="#94DDF4",label="Output Data: BS0.ROOT";];
    "others" [fillcolor="#94DDF4",label="Logs and summaries";];
    "KMKPKP_ROOT" [fillcolor="#94DDF4",label="Output Data: KMKPKP.ROOT";];
  }
  "transformation_2";
  "transformation_3";
  "transformation_4";
  "transformation_5";
  "transformation_1";
  "transformation_4" -> "PIPIPI_ROOT";
  "transformation_4" -> "KKPI_ROOT";
  "transformation_5" -> "B0_ROOT";
  "transformation_4" -> "KMPIPI_ROOT";
  "transformation_4" -> "KPPIPI_ROOT";
  "transformation_4" -> "KPKPPI_ROOT";
  "transformation_4" -> "BS0_ROOT";
  "transformation_1" -> "others";
  "transformation_2" -> "others";
  "transformation_3" -> "others";
  "transformation_4" -> "others";
  "transformation_5" -> "others";
  "transformation_4" -> "KMKPKP_ROOT";
  "output-prefix" -> "transformation_2";
  "transformation_1" -> "transformation_2";
  "output-prefix" -> "transformation_3";
  "transformation_2" -> "transformation_3";
  "output-prefix" -> "transformation_4";
  "transformation_2" -> "transformation_4";
  "transformation_2" -> "transformation_4";
  "transformation_2" -> "transformation_4";
  "transformation_2" -> "transformation_4";
  "transformation_2" -> "transformation_4";
  "transformation_2" -> "transformation_4";
  "transformation_2" -> "transformation_4";
  "output-prefix" -> "transformation_5";
  "transformation_3" -> "transformation_5";
  "output-prefix" -> "transformation_1";
  "input-data" -> "transformation_1";

  // Invisible links to force outputs to be at lowest rank
  "transformation_2" -> "PIPIPI_ROOT" [style=invis];
  "transformation_3" -> "PIPIPI_ROOT" [style=invis];
  "transformation_4" -> "PIPIPI_ROOT" [style=invis];
  "transformation_5" -> "PIPIPI_ROOT" [style=invis];
  "transformation_1" -> "PIPIPI_ROOT" [style=invis];
}

Which saving as graph with dot -Tpng test.dot -o test.png produces this:

[mr-c]

I copied this workflow from a project at CERN (credits to them, I never worked with this, just from communication with another researchers from there): https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon/-/blob/master/tests/example_workflows/complex_workflow_with_filtering/AnaProd_example_workflows_job_tuple%2Csplit%2Cfilter_B0.cwl?ref_type=heads

As the domain is not accepted by CWLViewer,

FYI, you can load workflows from arbitrary Git hosts. Put the path to the git repo (ending in .git) in the URL to workflow box, which then transforms to allow inputting of the Branch Name/Commit ID and the Path to workflow

So for this link the three components are:

• URL to workflow : https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon.git
• Branch Name/Commit ID: master
• Path to workflow: tests/example_workflows/complex_workflow_with_filtering/AnaProd_example_workflows_job_tuple,split,filter_B0.cwl

While this succeeded once in reproducing the error for me, additional attempts fail. My guess is that the Git host is rate-limiting repeated anonymous connections. So copying the entire Git repository to GitHub was a smart idea :-)

[kinow]

Ah, I think I never used this feature, @mr-c . That might be how the generic git URL regex is used. I will use it for testing as I'm changing those regexes here to pass the CodeQL build. Thank you!

[kinow]

Debugging it a bit more today, before I start a final MPI assignment and disappear for some days 🤓 I will write what I have here in case I don't finish it, so that I can restart from this point in the future.

---

@mr-c, the PNG is created successfully when the queued workflow is created.

When the Workflow object is being created, one of the steps is to call cwltool --print-rdf AnaProd....cwl. That produces this RDF: https://gist.github.com/kinow/3e544a87302888d3af7e73c3df95ab31

(There are 2 files in the gist above, original CWL (for posterity) and the RDF. I tried uploading here but the RDF produced is too large. )

If I copy the query for the inputs, and paste into Jena Fuseki, it returns an empty dataset.

PREFIX cwl: <https://w3id.org/cwl/cwl#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX sld: <https://w3id.org/cwl/salad#>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX doap: <http://usefulinc.com/ns/doap#>
PREFIX Workflow: <https://w3id.org/cwl/cwl#Workflow/>
PREFIX DockerRequirement: <https://w3id.org/cwl/cwl#DockerRequirement/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX s: <http://schema.org/>
SELECT ?name ?type ?items ?null ?format ?label ?doc
WHERE {
  GRAPH ?wf {
    ?wf rdf:type cwl:Workflow .
    ?wf cwl:inputs ?name .
    OPTIONAL {
      { 
        ?name sld:type ?type
        FILTER(?type != sld:null) 
        FILTER (!isBlank(?type))
      } UNION { 
        ?name sld:type ?arraytype .
        ?arraytype sld:type ?type .
        ?arraytype sld:items ?items 
      }
    }
    OPTIONAL { 
      ?name sld:type ?null
      FILTER(?null = sld:null)
    }
    OPTIONAL { ?name cwl:format ?format }
    OPTIONAL { ?name sld:label|rdfs:label ?label }
    OPTIONAL { ?name sld:doc|rdfs:comment ?doc }
  }
}

However, if I modify the query to assume the workflow & inputs are not necessarily in the same workflow…

PREFIX cwl: <https://w3id.org/cwl/cwl#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX sld: <https://w3id.org/cwl/salad#>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX doap: <http://usefulinc.com/ns/doap#>
PREFIX Workflow: <https://w3id.org/cwl/cwl#Workflow/>
PREFIX DockerRequirement: <https://w3id.org/cwl/cwl#DockerRequirement/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX s: <http://schema.org/>
SELECT ?name ?type ?items ?null ?format ?label ?doc
WHERE {
  GRAPH ?g1 {
    ?wf rdf:type cwl:Workflow .
  }

  GRAPH ?g2 {
    ?wf cwl:inputs ?name .

    OPTIONAL {
      { 
        ?name sld:type ?type
        FILTER(?type != sld:null)
        FILTER(!isBlank(?type))
      }
      UNION
      {
        ?name sld:type ?arraytype .
        ?arraytype sld:type ?type .
        ?arraytype sld:items ?items
      }
    }

    OPTIONAL { ?name sld:type ?null FILTER(?null = sld:null) }
    OPTIONAL { ?name cwl:format ?format }
    OPTIONAL { ?name sld:label|rdfs:label ?label }
    OPTIONAL { ?name sld:doc|rdfs:comment ?doc }
  }
}

Then I get results. I think the workflow from CERN might be using subworkflows. I saw one variable subworkflows in CWL Viewer that's never used. I found some other issues here about subworkflows and sub workflows. But I couldn't find anything specifically mentioning disjoint graphs, distinct graphs.

But there might already be an issue for this.

Not sure if it's safe to just modify every query to consider workflow & parts may be in separate graphs... I don't have the mental picture of the schema that CWL Viewer uses yet.

[kinow]

Huh, the query above is very broken. When binding the workflow, it doesn't work. And it shouldn't match inputs of different workflows. Tried some mix of substrings to isolate only the top workflow, and got the right inputs, but the graph generated was a mess.

I had a look at CWLTool, cwlrdf.py, but it's hard to translate that to CWL Viewer's Java logic. I guess I will just stash this one, and use this for now.

$ cwltool --print-dot AnaProd_example_workflows_job_tuple,split,filter_B0.cwl > workflow.dot
$ dot -Tpng workflow.dot -o workflow.png