Skip to content

Commit 54659bf

Browse files
fmigneaultfmigneault
authored
Merge pull request #566 from crim-ca/fix-cwl-enum-schema-name
2 parents f0abcd1 + 5f8962c commit 54659bf

File tree

23 files changed

+1339
-84
lines changed

23 files changed

+1339
-84
lines changed

‎CHANGES.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ Changes:
2323
have to provide the value explicitly, or update the deployed `Process` definition afterwards with the relevant
2424
``PUT`` request. Since ``public`` will now be used by default, the `CLI` will not automatically inject the value
2525
in the payload anymore when omitted.
26+
- Remove attribute ``WpsProcessInterface.stage_output_id_nested`` and enforce the behavior of nesting output by ID
27+
under corresponding directories for all remote `Process` execution when resolving `CWL` `Workflow` steps. This
28+
ensures a more consistent file and directory resolution between steps of different nature (`CWL`, `WPS`, `OGC` based)
29+
using multiple combinations of ``glob`` patterns and expected media-types.
2630

2731
Fixes:
2832
------
@@ -38,6 +42,10 @@ Fixes:
3842
Links will only be listed within the returned ``processSummary`` to respect the `OGC API - Processes` schema.
3943
- Fix `CLI` not removing embedded ``links`` in ``processSummary`` from ``deploy`` operation response
4044
when ``-nL``/``--no-links`` option is specified.
45+
- Fix `CWL` definitions combining nested ``enum`` types as ``["null", <enum>, {type: array, items: <enum>]`` without an
46+
explicit ``name`` or ``SchemaDefRequirement`` causing failing ``schema_salad`` resolution under ``cwltool``. A patch
47+
is applied for the moment to inject a temporary ``name`` to let the `CWL` engine succeed schema validation (relates
48+
to `common-workflow-language/cwltool#1908 <https://github.com/common-workflow-language/cwltool/issues/1908>`_).
4149

4250
.. _changes_4.31.0:
4351

‎setup.cfg

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ skip = *.egg*,build,env,src,venv,reports,node_modules
8686

8787
[bandit]
8888
skips = B101,B320,B410
89-
exclude = *.egg-info,build,dist,env,tests,./tests,test_*
89+
exclude = *.egg-info,./build,./dist,./env,./tests,test_*
9090
targets = .
9191

9292
[flake8]

‎tests/functional/application-packages/DockerCopyNestedOutDir/deploy.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ processDescription:
1010
- mimeType: text/plain
1111
default: true
1212
minOccurs: 1
13+
maxOccurs: "unbounded"
1314
outputs:
1415
- id: output_files
1516
formats:

‎tests/functional/application-packages/DockerCopyNestedOutDir/package.cwl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ inputs:
3232
position: 1
3333
outputs:
3434
output_files:
35+
# NOTE: always one, but using array to allow chaining itself any amount of times
3536
type:
3637
type: array
3738
items: File

‎tests/functional/application-packages/Finch_EnsembleGridPointWetdays/deploy.yml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
processDescription:
2+
process:
3+
id: Finch_EnsembleGridPointWetdays
4+
jobControlOptions:
5+
- async-execute
6+
outputTransmission:
7+
- reference
8+
executionUnit:
9+
# note: This does not work by itself! The test suite injects the file dynamically.
10+
- href: "tests/functional/application-packages/Finch_EnsembleGridPointWetdays/package.cwl"
11+
# note: alternative for WPS (applied in tests), inspired from:
12+
# https://finch.crim.ca/wps?service=WPS&request=DescribeProcess&version=1.0.0&identifier=ensemble_grid_point_wetdays
13+
deploymentProfileName: "http://www.opengis.net/profiles/eoc/dockerizedApplication"

‎tests/functional/application-packages/Finch_EnsembleGridPointWetdays/describe.xml

Lines changed: 424 additions & 0 deletions
Large diffs are not rendered by default.

‎tests/functional/application-packages/Finch_EnsembleGridPointWetdays/execute.yml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
lat: "45.35629610945964"
2+
lon: "-73.98748912005094"
3+
start_date: "1950"
4+
end_date: "1960"
5+
ensemble_percentiles: ""
6+
dataset: "candcs-u6"
7+
scenario: "ssp126"
8+
models: "26models"
9+
freq: "YS"
10+
data_validation: "warn"
11+
output_format: "csv"
12+
csv_precision: 0
13+
thresh: "15 mm\/day"

‎tests/functional/application-packages/Finch_EnsembleGridPointWetdays/package.cwl

Lines changed: 312 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,312 @@
1+
# NOTE:
2+
# Inspired from (but not an exact equivalent):
3+
# https://finch.crim.ca/wps?service=WPS&request=DescribeProcess&version=1.0.0&identifier=ensemble_grid_point_wetdays
4+
# Outputs are modified to only collect stdout, since we don't have the expected outputs produced by the real process.
5+
# The 'baseCommand' is also added to produce this stdout output.
6+
# All remaining arguments are identical.
7+
cwlVersion: v1.0
8+
class: CommandLineTool
9+
requirements:
10+
InlineJavascriptRequirement: {}
11+
# NOTE: replaced by 'baseCommand'
12+
#hints:
13+
# WPS1Requirement:
14+
# provider: https://finch.crim.ca/wps
15+
# process: ensemble_grid_point_wetdays
16+
baseCommand: echo
17+
inputs:
18+
- id: lat
19+
type:
20+
- string
21+
- type: array
22+
items: string
23+
- id: lon
24+
type:
25+
- string
26+
- type: array
27+
items: string
28+
- id: start_date
29+
type:
30+
- 'null'
31+
- string
32+
- id: end_date
33+
type:
34+
- 'null'
35+
- string
36+
- id: ensemble_percentiles
37+
type:
38+
- 'null'
39+
- string
40+
default: 10,50,90
41+
- id: average
42+
type:
43+
- 'null'
44+
- boolean
45+
default: false
46+
- id: dataset
47+
type:
48+
- 'null'
49+
- type: enum
50+
symbols:
51+
- humidex-daily
52+
- candcs-u5
53+
- candcs-u6
54+
- bccaqv2
55+
default: candcs-u5
56+
# WARNING:
57+
# Following definition combining 'enum' and its corresponding nested definition in 'array' caused a
58+
# schema-salad name resolution error. This CWL is used particularly to validate this *valid* type resolution.
59+
# see https://github.com/common-workflow-language/cwltool/issues/1908
60+
- id: scenario
61+
type:
62+
- 'null'
63+
- type: enum
64+
symbols:
65+
- ssp126
66+
- rcp85
67+
- rcp45
68+
- rcp26
69+
- ssp585
70+
- ssp245
71+
- type: array
72+
items:
73+
type: enum
74+
symbols:
75+
- ssp126
76+
- rcp85
77+
- rcp45
78+
- rcp26
79+
- ssp585
80+
- ssp245
81+
- id: models
82+
type:
83+
- 'null'
84+
- type: enum
85+
symbols:
86+
- KACE-1-0-G
87+
- CCSM4
88+
- MIROC5
89+
- EC-Earth3-Veg
90+
- TaiESM1
91+
- GFDL-ESM4
92+
- GFDL-CM3
93+
- CanESM5
94+
- HadGEM3-GC31-LL
95+
- INM-CM4-8
96+
- IPSL-CM5A-MR
97+
- EC-Earth3
98+
- GFDL-ESM2G
99+
- humidex_models
100+
- GFDL-ESM2M
101+
- MIROC-ESM
102+
- CSIRO-Mk3-6-0
103+
- MPI-ESM-LR
104+
- NorESM1-M
105+
- CNRM-CM5
106+
- all
107+
- GISS-E2-1-G
108+
- 24models
109+
- MPI-ESM1-2-HR
110+
- CNRM-ESM2-1
111+
- CNRM-CM6-1
112+
- CanESM2
113+
- FGOALS-g3
114+
- NorESM1-ME
115+
- IPSL-CM6A-LR
116+
- CMCC-ESM2
117+
- pcic12
118+
- EC-Earth3-Veg-LR
119+
- ACCESS-ESM1-5
120+
- MRI-CGCM3
121+
- MIROC-ESM-CHEM
122+
- NorESM2-MM
123+
- bcc-csm1-1-m
124+
- BNU-ESM
125+
- UKESM1-0-LL
126+
- CESM1-CAM5
127+
- MIROC-ES2L
128+
- MRI-ESM2-0
129+
- HadGEM2-ES
130+
- MIROC6
131+
- MPI-ESM-MR
132+
- INM-CM5-0
133+
- bcc-csm1-1
134+
- BCC-CSM2-MR
135+
- ACCESS-CM2
136+
- NorESM2-LM
137+
- IPSL-CM5A-LR
138+
- FGOALS-g2
139+
- HadGEM2-AO
140+
- 26models
141+
- MPI-ESM1-2-LR
142+
- KIOST-ESM
143+
- type: array
144+
items:
145+
type: enum
146+
symbols:
147+
- KACE-1-0-G
148+
- CCSM4
149+
- MIROC5
150+
- EC-Earth3-Veg
151+
- TaiESM1
152+
- GFDL-ESM4
153+
- GFDL-CM3
154+
- CanESM5
155+
- HadGEM3-GC31-LL
156+
- INM-CM4-8
157+
- IPSL-CM5A-MR
158+
- EC-Earth3
159+
- GFDL-ESM2G
160+
- humidex_models
161+
- GFDL-ESM2M
162+
- MIROC-ESM
163+
- CSIRO-Mk3-6-0
164+
- MPI-ESM-LR
165+
- NorESM1-M
166+
- CNRM-CM5
167+
- all
168+
- GISS-E2-1-G
169+
- 24models
170+
- MPI-ESM1-2-HR
171+
- CNRM-ESM2-1
172+
- CNRM-CM6-1
173+
- CanESM2
174+
- FGOALS-g3
175+
- NorESM1-ME
176+
- IPSL-CM6A-LR
177+
- CMCC-ESM2
178+
- pcic12
179+
- EC-Earth3-Veg-LR
180+
- ACCESS-ESM1-5
181+
- MRI-CGCM3
182+
- MIROC-ESM-CHEM
183+
- NorESM2-MM
184+
- bcc-csm1-1-m
185+
- BNU-ESM
186+
- UKESM1-0-LL
187+
- CESM1-CAM5
188+
- MIROC-ES2L
189+
- MRI-ESM2-0
190+
- HadGEM2-ES
191+
- MIROC6
192+
- MPI-ESM-MR
193+
- INM-CM5-0
194+
- bcc-csm1-1
195+
- BCC-CSM2-MR
196+
- ACCESS-CM2
197+
- NorESM2-LM
198+
- IPSL-CM5A-LR
199+
- FGOALS-g2
200+
- HadGEM2-AO
201+
- 26models
202+
- MPI-ESM1-2-LR
203+
- KIOST-ESM
204+
default: all
205+
- id: thresh
206+
type:
207+
- 'null'
208+
- string
209+
default: 1.0 mm/day
210+
- id: freq
211+
type:
212+
- 'null'
213+
- type: enum
214+
symbols:
215+
- YS
216+
- QS-DEC
217+
- AS-JUL
218+
- MS
219+
default: YS
220+
- id: op
221+
type:
222+
- 'null'
223+
- type: enum
224+
symbols:
225+
- '>='
226+
- '>'
227+
- gt
228+
- ge
229+
default: '>='
230+
- id: month
231+
type:
232+
- 'null'
233+
- int
234+
- type: array
235+
items: int
236+
inputBinding:
237+
valueFrom: "\n ${\n const values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12];\n if (Array.isArray(self)) {\n \n if (self.every(item => values.includes(item))) {\n return self;\n }\n else {\n throw \"invalid value(s) in [\" + self + \"] are not all allowed values from [\" + values + \"]\";\n }\n \n }\n else {\n \n if (values.includes(self)) {\n return self;\n }\n else {\n throw \"invalid value \" + self + \" is not an allowed value from [\" + values + \"]\";\n }\n \n }\n }\n "
238+
- id: season
239+
type:
240+
- 'null'
241+
- type: enum
242+
symbols:
243+
- SON
244+
- MAM
245+
- JJA
246+
- DJF
247+
- id: check_missing
248+
type:
249+
- 'null'
250+
- type: enum
251+
symbols:
252+
- pct
253+
- at_least_n
254+
- wmo
255+
- skip
256+
- from_context
257+
- any
258+
default: any
259+
- id: missing_options
260+
type:
261+
- 'null'
262+
- File
263+
format: iana:application/json
264+
- id: cf_compliance
265+
type:
266+
- 'null'
267+
- type: enum
268+
symbols:
269+
- raise
270+
- log
271+
- warn
272+
default: warn
273+
- id: data_validation
274+
type:
275+
- 'null'
276+
- type: enum
277+
symbols:
278+
- raise
279+
- log
280+
- warn
281+
default: raise
282+
- id: output_name
283+
type:
284+
- 'null'
285+
- string
286+
- id: output_format
287+
type:
288+
- 'null'
289+
- type: enum
290+
symbols:
291+
- csv
292+
- netcdf
293+
default: netcdf
294+
- id: csv_precision
295+
type:
296+
- 'null'
297+
- int
298+
# NOTE:
299+
# Following structure is permitted in standard CWL, but not supported in Weaver.
300+
# Must use the equivalent 'long form' in the meantime.
301+
#outputs:
302+
#- id: output
303+
# type: stdout
304+
outputs:
305+
- id: output
306+
type: File
307+
outputBinding:
308+
glob: "stdout.log"
309+
stdout: stdout.log
310+
$namespaces:
311+
iana: https://www.iana.org/assignments/media-types/
312+
edam: http://edamontology.org/

0 commit comments

Comments
 (0)