42

42 Release Notes

Womtool endpoint

The /describe endpoint now differentiates between an invalid workflow and a valid workflow with invalid inputs.

Specifically, the new validWorkflow key indicates whether the workflow file is valid by itself. If inputs are provided, they are not considered when calculating this field; if inputs are not provided, the value is identical to valid.

Configuration Changes

• Virtual private networks can now be configured. See the section below for details.

Batch Request Timeouts

The timeout on Cromwell's requests to PAPIv2 can now be configured. See the sample PAPIv2.conf for more documentation:

backend {
  providers {
    PAPIv2 {
      config { 
        batch-requests {
          timeouts {
            read = 10 seconds
            connect = 10 seconds
          }
        }
      }
    }
  }
}

Virtual Private Networks

Cromwell now allows PAPIV2 jobs to run on a private network by adding the network name inside virtual-private-cloud in backend configuration.
More info here.

AWS Backend

Now includes background job status polling to hopefully reduce the incidence of 'HTTP 429' errors for large workflows.

41

41 Release Notes

Workflow Options

• It is now possible to supply custom google-labels in workflow options.

AWS backend

It is now possible to use WDL disk attributes with the following formats on AWS.

disks: "local-disk 20 SSD"

disks: "/some/mnt 20 SSD"

Because Cromwell's AWS backend auto-sizes disks, the size specification is simply discarded.

Time Formatting

In previous versions of Cromwell, times were converted to strings using
the default Java formatter which
generates a variety of ISO-8601 formats. String conversions also retained whatever server time zone generated that
specific time instance.

Going forward, times stored in Cromwell metadata, and later returned via the HTTP endpoint, are now converted to UTC
then formatted with exactly three digits of milliseconds.

For example:

• 2017-01-19T12:34:56-04:00 will now be formatted as
• 2017-01-19T16:34:56.000Z

This change only affects newly formatted dates. Older dates already formatted and stored by previous versions of
Cromwell will not be updated however they will still return a
valid ISO-8601 format. The older format may be in various non-UTC time zones,
and may or may not include microseconds or even nanoseconds, for example 2017-01-19T12:34:56.123456789-04:00.

Config Changes

Heartbeat failure shutdown

When a Cromwell instance is unable to write heartbeats for some period of time it will automatically shut down. For more
information see the docs on configuring Workflow Hearbeats.

NOTE: In the remote chance that the system.workflow-heartbeats.ttl has been configured to be less than 5 minutes
then the new configuration value system.workflow-heartbeats.write-failure-shutdown-duration must also be explicitly
set less than the ttl.

nVidia Driver Attribute Change

The runtime attribute nvidia-driver-version was previously allowed only as a default runtime attribute in configuration.
Because WDL does not allow attribute names to contain - characters, this has been changed to nvidiaDriverVersion.
This field is now accepted within WDL files as well as within the configuration file.

Logging long running jobs

All backends can now emit slow job warnings after a configurable time running.
NB This example shows how to configure this setting for the PAPIv2 backend:

# Emit a warning if jobs last longer than this amount of time. This might indicate that something got stuck.
backend {
  providers {
    PAPIv2 {
      config { 
        slow-job-warning-time: 24 hours
      }
    }
  }
}

Runtime Attributes

GPU Attributes

• The gpuType attribute is no longer validated against a whitelist at workflow submission time. Instead, validation now happens at runtime. This allows any valid accelerator to be used.
• The nvidiaDriverVersion attribute is now available in WDL runtime sections. The default continues to be 390.46 which applies if and only if GPUs are being used.
• A default gpuType ("nvidia-tesla-k80") will now be applied if gpuCount is specified but gpuType is not.
• Similarly, a default gpuCount (1) will be applied if gpuType is specified but cpuCount is not.

Bug fixes

Better validation of workflow heartbeats

An error will be thrown on startup when the system.workflow-heartbeats.heartbeat-interval is not less than the
system.workflow-heartbeats.ttl.

40

40 Release Notes

Config Changes

Cromwell ID in instrumentation path

When set, the configuration value of system.cromwell_id will be prepended to StatsD metrics. More info here.

HealthMonitor Configuration

The HealthMonitor configuration has been refactored to provide a simpler interface:

• You no longer need to specify a monitor class in your cromwell.conf as this will now be inherited from the reference.conf value.
• You can now opt-in and opt-out of any combination of status monitors.
• The PAPI backends to monitor can now be listed in a single field.

Upgrading

You are no longer tied to the previous preset combinations of health checks. However if you just want to carry forward
the exact same set of health checks, you can use one of the following standard recipes:

From default, or NoopHealthMonitorActor:

If you're currently using the (default) NoopHealthMonitorActor, no action is required.

From StandardHealthMonitorServiceActor:

If you're currently using the StandardHealthMonitorServiceActor, replace this stanza:

services {
    HealthMonitor {
        class = "cromwell.services.healthmonitor.impl.standard.StandardHealthMonitorServiceActor"
    }
}

With this one:

services {
    HealthMonitor {
        config {
            check-dockerhub: true
            check-engine-database: true
        }
    }
}

From WorkbenchHealthMonitorServiceActor:

Replace this stanza:

services {
    HealthMonitor {
        class = "cromwell.services.healthmonitor.impl.workbench.WorkbenchHealthMonitorServiceActor"

        config {
            papi-backend-name = PAPIv1
            papi-v2-backend-name = PAPIv2

            google-auth-name = service-account
            gcs-bucket-to-check = "cromwell-ping-me-bucket"
        }
    }
}

With this one:

services {
    HealthMonitor {
        config {
            check-dockerhub: true
            check-engine-database: true
            check-gcs: true
            check-papi-backends: [PAPIv1, PAPIv2]

            google-auth-name = service-account
            gcs-bucket-to-check = "cromwell-ping-me-bucket"
    }
  }
}

Bug fixes

WDL 1.0 strings can contain escaped quotes

For example, the statement String s = "\"" is now supported, whereas previously it produced a syntax error.

Empty call blocks in WDL 1.0

Cromwell's WDL 1.0 implementation now allows empty call blocks, e.g. call task_with_no_inputs {}. This brings 1.0 in line with draft-2, which has always supported this syntax.

Packed CWL bugfix

Fixed a bug that caused an error like Custom type was referred to but not found to be issued when using an imported type as a SchemaDefRequirement in packed CWL.

39

39 Release Notes

Cromwell ID changes

When set, the configuration value of system.cromwell_id will now have a random suffix appended, unless the
configuration key system.cromwell_id_random_suffix is set to false.

The generated id also appears more places in the logs, including when picking up workflows from the database and during
shutdown.

Bug fixes

Format fix for write_map()

Fixed an issue that caused the write_map() function in Cromwell's WDL 1.0 implementation to produce output in the wrong format. Specifically, the output's rows and columns were swapped. WDL draft-2 was not affected.

Incorrect write_map() output in Cromwell 38 and earlier:

key1    key2    key3
value1  value2  value3

Corrected write_map() output in Cromwell 39 and later:

key1  value1
key2  value2
key3  value3

38

38 Release Notes

Docker images

• The docker image for this release is available using the tag:
  • broadinstitute/cromwell:38-bf46f39
• Other Cromwell family tools are also available:
  • broadinstitute/womtool:38-bf46f39
  • broadinstitute/cromiam:38-bf46f39

HPC paths with Docker

The ConfigBackendLifecycleActorFactory path variables script, out and err are now consistent when running with
and without docker. Similarly, when killing a docker task the kill-docker configuration key is now used instead of
kill. For more information see the online documentation.

No-op Health Monitor is now the default health monitor

Previous versions of Cromwell defaulted to using a health monitor service that checked Docker Hub and engine database status.
Neither check was useful if the status endpoint was never consulted as is likely the case in most deployments. Cromwell 38
now defaults to a NoopHealthMonitorServiceActor which does nothing. The previous health service implementation is still
available as StandardHealthMonitorServiceActor.

WOM API changes

• The toDisplayString method on WomType has been replaced with two possible alternatives.
  • stableName will return the same as the previous toDisplayString method. If in doubt over which to use, this option will ensure backwards compatibility.
  • friendlyName which is intended to be friendlier to display to humans.

Bug fixes

• Fixed an issue that could cause Cromwell to consume disk space unnecessarily when using zipped dependencies

HTTP responses

• When returning errors as json the Content-Type header is set to application/json.

36.1

36.1 Release Notes

Note: This is an interim release containing many of the good features in Cromwell 37. It should be used in preference to Cromwell 37 while we address some known issues in that version.

Docker images are available for this release:

• broadinstitute/cromwell:36-858f647
• broadinstitute/womtool:36-858f647

Changing configuration options

Logging Token Distribution

In cases where its not obvious why jobs are queued in Cromwell, you can enable logging for the Job Execution Token Dispenser, using
the system.hog-safety.token-log-interval-seconds configuration value.

The default, 0, means that no logging will occur.

Backend name for call caching purposes

Previous versions of Cromwell incorporated the name of the backend on which a call was run into the call cache hashes generated for that call.
Unfortunately this made it impossible to change the name of a backend without losing all previously run calls as potential cache hits.
Cromwell 36.1 introduces the name-for-call-caching-purposes backend configuration option as a means of decoupling the backend name from the
value used for the backend name for call caching purposes.

Docker

• Adds support for retrieving docker digests of asia.gcr.io images
• Adds configuration settings for docker digest lookups. See the docker section of the reference.conf for more information
• Attempt to automatically adjust the boot disk size on the Google Cloud Backend (version 2) if the size of the image is greater than the default disk size or the required disk size in the runtime attributes.
  Only works for registries that support the version 2 of the manifest schema (https://docs.docker.com/registry/spec/manifest-v2-2/)
  At this date (12/09/18) this includes GCR and Dockerhub.

Call cache blacklisting

The Google Pipelines API (PAPI) version 1 and 2 backends now offer the option of call cache blacklisting on a per-bucket basis.
More info here.

37

37 Release Notes

WARNING

1. Issues discovered with the HPC (SLURM, SGE, etc) backend.
2. Issues with metadata summarization, affecting workflow status updates

Docker

• Adds support for retrieving docker digests of asia.gcr.io images
• Adds configuration settings for docker digest lookups. See the docker section of the reference.conf for more information
• Attempt to automatically adjust the boot disk size on the Google Cloud Backend (version 2) if the size of the image is greater than the default disk size or the required disk size in the runtime attributes.
  Only works for registries that support the version 2 of the manifest schema (https://docs.docker.com/registry/spec/manifest-v2-2/)
  At this date (12/09/18) this includes GCR and Dockerhub.

Added new call cache path+modtime hashing strategy.

Call caching hashes with this new strategy are based on the path and the last modified time of the file.

Instance independent abort

For multi-instance Cromwell deployments sharing a single database, earlier versions of Cromwell required abort
requests to be sent specifically to the instance that was running the targeted workflow. Cromwell 37 now
allows abort commands to be sent to any Cromwell instance in the shared-database deployment. Configuration details
here.

Call cache blacklisting

The Google Pipelines API (PAPI) version 1 and 2 backends now offer the option of call cache blacklisting on a per-bucket basis.
More info here.

WDL

• All memory units in WDL are now treated as base-2.
  For instance 1 KB == 1 KiB == 1024 Bytes.

Backend name for call caching purposes

Previous versions of Cromwell incorporated the name of the backend on which a call was run into the call cache hashes generated for that call.
Unfortunately this made it impossible to change the name of a backend without losing all previously run calls as potential cache hits.
Cromwell 37 introduces the name-for-call-caching-purposes backend configuration option as a means of decoupling the backend name from the
value used for the backend name for call caching purposes.

CWL

Support InputResourceRequirement hint

Changing configuration options

Logging Token Distribution

In cases where its not obvious why jobs are queued in Cromwell, you can enable logging for the Job Execution Token Dispenser, using
the system.hog-safety.token-log-interval-seconds configuration value.

The default, 0, means that no logging will occur.

HTTP Filesystem

• The HTTP filesystem is now enabled for engine use by default. To continue without an HTTP filesystem, you can add the
  following content into the appropriate stanza of your configuration file:

engine {
  filesystems {
    http { 
      enabled: false 
    }
  }
}

• When the value exit-code-timeout-seconds is set, check-alive command is now only called once every timeout interval instead of each poll.

Beta preview of new Womtool /describe endpoint

This new endpoint brings the functionality of Womtool to the world of web services. Submit workflows for validation and receive a JSON description in response.

The endpoint is still undergoing heavy development and should not be used in production. The final version will ship in a future release of Cromwell; watch this space.

Bug fixes

• Fixed a regression in Cromwell 36 that could cause operations on empty arrays to fail with a spurious type error (closes #4318)

Abort On Hold Workflows

On Hold workflows may now be aborted.

Command fixes for AWS and TES

The AWS and TES backends can now handle calls that generate longer command lines. Like the other
backends, commands scripts are first written to a file, the file is downloaded to the execution
host, and then the localized script is run.

Also fixed are AWS command {} blocks that use | at the start of a line. For example:

command {
  echo hello world \
  | cat
}

36

36 Release Notes

Extra configuration options

The value exit-code-timeout-seconds can now set in a backend configuration.
Details here

Extra configuration options for Firecloud

In addition to a "PAPI" status key, the FC custom health monitor now supports "PAPIV1" and "PAPIV2" status keys.

AWS S3 file transfers are now encrypted

Bug fixes

Metadata Request Coalescing

Coalesce metadata requests to eliminate expensive and redundant queries and metadata construction.

Eliminate redundant SFS logging and metadata

Eliminate superfluous logging and metadata publishing in the shared filesystem backend on poll intervals where there was not a state change.

AWS region configuration respected throughout

Previously US-EAST-1 was hardcoded in places.

Server jar back under 200MB

Removed a large number of unused AWS dependencies.

Sub-workflows logging to the correct file

Sub-workflow logs are always written to the root-workflow's log instead of a separate log file that used to leak file pointers.

35

35 Release Notes

Submit workflow using URL

Cromwell now allows for a user to submit the URL pointing to workflow file to run a workflow.
More details on how to use it in:

• Server mode can be found here.
• Run mode can be found here.

Languages

• Added an opt-in namespace cache for the WDL Draft 2 language factory. Please see the Cromwell example configuration for details. NOTE: if upgrading from a hotfix version of Cromwell
  that relied upon this cache, the cache is now opt-in and must be turned on explicitly in config.
• To maintain conformance with the OpenWDL spec, Cromwell drops support for the version draft-3 identifier in this release. In the rare case where end users may have been using version draft-3, version 1.0 is a drop-in replacement with no effect on functionality.

HTTP Workflow Inputs for Shared File System and Google Pipelines API Version 2 Backends

http and https workflow inputs are now supported for shared filesystem and Google Pipelines API (PAPI) version 2
backends. Configuration details are described here.

Call cache hint support

More efficient cache hit copying in multi-user environments is now supported through the call_cache_hit_path_prefixes workflow option.
Details here

Extra configuration options

The value dockerRoot can now be set in a backend configuration.
This will set the execution folder in the container (default: /cromwell-executions).

Bug Fixes

API

• The releaseHold endpoint will now return 404 Not Found for an unrecognized workflow ID and 400 Bad Request for a malformed or invalid workflow ID.

Languages

• Fixed a bug that allowed values to be "auto-boxed" into a single-element Array of that type, which is not allowed in the WDL spec (Closes #3478).

PAPI version 1

• Restored standard output and error streaming for jobs.

34

34 Release Notes

Query API

• Fixes a bug which stopped includeSubworkflow=false from paging correctly and subworkflows from being discounted correctly from totalResultsCount.
• Query results will now be returned in reverse chronological order, with the most-recently submitted workflows returned first.

Requester Pays on GCS

Access of Google Cloud Storage buckets with Requester Pays enabled is now supported.
Please read the relevant documentation for information on how to enable it and the consequences.

Private Docker Support on Pipelines API v2

Support for private Docker Hub images is now included in the Google Pipelines API v2 backend. PAPI v2 private Docker support is
equivalent to that in PAPI v1 but the configuration differs, please see
Docker configuration for more details.

Updated MySQL client with 8.0 support

Updated the MySQL connector client from 5.1.42 to 5.1.46 which adds support for connecting to MySQL 8.0. See the
documentation on Changes in MySQL Connector/J for
more information.