28

WARNING! If you downloaded the Cromwell 28 JAR prior to 7/17/2017 please redownload as we discovered and patched a critical bug.

28

Bug Fixes

WDL write_* functions add a final newline

The following WDL functions now add a newline after the final line of output (the previous behavior of not adding this
newline was inadvertent):

• write_lines
• write_map
• write_object
• write_objects
• write_tsv

For example:

task writer {
  Array[String] a = ["foo", "bar"]
  command {
    # used to output: "foo
bar"
    # now outputs: "foo
bar
"
    cat write_lines(a)
  }
}

ContinueWhilePossible

A workflow utilizing the WorkflowFailureMode Workflow Option ContinueWhilePossible will now successfully reach a terminal state once all runnable jobs have completed.

FailOnStderr

When FailOnStderr is set to false, Cromwell no longer checks for the existence of a stderr file for that task.

WDL Functions

New functions: floor, ceil and round:

Enables the floor, ceil and round functions in WDL to convert floating point numbers to integers.

For example we can now use the size of an input file to influence the amount of memory the task is given. In the example below a 500MB input file will result in a request for a VM with 2GB of memory:

task foo {
    File in_file
    command { ... }
    runtime {
      docker: "..."
      memory: ceil(size(in_file)) * 4 
    }
}

Call Caching

• Hash values calculated by Cromwell for a call when call caching is enabled are now published to the metadata.
  It is published even if the call failed. However if the call is attempted multiple times (because it has been preempted for example),
  since hash values are strictly identical for all attempts, they will only be published in the last attempt section of the metadata for this call.
  If the hashes fail to be calculated, the reason is indicated in a hashFailures field in the callCaching section of the call metadata.
  Important: Hashes are not retroactively published to the metadata. Which means only workflows run on Cromwell 28+ will have hashes in their metadata.

See the README for an example metadata response.

• New endpoint returning the hash differential for 2 calls.

GET /api/workflows/:version/callcaching/diff

See the README for more details.

Workflow Submission

• The workflow submission parameters wdlSource and wdlDependencies have been deprecated in favor of workflowSource and
  workflowDependencies respectively. The older names are still supported in Cromwell 28 with deprecation warnings but will
  be removed in a future version of Cromwell.

Labels

• A new /labels endpoint has been added to update labels for an existing workflow. See the README for more information.
• Label formatting requirements have been updated, please check the README for more detailed documentation.

JES Backend

The JES backend now supports a filesystems.gcs.caching.duplication-strategy configuration entry.
It can be set to specify the desired behavior of Cromwell regarding call outputs when a call finds a hit in the cache.
The default value is copy which will copy all output files to the new call directory.
A second value is allowed, reference, that will instead point to the original output files, without copying them.

filesystems {
  gcs {
    auth = "application-default"
    
    caching {
      duplication-strategy = "reference"
    }
  }
}

A placeholder file will be placed in the execution folder of the cached call to explain the absence of output files and point to the location of the original ones.

Metadata Write Batching

Metadata write batching works the same as in previous versions of Cromwell, but the default batch size has been changed from 1 to 200. It's possible that 200 is too high in some environments, but 200 is more likely to be an appropriate value
than the previous default.

27

27

Migration

• Call Caching has been improved in this version of Cromwell, specifically the time needed to determine whether or not a job can be cached
  has drastically decreased. To achieve that the database schema has been modified and a migration is required in order to preserve the pre-existing cached jobs.
  This migration is relatively fast compared to previous migrations. To get an idea of the time needed, look at the size of your CALL_CACHING_HASH_ENTRY table.
  As a benchmark, it takes 1 minute for a table with 6 million rows.
  The migration will only be executed on MySQL. Other databases will lose their previous cached jobs.
  In order to run properly on MySQL, the following flag needs to be adjusted: https://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_group_concat_max_len
  The following query will give you a minimum to set the group_concat_max_len value to:

SELECT MAX(aggregated) as group_concat_max_len FROM
     (
           SELECT cche.CALL_CACHING_ENTRY_ID, SUM(LENGTH(CONCAT(cche.HASH_KEY, cche.HASH_VALUE))) AS aggregated
           FROM CALL_CACHING_HASH_ENTRY cche
           GROUP BY cche.CALL_CACHING_ENTRY_ID
     ) aggregation

Here is the SQL command to run to set the group_concat_max_len flag to the proper value:

SET GLOBAL group_concat_max_len = value

Where value is replaced with the value you want to set it to.

Note that the migration will fail if the flag is not set properly.

Breaking Changes

• The response from the metadata endpoint now returns compressed HTTP responses depending on the request's headers using the following rules:

Accept-Encoding header	Resulting response
Accept-Encoding: gzip	compressed with Gzip
Accept-Encoding: deflate	compressed with Deflate
Accept-Encoding: deflate, gzip	compressed with Gzip
Accept-Encoding: identity	uncompressed
no Accept-Encoding header present	compressed with Gzip

• The update to Slick 3.2 requires a database stanza to
  switch from using driver to profile.

database {
  #driver = "slick.driver.MySQLDriver$" #old
  profile = "slick.jdbc.MySQLProfile$"  #new
  db {
    driver = "com.mysql.jdbc.Driver"
    url = "jdbc:mysql://host/cromwell?rewriteBatchedStatements=true"
    user = "user"
    password = "pass"
    connectionTimeout = 5000
  }
}

Call Caching

Cromwell now supports call caching with floating Docker tags (e.g. docker: "ubuntu:latest"). Note it is still considered
a best practice to specify Docker images as hashes where possible, especially for production usages.

Within a single workflow Cromwell will attempt to resolve all floating tags to the same Docker hash, even if Cromwell is restarted
during the execution of a workflow. In call metadata the docker runtime attribute is now the same as the
value that actually appeared in the WDL:

   "runtimeAttributes": {
     "docker": "ubuntu:latest",
     "failOnStderr": "false",
     "continueOnReturnCode": "0"
   }

Previous versions of Cromwell rewrote the docker value to the hash of the Docker image.

There is a new call-level metadata value dockerImageUsed which captures the hash of the Docker image actually used to
run the call:

   "dockerImageUsed": "library/ubuntu@sha256:382452f82a8bbd34443b2c727650af46aced0f94a44463c62a9848133ecb1aa8"

Important note: There is a Cromwell-wide cache for the resolution of docker tags to docker hashes.
By default, this cache retains a tag -> hash association for 20 minutes before discarding it.
This means that if 2 workflows with a task using a docker tag (e.g. ubuntu:latest) are started within that time span, they will both be run with the same hash even if the tag latest is updated.
The purpose of this cache is to reduce the time spent by Cromwell resolving docker tags.
This cache can be adjusted or completely disabled in the configuration.

Docker

• The Docker section of the configuration has been slightly reworked
  An option to specify how a Docker hash should be looked up has been added. Two methods are available.
  "local" will try to look for the image on the machine where cromwell is running. If it can't be found, Cromwell will try to pull the image and use the hash from the retrieved image.
  "remote" will try to look up the image hash directly on the remote repository where the image is located (Docker Hub and GCR are supported)
  Note that the "local" option will require docker to be installed on the machine running cromwell, in order for it to call the docker CLI.
• Adds hash lookup support for public quay.io images.

WDL Feature Support

• Added support for the new WDL basename function. Allows WDL authors to get just the file name from a File (i.e. removing the directory path)
• Allows coercion of Map objects into Arrays of Pairs. This also allows WDL authors to directly scatter over WDL Maps.

Miscellaneous

• Adds support for JSON file format for google service account credentials. As of Cromwell 27, PEM credentials for PAPI are deprecated and support might be removed in a future version.

google {

  application-name = "cromwell"

  auths = [
    {
      name = "service-account"
      scheme = "service_account"
      json-file = "/path/to/file.json"
    }
  ]
}

General Changes

• The /query endpoint now supports querying by label. See the README for more information.
• The read_X standard library functions limit accepted filesizes. These differ by type, e.g. read_bool has a smaller limit than read_string. See reference.conf for default settings.

Release 26

Breaking Changes

• Failure metadata for calls and workflows was being displayed inconsistently, with different formats depending on the originating Cromwell version. Failures will now always present as an array of JSON objects each representing a failure. Each failure will have a message and a causedBy field. The causedBy field will be an array of similar failure objects. An example is given below:

failures: [{
  message: "failure1",
  causedBy: [{
    message: "cause1",
    causedBy: []
   }, {
    message: "cause2",
    causedBy: []
  }]
 }, {
  message: "failure2",
  causedBy: []
}]

Additional Upgrade Time

• Upgrading to Cromwell 26 will take additional time due to the migration of failure metadata. Cromwell will automatically run a database query during the upgrade which appears to be roughly linear to the number of rows in the METADATA_ENTRY table. You can estimate upgrade time using the following equation: time to migrate (in seconds) ~= (rows in METADATA_ENTRY) / 65000 Note that due to differences in hardware and database speed, this is only a rough estimate.

Config Changes

• Added a configuration option under system.io to throttle the number of I/O queries that Cromwell makes, as well as configure retry parameters.
  This is mostly useful for the JES backend and should be updated to match the GCS quota available for the project.

system.io {
  # Global Throttling - This is mostly useful for GCS and can be adjusted to match
  # the quota availble on the GCS API
  number-of-requests = 100000
  per = 100 seconds
  
  # Number of times an I/O operation should be attempted before giving up and failing it.
  number-of-attempts = 5
}

• Added a script-epilogue configuration option to adjust the logic that runs at the end of the scripts which wrap call executions.
  This option is adjustable on a per-backend basis. If unspecified, the default value is sync.

WDL Features

With Cromwell 26, Cromwell will support if x then y else z expressions (see: https://github.com/broadinstitute/wdl/blob/develop/SPEC.md#if-then-else). For example:

Boolean b = true
String s = if b then "value if True" else "value if False"

Release 25

External Contributors

• A special thank you to @adamstruck, @antonkulaga and @delocalizer for their contributions to Cromwell.

Breaking Changes

• Metadata keys for call caching are changed. All call caching keys are now in a callCaching stanza. Call cache read result has moved here and is now result. The allowResultReuse and effectiveCallCachingMode have moved here. The hit boolean is a simple indication of whether or not it was a hit, with no additional information. An example using the new format is:

"callCaching": {
  "hit": false,
  "effectiveCallCachingMode": "ReadAndWriteCache",
  "result": "Cache Miss",
  "allowResultReuse": true
}

Config Changes

• Added a field insert-batch-size to the database stanza which defines how many values from a batch insert will be processed at a time. This value defaults to 2000.
• Moved the config value services.MetadataService.metadata-summary-refresh-interval to services.MetadataService.config.metadata-summary-refresh-interval
• Added ability to override the default zone(s) used by JES via the config structure by setting genomics.default-zones in the JES configuration
• The cromwell server TCP binding timeout is now configurable via the config key webservice.binding-timeout, defaulted
  to the previous value 5s (five seconds) via the reference.conf.
• For MySQL users, a massive scalability improvement via batched DB writing of internal metadata events. Note that one must add rewriteBatchedStatements=true to their JDBC URL in their config in order to take advantage of this

General Changes

• Cromwell's WDL parser now recognizes empty array literals correctly, e.g. Array[String] emptyArray = [].
• Cromwell now applies default labels automatically to JES pipeline runs.
• Added support for new WDL functions:
  • length: (Array[X]) => Integer - report the length of the specified array
  • prefix: (String, Array[X]) => Array[String] - generate an array consisting of each element of the input array prefixed
    by a specified String. The input array can have elements of any primitive type, the return array will always have
    type Array[String].
  • defined: (Any) => Boolean - Will return false if the provided value is an optional that is not defined. Returns true in all other cases.
• Cromwell's Config (Shared Filesystem) backend now supports invocation of commands which run in a Docker image as a non-root user.
  The non-root user could either be the default user for a given Docker image (e.g. specified in a Dockerfile via a USER directive),
  or the Config backend could pass an optional "-u username" as part of the submit-docker command.
• In some cases the SFS backend, used for Local, SGE, etc., coerced WdlFile to WdlString by using .toUri. This
  resulted in strings prepended with file:///path/to/file. Now absolute file paths will not contain the uri scheme.
• Launch jobs on servers that support the GA4GH Task Execution Schema using the TES backend.
• Call caching: Cromwell will no longer try to use the cache for WDL tasks that contain a floating docker tag.
  Call caching will still behave the same for tasks having a docker image with a specific hash.
  See https://github.com/broadinstitute/cromwell#docker-tags for more details.
• Added docker hash lookup. Cromwell will try to lookup the hash for a docker image with a floating tag, and use that hash when executing the job. This will be reflected in the metadata where the docker runtime attribute will contains the hash that was used. If Cromwell is unable to lookup the docker hash, the job will be run with the original user defined floating tag. Cromwell is currently able to lookup public and private docker hashes for images on Docker Hub and Google Container Engine for job running on the JES backend. For other backends, cromwell is able to lookup public docker hashes for Docker Hub and Google Container Engine. See https://github.com/broadinstitute/cromwell#docker-tags for more details.

Database schema changes

• Added CUSTOM_LABELS as a field of WORKFLOW_STORE_ENTRY, to store workflow store entries.

Release 24

24

• When emitting workflow outputs to the Cromwell log only the first 1000 characters per output will be printed
• Added support for conditional (if) statements.
• Globs for Shared File System (SFS) backends, such as local or SGE, now use bash globbing instead of Java globbing, consistent with the JES backend.
• A new /version endpoint has been added that returns the version of the Cromwell being run in server mode. There is also a new -version flag for determining the Cromwell version in command line mode, java -jar cromwell.jar -version .

Release 23

• The meta and parameter_meta blocks are now valid within workflow blocks, not just task
• The JES backend configuration now has an option genomics-api-queries-per-100-seconds to help tune the rate of batch polling against the JES servers. Users with quotas larger than default should make sure to set this value.
• Added an option call-caching.invalidate-bad-cache-results (default: true). If true, Cromwell will invalidate cached results which have failed to copy as part of a cache hit.
• Timing diagrams and metadata now receive more fine grained workflow states between submission and Running.
• Support for the Pair WDL type (e.g. Pair[Int, File] floo = (3, "gs://blar/blaz/qlux.txt"))
• Added support for new WDL functions:
  • zip: (Array[X], Array[Y]) => Array[Pair[X, Y]] - align items in the two arrays by index and return them as WDL pairs
  • cross: (Array[X], Array[Y]) => Array[Pair[X, Y]] - create every possible pair from the two input arrays and return them all as WDL pairs
  • transpose: (Array[Array[X]]) => Array[Array[X]] compute the matrix transpose for a 2D array. Assumes each inner array has the same length.
• Support for sub workflows (see Annex A)
• Support for WDL imports through an additional imports.zip parameter
• Corrected file globbing in JES to correctly report all generated files. Additionally, file globbing in JES now uses bash-style glob syntax instead of python style glob syntax
• Support declarations as graph nodes
• Added the ability to override the default service account that the compute VM is started with via the configuration option JES.config.genomics.compute-service-account or through the workflow options parameter google_compute_service_account. More details can be found in the README.md
• By default, system.abort-jobs-on-terminate is false when running java -jar cromwell.jar server, and true when running java -jar cromwell.jar run <wdl> <inputs>.
• Bug fix which allows Cromwell to successfully exit once a workflow completes in Single Workflow Runner Mode.

Annex A - Workflow outputs

The WDL specification has changed regarding workflow outputs to accommodate sub workflows.
This change is backward compatible in terms of runnable WDLs (WDL files using the deprecated workflow outputs syntax will still run the same).
The only visible change lies in the metadata (as well as the console output in single workflow mode, when workflow outputs are printed out at the end of a successful workflow).

TL;DR Unless you are parsing or manipulating the "key" by which workflow outputs are referenced in the metadata (and/or the console output for single workflow mode), you can skip the following explanation.

Metadata Response

{
  ...
  outputs {
    "task_output_1": "hello",
    "task_output_2": "world"
            ^
       If you don't manipulate this part of the metadata, then skip this section
  }
}

In order to maintain backward compatibility, workflow outputs expressed with the deprecated syntax are "expanded" to the new syntax. Here is an example:

task t {
    command {
        #do something
    }
    output {
        String out1 = "hello"
        String out2 = "world"
    }
}

    workflow old_syntax {
        call t
        output {
            t.*
        }
    }

    workflow new_syntax {
        call t
        output {
            String wf_out1 = t.out1
            String wf_out2 = t.out2
        }
    }

The new syntax allows for type checking of the outputs as well as expressions. It also allows for explicitly naming to the outputs.
The old syntax doesn't give the ability to name workflow outputs. For consistency reasons, Cromwell will generate a "new syntax" workflow output for each task output, and name them.
Their name will be generated using their FQN, which would give

output {
   String w.t.out1 = t.out1
   String w.t.out2 = t.out2
}

However as the FQN separator is ., the name itself cannot contain any ..
For that reason, . are replaced with _ :

Old syntax expanded to new syntax

output {
   String w_t_out1 = t.out1
   String w_t_out2 = t.out2
}

The consequence is that the workflow outputs section of the metadata for old_syntax would previously look like

    outputs {
        "w.t.out1": "hello",
        "w.t.out2": "hello"
    }

but it will now look like

    outputs {
        "w_t_out1": "hello",
        "w_t_out2": "hello"
    }

The same applies for the console output of a workflow run in single workflow mode.

0.22

0.22

• Improved retries for Call Caching and general bug fixes.
• Users will experience better scalability of status polling for Google JES.
• Now there are configurable caching strategies for a SharedFileSystem backend (i.e. Local, SFS) in the backend's stanza:
  See below for detailed descriptions of each configurable key.

backend {
  ...
  providers {
    SFS_BackendName {
      actor-factory = ...
      config {
        ...
        filesystems {
          local {
            localization: [
               ...
            ]
            caching {
              duplication-strategy: [
                "hard-link", "soft-link", "copy"
              ]
              # Possible values: file, path
              # "file" will compute an md5 hash of the file content.
              # "path" will compute an md5 hash of the file path. This strategy will only be effective if the duplication-strategy (above) is set to "soft-link",
              # in order to allow for the original file path to be hashed.
              hashing-strategy: "file"

              # When true, will check if a sibling file with the same name and the .md5 extension exists, and if it does, use the content of this file as a hash.
              # If false or the md5 does not exist, will proceed with the above-defined hashing strategy.
              check-sibling-md5: false
            }

• Multiple Input JSON files can now be submitted in server mode through the existing submission endpoint: /api/workflows/:version.
  This endpoint accepts a POST request with a multipart/form-data encoded body. You can now include multiple keys for workflow inputs.

  Each key below can contain an optional JSON file of the workflow inputs. A skeleton file can be generated from wdltool using the "inputs" subcommand.
  NOTE: In case of key conflicts between multiple JSON files, higher values of x in workflowInputs_x override lower values. For example, an input
  specified in workflowInputs_3 will override an input with the same name that was given in workflowInputs or workflowInputs_2. Similarly, an input
  specified in workflowInputs_5 will override an input with the same name in any other input file.
  
  workflowInputs
  workflowInputs_2
  workflowInputs_3
  workflowInputs_4
  workflowInputs_5
  

• You can now limit the number of concurrent jobs for a backend by specifying the following option in the backend's config stanza:

backend {
  ...
  providers {
    BackendName {
      actor-factory = ...
      config {
        concurrent-job-limit = 5

0.21

Cromwell Change Log

0.21

• Warning: Significant database updates when you switch from version 0.19 to 0.21 of Cromwell.
  There may be a long wait period for the migration to finish for large databases.
  Please refer to MIGRATION.md for more details.
• There are significant architectural changes related to increases in performance and scaling.
• The biggest user-facing changes from 0.19 to 0.21 are related to the application.conf file, which has been restructured significantly.
  The configuration for backends now is all contained within a backend stanza, which specifies 1 stanza per name per backend and a default backend, as follows:

backend {
    default=Local
    providers {
        Local {
            actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
            config {
                ... backend specific config ...
            }
        }
        JES {
            actor-factory = "cromwell.backend.impl.jes.JesBackendLifecycleActorFactory"
            config {
                ... backend specific config ...
            }
        }
        SGE {
            actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
            config {
                ... backend specific config ...
            }r
        }
    }
}

• A new /stats endpoint has been added to get workflow and job count for a Cromwell running in server mode.
• Renamed Workflow Options:
  “workflow_log_dir” -> “final_workflow_log_dir”
  “call_logs_dir” -> “final_call_logs_dir”
  “outputs_path” -> “final_workflow_outputs_dir”
  “defaultRuntimeOptions” -> “default_runtime_attributes”
• Timing diagrams endpoint has been updated to include additional state information about jobs.
• Add support for Google Private IPs through noAddress runtime attribute. If set to true, the VM will NOT be provided with a public IP address.
  Important: Your project must be whitelisted in "Google Access for Private IPs Early Access Program". If it's not whitelisted and you set this attribute to true, the task will hang.
  Defaults to false.
  e.g:

task {
    command {
        echo "I'm private !"
    }

    runtime {
        docker: "ubuntu:latest"
        noAddress: true
    }
}

• The Local and the SGE backend have been merged into a generic
  Shared File System (SFS) backend. This updated backend can be configured
  to work with various other command line dispatchers such as LSF. See the
  README for more info.
• On the JES and SFS backends, task command blocks are now always
  passed absolute paths for input Files.
• On the SFS backends, the call directory now contains two sub-directories:
  • inputs contains all the input files that have been localized for this task (see next below for more details)
  • execution contains all other files (script, logs, rc, potential outputs etc...)
• Override the default database configuration by setting the keys
  database.driver, database.db.driver, database.db.url, etc.
• Override the default database configuration by setting the keys
  database.driver, database.db.driver, database.db.url, etc.

For example:

# use a mysql database
database {
  driver = "slick.driver.MySQLDriver$"
  db {
    driver = "com.mysql.jdbc.Driver"
    url = "jdbc:mysql://host/cromwell"
    user = "user"
    password = "pass"
    connectionTimeout = 5000
  }
}

0.20

• The default per-upload bytes size for GCS is now the minumum 256K
  instead of 64M. There is also an undocumented config key
  google.upload-buffer-bytes that allows adjusting this internal value.
• Updated Docker Hub hash retriever to parse json with custom media
  types.
• Added a /batch submit endpoint that accepts a single wdl with
  multiple input files.
• The /query endpoint now supports querying by id, and submitting
  parameters as a HTTP POST.

0.19.3

• Fix an issue with symbolic linking localization strategy. Intermediate directories were not being created.
• Add LSF backend

0.19

• Workflow options may now contain a choice of backend specific for that workflow:

{
  "backend": "JES"
}

• To support the above change, the configuration file has changed how backends are specified. You must replace backend.backend with backend.defaultBackend.
  In addition the option backend.backendsAllowed must be specified (and should include the default), for example:

backend {
  defaultBackend = "local"
  backendsAllowed = [ "local", "JES", "SGE" ]
  ...

• New runtime option for JES: bootDiskSizeGb. Allows specification of a boot disk size (as an Integer number of GB) that can be increased to boot a larger docker image.
• Workflow options now allows you to specify a workflowFailureMode to control workflow behavior after a call has failed, for example:{ "workflowFailureMode": "..." }. The options are:
  • ContinueWhilePossible - continues to start and process calls in the workflow, as long as they did not depend on the failing call
  • NoNewCalls - no new calls are started but existing calls are allowed to finish
  • The default is NoNewCalls but this can be changed using the workflow-options.workflow-failure-mode configuration option.
• Bug fix: Tasks that changed directory would fail on JES because their return code file was written to the new directory instead of an absolute path
• Bug fix: Using write_* functions in a Task's command (e.g. ./my_script --file=${write_file(my_array)}) will now work with JES
• Changing format of the 'disks' runtime attribute slightly to allow for mounting disks at specific mountpoints

task disk_test {
  command { ... }
  runtime {
    disks: "local-disk 20 SSD, /mnt/mnt1 200 HDD"
  }
}

• Metadata now contains a list of failures for calls and workflows. This is an optional element of both call and workflow and is shaped thus:

"failures": [
  {
    "failure": "The failure message",
    "timestamp": "2016-02-25T10:49:02.066-05:00"
  }
]

• Added workflow options to copy the call logs and/or the workflow logs to a call_logs_dir or a workflow_log_dir,
  respectively.
• The system properties LOG_MODE and LOG_LEVEL used by Logback may now be specified as environment variables.
• Implemented write_tsv function
• Support sub function from the WDL Standard Library. See https://github.com/broadinstitute/wdl/blob/subFunction/SPEC.md#string-substring-string-string