Data lineage channel factories

[jorgee]

Initial fromLineage factory implementation

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	cdd9e89
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/680be16db7ea9d0008b38953

[bentsherman]

I think we can have a nice unification of the CLI and programmatic API here

Viewing a single LID:

• CLI: view lid://<hash>[/<path>] -> use jq to inspect further
• API: channel.fromLineage('lid://<hash>[/<path>]') -> use json-path param to inspect further

Querying a collection of LIDs:

• CLI: find <name=value> <name=value> ...
• API: channel.queryLineage(foo: 'bar', baz: 'qux') -> returns queue channel of items, use json-path param to inspect further

This analysis suggests that the json-path functionality should be provided as a standalone function so that it can be used in different ways. You can add it to Nextflow.groovy. Actually you could just add it as an extra param to both APIs if that would be easier.

It might also make sense to refactor channel.fromLineage as a function instead of a channel factory, e.g. lineage(lid), because as a channel factory it will always return a value channel which is a needless constraint. The queryLineage factory makes sense because it can emit results while it's querying.

Finally, I think that find / queryLineage should really just use key-value pairs, rather than the URI query syntax which is unnecessary and error-prone

[jorgee]

I have pushed what I showed today.

• Files published with PublishDir included in outputs
• fromLinage channel factory: It should return the same as CLI view command:
  Usage: Channel.fromLineage("lid://xxxx")
  -queryLineage channel factory: It return a channel with the LinPaths matching with the queryString. Almost the same as the CLI find command
  Usage: Channel.queryString("type=FileOutput&annotations.value=test")

[bentsherman]

Let's just use a Map here instead of query string. I would apply the same change to the find command as well. There is no need to add the extra complexity of URL encoding

[bentsherman]

Let's keep this PR focused on the channel factories, we can address publishDir support separately

[bentsherman]

We don't need to add query support to fromPath because it will already be supported through queryLineage. Besides I would like to get away from using URI query params everywhere.

The queryLineage factory should return a channel of metadata objects, so you can chain it with a map operator to extract the actual files

[jorgee]

It seems one of the use cases that would be nice to support is to get outputs annotated with some metadata. This is only possible to do with a query, so I implemented it within fromPath and queryLineage to see what's the best option.

I had doubts about what queryLineage should return. Command find is returning the lids, not the objects. I returned the LinPaths thinking in the mentioned use case. If the LinPath is a FileOutput, it accesses the real file, checking the integrity. If we return the FileOutput description, users must get the path and check the integrity.

Maybe the best is returning the lid, letting the user do what they want according to the query. If it is cast to Path, it will be converted to a LinPath, and if it is passed to the lineage function, it will be converted to the object.