Skip to content

Print process dictionary in logs (#5940) #5944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

kdesnos
Copy link
Contributor

@kdesnos kdesnos commented Apr 4, 2025

What

Update the logs in the following manner:

  1. Replace workflow process name list, which mixes defined process names and aliased process names

    [main] DEBUG nextflow.Session - Workflow process names [dsl2]: a
    

    with two lists: one of process definitions from each file, one of aliases used for each process.

    [main] DEBUG nextflow.Session - Workflow process definitions [dsl2]: main.nf [a, other_process, ...], sub.nf [a, ...]
    [main] DEBUG nextflow.Session - Workflow resolved process names: a[main.nf:a], sub:a[sub.nf:a], x[sub.nf:a]  
    
  2. Complement the following log line:

    [main] DEBUG nextflow.processor.TaskProcessor - Starting process > x
    

    with typed list of inputs and outputs of the process.

    Starting process > x (type:ArgName, ..., default:$) -> (type:OutName, ...)
    

Motivations

  • Makes process info printed logs more complete. Human parsing of nextflow files is no longer needed to easily identify where the code of a process originates from.
  • Corrects ambiguous merge of "conflicting" resolved names in the currently printed list: See Merge of conflicting process names in logs #5940 for more details.

Copy link

netlify bot commented Apr 4, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit e3d30ad
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/68143c57b06fdd0008c4fb21

@kdesnos kdesnos marked this pull request as ready for review April 4, 2025 04:50
@kdesnos
Copy link
Contributor Author

kdesnos commented Apr 4, 2025

Oops.. I thought I'd been very cautious not to alter any existing behavior when modifying the code.. but the unit tests seem broken.

After a quick look, I think one issue is that in ProcessDef unit test, ProcessDef instance is cloned without originating from a script already registered in ScriptMeta. Because of this, the cloneWithName() function I implemented fails. Of course, this scenario never occurr.s.ed in real workflow I tested my code with, as a cloned process always originates from a script.

Unfortunately, I'm off for two weeks so I can't have a look at why the tests are failing at the moment. (Also, I'm not using a proper IDE, only building and running, so I probably need to setup a proper dev environment to debug these issue.)

kdesnos added 5 commits April 28, 2025 10:42

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Signed-off-by: kdesnos <[email protected]>
@kdesnos kdesnos force-pushed the printProcessDictionnaryInLogs branch from d53b9ae to 6372902 Compare April 28, 2025 01:58
Signed-off-by: kdesnos <[email protected]>
@kdesnos kdesnos force-pushed the printProcessDictionnaryInLogs branch from 6372902 to 0bf2324 Compare April 28, 2025 01:59
@kdesnos
Copy link
Contributor Author

kdesnos commented Apr 28, 2025

The PR is ready to be merged.

…cates. Duplicates may appear when applying a configuration to an identical process alias used in several workflows.

Signed-off-by: kdesnos <[email protected]>
@kdesnos kdesnos marked this pull request as draft May 1, 2025 07:10
@kdesnos
Copy link
Contributor Author

kdesnos commented May 1, 2025

I noticed that the "resolved" name of the included process may be ambiguous, in case the same alias is used in different workflow, for different original process.

This is because when an include occurs in a workflow sub, say with process p being included as toto, the process is first cloned with name toto, before being re-cloned with the unique name sub:toto. If main workflow clones process q with the name toto then, the resolved name toto will appear twice, once for each workflow. The aliased process toto included in the main file, will not be recloned with a different name. Hence, the process toto will be executed, but it will be ambiguous whether it is the clone of p, or q.

I'll fix this issue before reopening the PR.

kdesnos added 2 commits May 2, 2025 09:41
…esolved name is no longer kept, only the name with full workflow path is, as in reports.

Signed-off-by: kdesnos <[email protected]>
…able

Signed-off-by: kdesnos <[email protected]>
@kdesnos kdesnos marked this pull request as ready for review May 2, 2025 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant