Implementation of Git multiple revisions #6620

jorgee · 2025-12-01T10:24:16Z

Summary

This PR implements Git multi-revision asset management, enabling concurrent execution of different pipeline revisions (e.g., production on v1.0, testing on v2.0-dev) while maintaining efficient disk usage through object sharing.

Closes #2870. Alternative for #5089.

Solution

The implementation uses a bare repository + shared clones approach with the Strategy pattern for backward compatibility:

~/.nextflow/assets/.repos/<org>/<project>/
├── bare/                    # Bare repository (shared objects)
│   ├── objects/             # All Git objects stored here
│   ├── refs/
│   └── config
│
└── clones/                  # Revision-specific clones
    ├── <commit-sha-1>/      # Clone using JGit alternates
    └── <commit-sha-2>/      # (points to bare/objects)

Key mechanisms:

Bare repository as source of truth: Fetched/updated from remote, keeping refs current
Dynamic resolution: Revisions resolved to commit SHAs using bare repo refs
Object sharing: Clones use JGit alternates to reference bare repo objects, avoiding duplication
Atomic operations: Each clone is independent; downloading new revisions doesn't affect running pipelines
Lazy creation: Clones created on-demand when a specific revision is requested

Backward Compatibility

Uses the Strategy pattern to maintain compatibility with existing repositories:

RepositoryStrategy interface with LegacyRepositoryStrategy and MultiRevisionRepositoryStrategy implementations
AssetManager acts as facade with unchanged public API
Strategy selection: NXF_SCM_LEGACY=true forces legacy mode, otherwise auto-detected based on repository state
Existing legacy repositories continue to work without migration

Changes

Core Implementation

Added RepositoryStrategy interface and AbstractRepositoryStrategy base class
Implemented MultiRevisionRepositoryStrategy for bare repo + shared clones
Implemented LegacyRepositoryStrategy preserving original behavior
Refactored AssetManager to delegate to strategy implementations

CLI Updates

Changed -a, --all-revisions flag to -a, --all in CmdDrop
Updated nf-info command output to show clones with revision indicators

Concurrency & Reliability

Added FileMutex for safe concurrent clone operations
Improved error handling with cleanup of incomplete clones

Documentation

ADR: adr/20251205-git-multi-revision-asset-management.md

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

… operation Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

…f "master" Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

Signed-off-by: Ben Sherman <[email protected]>

Signed-off-by: Dr Marco De La Pierre <[email protected]>

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

pditommaso · 2025-12-15T08:58:08Z

Special recognition for the excellent ADR documenting the decision-making process - this sets a great example for future architectural changes.

Wow! great 😆 🚀

Signed-off-by: Paolo Di Tommaso <[email protected]>

docs/reference/cli.md

modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy

modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy

modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy

modules/nextflow/src/main/groovy/nextflow/scm/GithubRepositoryProvider.groovy

pditommaso · 2025-12-15T09:23:23Z

Little glitch, but likely there isn't much it can be done here

» nextflow list
nextflow-io/rnaseq-nf
nf-core/modules
pditommaso/hello
.repos/nextflow-io

jorgee · 2025-12-15T09:53:57Z

Little glitch, but likely there isn't much it can be done here
» nextflow list
nextflow-io/rnaseq-nf
nf-core/modules
pditommaso/hello
.repos/nextflow-io

Is this when runninglistwith the previous version?

pditommaso · 2025-12-15T09:55:28Z

Yes

Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>

jorgee · 2025-12-15T10:02:06Z

I can create another PR with a patch ignoring the path to backport to stable releases.

…) [ci skip] Update the multi-revision asset management ADR: - Fix directory structure to match implementation: - bare/ instead of .nextflow/bare_repo/ - commits/ instead of .nextflow/commits/ - Legacy repo at separate location ~/.nextflow/assets/<project>/ - Merge Option 4 (Strategy Pattern) into Option 3 since the Strategy Pattern is an implementation detail of the multi-revision approach, not a standalone option Signed-off-by: Paolo Di Tommaso <[email protected]>

Signed-off-by: jorgee <[email protected]>

adr/20251205-multi-revision-asset-management-strategy-pattern.md

docs/cli.md

Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>

Signed-off-by: jorgee <[email protected]>

jorgee · 2025-12-17T12:39:57Z

Addressed the last review comments.
I have also double check the other two issues linked to the PR, and they do not happen when using multi-revision. Still happening when using legacy strategy. Users must migrate to fix these issues.

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso · 2025-12-17T13:18:59Z

Just replaced the • with > because i'm quite not all terminal can support it

Signed-off-by: Paolo Di Tommaso <[email protected]>

Signed-off-by: jorgee <[email protected]>

jorgee · 2025-12-17T14:12:27Z

Changing the symbol in docs

marcodelapierre and others added 30 commits January 15, 2024 17:29

Multiple revisions 1st: AssetManager, Cmd Clone Pull Run

7d027ec

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

removed revision paramter from clone pull run

106e8df

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

updated unit tests for AssetManager

0a9e6f8

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

Merge branch 'master' into add/mult_revisions

37d332e

AssetManager provider.revision assigned at AssetManager object creation

5a9b159

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

Codespell typo in changelog

94d34f3

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

AssetManager: removed checkout method

4744774

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

Merge branch 'master' into add/mult_revisions

d9e7b2b

assetmanager: fixed build signature, and unit tests

08718ef

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

assetmanager: one more build signature fix

242a600

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

AssetManager: fix for multi revs in find() method

35ad1e9

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

Multiple revisions: added for Cmds Drop, View, Config, Info

463b6f1

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

minor fixes to log outputs in CmdRun

2d3273e

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

AssetManager: documented new localPath schema

d2e5ffe

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

K8sDriverLauncher : added revision support

60c3c5e

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

updates to AssetManagerTest

4bf1e37

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

edit to AssetManagerTest, git.pull for TAGS does not output result of…

7e32111

… operation Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

minor fix in GitlabRepositoryProvider: using DEFAULT_BRANCH instead o…

3b2d461

…f "master" Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

Minor edits

2e6461e

Signed-off-by: Ben Sherman <[email protected]>

Merge branch 'master' into add/mult_revisions

a8d2dee

[ci fast] Merge branch 'master' into add/mult_revisions

eaf45d8

Merge branch 'master' into add/mult_revisions

ed17284

parametrised revision delimiter

b3f141d

Signed-off-by: Dr Marco De La Pierre <[email protected]>

nf pull: option to list or not revs for each project

6706bd9

Signed-off-by: Dr Marco De La Pierre <[email protected]>

nicer output for nf list -r

e62f543

Signed-off-by: Dr Marco De La Pierre <[email protected]>

minor edit to CmdList

bd19877

Signed-off-by: Dr Marco De La Pierre <[email protected]>

AssetManager: adding listRevisions method (work in progress)

c0268af

Signed-off-by: Dr Marco De La Pierre <[email protected]>

small tune to CmdList

aae2afc

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

minor tweak to CmdList

268ec79

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

AssetManager: listRevisions method now working

ac8d755

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

pditommaso added 2 commits December 15, 2025 10:00

Merge branch 'master' into mult_revisions_no_map

22d6101

Refactor CmdPull to use for loop instead of each closure [ci fast]

f00d6a2

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso requested changes Dec 15, 2025

View reviewed changes

Update docs/reference/cli.md

7fcf3c6

Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>

pditommaso and others added 2 commits December 15, 2025 11:27

address review comments

f30022e

Signed-off-by: jorgee <[email protected]>

pditommaso reviewed Dec 15, 2025

View reviewed changes

adr/20251205-multi-revision-asset-management-strategy-pattern.md Outdated Show resolved Hide resolved

pditommaso changed the title ~~Implementation of multiple revisions without revisions map~~ Implementation of Git multiple revisions Dec 15, 2025

pditommaso reviewed Dec 15, 2025

View reviewed changes

jorgee and others added 8 commits December 15, 2025 17:09

Update adr/20251205-multi-revision-asset-management-strategy-pattern.md

27a6acf

Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>

Apply suggestions from code review [ci fast]

575dd40

Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>

Remove unnecesary RepositoryStatus

d48a798

Signed-off-by: jorgee <[email protected]>

fix test

509e70e

Signed-off-by: jorgee <[email protected]>

Merge branch 'master' into mult_revisions_no_map

f37b17c

change commits dir by revs and '*' by '·' for checkouts

5e81412

Signed-off-by: jorgee <[email protected]>

fix info command in docs

3fe441a

Signed-off-by: jorgee <[email protected]>

fix test and change revs to clones

b517c10

Signed-off-by: jorgee <[email protected]>

jorgee requested a review from pditommaso December 17, 2025 12:40

Minor change [ci fast]

2883bc7

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso approved these changes Dec 17, 2025

View reviewed changes

pditommaso and others added 2 commits December 17, 2025 14:26

Rename adr filename [ci skip]

df8731c

Signed-off-by: Paolo Di Tommaso <[email protected]>

Change '·' by '>' in docs [ci fast]

9adaa63

Signed-off-by: jorgee <[email protected]>

Implementation of Git multiple revisions #6620

Are you sure you want to change the base?

Implementation of Git multiple revisions #6620

Conversation

jorgee commented Dec 1, 2025 • edited by pditommaso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Solution

Backward Compatibility

Changes

Core Implementation

CLI Updates

Concurrency & Reliability

Documentation

Uh oh!

pditommaso commented Dec 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pditommaso commented Dec 15, 2025

Uh oh!

jorgee commented Dec 15, 2025

Uh oh!

pditommaso commented Dec 15, 2025

Uh oh!

jorgee commented Dec 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorgee commented Dec 17, 2025

Uh oh!

pditommaso commented Dec 17, 2025

Uh oh!

jorgee commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jorgee commented Dec 1, 2025 •

edited by pditommaso

Loading