Skip to content

Container configs #1515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 15 commits into
base: dev
Choose a base branch
from
Open

Container configs #1515

wants to merge 15 commits into from

Conversation

edmundmiller
Copy link
Contributor

@edmundmiller edmundmiller commented Mar 17, 2025 •
edited
Loading

This is the first iteration of the larger effort of supporting ARM in nf-core pipelines.

Not the final state but trying to do this in steps to get to a final code product.

https://nf-co.re/blog/2024/seqera-containers-part-2

@edmundmiller edmundmiller added this to the 3.18.0 milestone Mar 17, 2025
@edmundmiller edmundmiller requested a review from ewels March 17, 2025 15:29
@edmundmiller edmundmiller self-assigned this Mar 17, 2025
@edmundmiller edmundmiller changed the base branch from master to dev March 17, 2025 15:29
@maxulysse
Copy link
Member

maxulysse commented Mar 17, 2025 •
edited
Loading

How do you generate/update the list?

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 17, 2025 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 2b678b1

+| ✅ 291 tests passed       |+
#| ❔   7 tests were ignored |#
!| ❗   9 tests had warnings |!

❗ Test warnings:

  • files_exist - File not found: assets/multiqc_config.yml
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_if_empty_null - ifEmpty(null) found in main.nf: _ versions = ch_versions.ifEmpty(null) // channel: [ versions.yml ]
    _

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 3.3.2
  • Run at 2025-09-16 19:57:27

@nf-core nf-core deleted a comment from github-actions bot Mar 17, 2025
@MatthiasZepper
Copy link
Member

Nice! I love to see some work in that direction and will need to take a closer look. A few quick questions that came to my mind at the first glance:

  • I seem to recall the idea was creating these configs upon release of a pipeline from YAMLs in the modules?
  • Do you think nf-core download should still run nextflow inspect by itself, or just read the config and use those paths?
  • Isn't it a bit dicey that we are always using the rnaseq pipeline for experimentation? After all, it is the most used nf-core pipeline, so any oversight or bug will affect lots of people and projects.

@edmundmiller
Copy link
Contributor Author

  • I seem to recall the idea was creating these configs upon release of a pipeline from YAMLs in the modules?

Yeah, that was the original idea. I think we've fallen into the waterfall trap and tried to plan before we implemented it. Still a high possibility, but just building out the parts that we can at this point, and trying to figure out where users are getting value out of it.

  • Do you think nf-core download should still run nextflow inspect by itself, or just read the config and use those paths?

Either or! I think it'll make everyone's lives easier if we at least write down the containers somewhere(and maybe back them up) on a version release. I'd imagine you could use the "latest" on nf-core download.

  • Isn't it a bit dicey that we are always using the rnaseq pipeline for experimentation? After all, it is the most used nf-core pipeline, so any oversight or bug will affect lots of people and projects.

This one specifically had to be rnaseq(for now) because I knew all of the software built on ARM via conda. So I 100% agree it's dicey to use for experimentation. It's also got the most eyes on it, so bugs are more likely to be caught sooner rather than later. For example, you and @maxulysse hopping on draft PR, moments after I open it and asking questions 😉

delagoya
delagoya reviewed Mar 18, 2025
View reviewed changes
conf/containers/containers_docker_arm64.config Outdated
process { withName: 'GFFREAD' { container = 'community.wave.seqera.io/library/gffread:0.12.7--33b95f1cfcc0e572' } }
process { withName: 'GTF2BED' { container = 'community.wave.seqera.io/library/perl:5.26.2--e6570c5f08d3e12c' } }
process { withName: 'GTF_FILTER' { container = 'community.wave.seqera.io/library/python:3.9.5--0ee01a3faebf676a' } }
process { withName: 'GUNZIP' { container = 'community.wave.seqera.io/library/grep_sed_tar:4f61255d5eff93dc' } }
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an linux/amd64 container

$ docker inspect community.wave.seqera.io/library/grep_sed_tar:4f61255d5eff93dc | grep Architecture
        "Architecture": "amd64",

delagoya
delagoya reviewed Mar 18, 2025
View reviewed changes
@delagoya
Copy link

Tested this out on AWS Graviton instance and did not work. What did work was copying the containers from the deleted conf/arm.config to conf/containers/containers_docker_arm64.config and removing --platform=linux/amd64 from profiles.arm. docker.runOptions in nextflow.config file

@edmundmiller edmundmiller mentioned this pull request Mar 18, 2025
Closed
@ewels
Copy link
Member

ewels commented Mar 19, 2025

Yeah, containers_docker_arm64.config seems to be identical to containers_docker_amd64.config so need to update the arm config to have the arm containers.

Also need several more config files in here - singularity arm / amd, https / oras, conda-lock for example. But moving in the right direction.

@edmundmiller
Copy link
Contributor Author

@delagoya Good catch! And thanks for the docker inspect snippet:

$ docker inspect community.wave.seqera.io/library/coreutils:9.5--3931e2d27d9884e5 | grep Architecture

        "Architecture": "arm64",

That looks better!

@edmundmiller edmundmiller marked this pull request as ready for review April 11, 2025 19:46
@edmundmiller edmundmiller moved this to Waiting to be merged in arm64 software builds Apr 11, 2025
@edmundmiller edmundmiller moved this to In progress in Seqera Containers Apr 11, 2025
mirpedrol
mirpedrol reviewed Apr 14, 2025
View reviewed changes
nextflow.config
docker.runOptions = '-u $(id -u):$(id -g) --platform=linux/amd64'
includeConfig 'conf/arm.config'
}
arm {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you mentioned in nf-core/tools#3536, this should be renamed but with a different name, to avoid confusion.
The rest looks good :)

mashehu
mashehu reviewed Apr 15, 2025
View reviewed changes
@pinin4fjords
Copy link
Member

@edmundmiller this is looking good, but not baked yet, removing it from the milestone since we're about to release

@pinin4fjords pinin4fjords removed this from the 3.19.0 milestone Jun 4, 2025
@edmundmiller edmundmiller moved this from In Progress to In Review in nf-core infrastructure projects Sep 9, 2025
ewels
ewels requested changes Sep 9, 2025
View reviewed changes
Copy link
Member

@ewels ewels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strip back to docker arm builds for now, with a TODO comment to come back to it when the singularity image stuff is fixed.

@github-project-automation github-project-automation bot moved this from In Review to In Progress in nf-core infrastructure projects Sep 9, 2025
@nf-core-bot
Copy link
Member

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.2.0.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

edmundmiller and others added 2 commits September 16, 2025 14:54
@edmundmiller
build: Rename arm profile to apple and make a real arm profile
9e22ce2 
Update container versions for ARM architecture and
@edmundmiller @claude
ci: optimize container config workflow
2b678b1 
- Update to Nextflow stable version 25.04.7
- Strip back to docker builds only (singularity disabled temporarily)
- Update ARM profile from 'arm' to 'arm64' following nf-core standards
- Add path filter to run only on modules.json changes

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@delagoya delagoya delagoya left review comments

@mashehu mashehu mashehu left review comments

@mirpedrol mirpedrol mirpedrol left review comments

@ewels ewels ewels requested changes

@bentsherman bentsherman Awaiting requested review from bentsherman

Requested changes must be addressed to merge this pull request.

Assignees

@edmundmiller edmundmiller

Labels
None yet
Projects
Seqera Containers
Status: In progress
arm64 software builds
Status: Waiting to be merged
nf-core infrastructure projects
Status: In Progress
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@edmundmiller @maxulysse @MatthiasZepper @delagoya @ewels @pinin4fjords @nf-core-bot @mashehu @mirpedrol