add GPU AMIP scaling runs #673
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
juliasloan25
force-pushed
the
js/gpu-scaling
branch
from
f7a9b60 to
8aeaeed
Compare
juliasloan25
force-pushed
the
js/gpu-scaling
branch
from
8aeaeed to
41dff6f
Compare
juliasloan25
force-pushed
the
js/gpu-scaling
branch
from
4d1f1a2 to
9d69913
Compare
|
See here: - SLURM_GPU_BIND: none # https://github.com/open-mpi/ompi/issues/11949#issuecomment-1737712291
+ SLRUM_GRES_FLAGS: "allow-task-sharing" |
juliasloan25
force-pushed
the
js/gpu-scaling
branch
from
48844ec to
503612c
Compare
juliasloan25
force-pushed
the
js/gpu-scaling
branch
2 times, most recently
from
e8da583 to
c357da3
Compare
juliasloan25
force-pushed
the
js/gpu-scaling
branch
from
c357da3 to
9031045
Compare
juliasloan25
force-pushed
the
js/gpu-scaling
branch
from
2e39025 to
187e958
Compare
juliasloan25
force-pushed
the
js/gpu-scaling
branch
from
187e958 to
f77dfab
Compare
| @@ -507,6 +507,9 @@ function update_surface_fractions!(cs::CoupledSimulation) | |||
| cs.surface_fractions.ice .= max.(min.(ice_d, FT(1) .- land_s), FT(0)) | |||
| cs.surface_fractions.ocean .= max.(FT(1) .- (cs.surface_fractions.ice .+ land_s), FT(0)) | |||
|
|
|||
| comms_ctx = axes(land_s).grid.topology.context | |||
There was a problem hiding this comment.
Suggested change
| comms_ctx = axes(land_s).grid.topology.context | |
| comms_ctx = ClimaComms.context(land_s) |
I think we have a helper for this, so that you don't need to use internals.
| key: "gpu_amip_dyamond_ws" | ||
| command: | ||
| - > | ||
| julia --threads=3 --color=yes --project=experiments/AMIP experiments/AMIP/coupler_driver.jl |
There was a problem hiding this comment.
Can we add an nsys profile to the non-MPI jobs?
14 tasks
|
CHAP is no longer our target; we want to set up scaling runs using the benchmarks pipeline and setups in the future |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
closes #663
note: we want to use the
slurm_exclusive:flag to get a GPU that's being used for only the performance job, but this seems not to be working when used in AtmosNeed to use ClimaAtmos#fdf1df4 commit to include PR containing atmos config files. Can't use main because of dependency incompatibilities.
Content
Status
2 GPU error is preventing us from getting scaling results - see #687. This needs to be addressed before we can set up reliable scaling runs