surface_fraction error with 2 GPUs

[juliasloan25]

When we run simulations on 2 GPUs, we get an error that the surface fractions don't sum up to 1. This doesn't happen when running on 1 or 4 GPUs. See examples on buildkite here or here. Seen both when run on buildkite and when run via slurm directly on clima

stacktrace:

ERROR: LoadError: AssertionError: minimum((cs.surface_fractions.ice .+ cs.surface_fractions.land) .+ cs.surface_fractions.ocean) ≈ FT(1)
--
  | Stacktrace:
  | [1] update_surface_fractions!(cs::CoupledSimulation{…})
  | @ ClimaCoupler.Regridder /scratch/clima/slurm-buildkite/climacoupler-target-gpu-simulations/44/climacoupler-target-gpu-simulations/src/Regridder.jl:513
  | [2] top-level scope
  | @ /scratch/clima/slurm-buildkite/climacoupler-target-gpu-simulations/44/climacoupler-target-gpu-simulations/experiments/AMIP/coupler_driver.jl:559
  | in expression starting at /scratch/clima/slurm-buildkite/climacoupler-target-gpu-simulations/44/climacoupler-target-gpu-simulations/experiments/AMIP/coupler_driver.jl:559

when run interactively, we see that minimum((cs.surface_fractions.ice .+ cs.surface_fractions.land) .+ cs.surface_fractions.ocean) == 0. We expect this sum to always be exactly 1 at each point on the sphere.

[juliasloan25]

it seems like this is related to CliMA/ClimaAtmos.jl#2993

[juliasloan25]

consider #807 - for a coupler solution we could consider using the get_field methods, if we test them and see that this issue doesn't arise using those

edit: we still have the same issue when using get_fields

[szy21]

FYI I got the same error with 8 or 12 GPUs (on derecho), but not with 16 GPUs.

[ph-kev]

Replacing the assert statements with

@assert minimum((ice_fraction .+ land_fraction .+ ocean_fraction) |> parent) ≈ FT(1)
@assert maximum((ice_fraction .+ land_fraction .+ ocean_fraction)) ≈ FT(1)

fix the problem.

Related issue: here

The problem should be fixed once this issue in ClimaCore is fixed.