Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem using immersed boundary with PartialCellBottom and Flat horizontal dimensions #2798

Closed
francispoulin opened this issue Nov 3, 2022 · 10 comments
Closed

Problem using immersed boundary with PartialCellBottom and Flat horizontal dimensions #2798

francispoulin opened this issue Nov 3, 2022 · 10 comments
Labels
immersed boundaries ⛰️ Less Ocean, more anigans question 💭 No such thing as a stupid question

Comments

@francispoulin
Copy link
Collaborator

After updating to julia 1.8.2 and updating branch to main, I tried running the validaton experiment resting_stratified_bumpy_ocean.jl and it failed. @fadaie91 also tried the same thing, and saw the same error.

The problem is when we define the model, it complains about a nested task error.

grid = 1×128×64 ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded} on CPU with 0×3×3 halo:
├── immersed_boundary: PartialCellBottom(min(h)=-1.00e+00, max(h)=-5.00e-01, ϵ=0.2)
├── underlying_grid: 1×128×64 RectilinearGrid{Float64, Flat, Periodic, Bounded} on CPU with 0×3×3 halo
├── Flat x
├── Periodic y ∈ [-1.0, 1.0)      regularly spaced with Δy=0.015625
└── Bounded  z ∈ [-1.0, 0.0]      regularly spaced with Δz=0.015625
ERROR: LoadError: TaskFailedException

    nested task error: BoundsError: attempt to access 1×134×1 OffsetArray(::Array{Float64, 3}, 1:1, -2:131, 1:1) with eltype Float64 with indices 1:1×-2:131×1:1 at index [0, 1]
    Stacktrace:
      [1] throw_boundserror(A::OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, I::Tuple{Int64, Int64})
        @ Base ./abstractarray.jl:703
      [2] overdub
        @ ~/.julia/packages/KernelAbstractions/3ZHln/src/compiler.jl:51 [inlined]
      [3] overdub
        @ ./abstractarray.jl:668 [inlined]
      [4] overdub
        @ ./abstractarray.jl:1273 [inlined]
      [5] overdub
        @ ./abstractarray.jl:1241 [inlined]
      [6] overdub
        @ ~/Software/Oceananigans.jl/src/ImmersedBoundaries/partial_cell_immersed_boundaries.jl:49 [inlined]
...
@glwagner
Copy link
Member

glwagner commented Nov 3, 2022

@francispoulin can you post the whole code that creates this error?

@fadaie91
Copy link
Contributor

fadaie91 commented Nov 3, 2022 •
edited
Loading

@glwagner Here is the whole code:

And the error is after defining the "model".

using Oceananigans
using Oceananigans.BoundaryConditions: fill_halo_regions!
using Oceananigans.ImmersedBoundaries: ImmersedBoundaryGrid, GridFittedBottom, PartialCellBottom
using Printf
using GLMakie

arch = CPU()
tracer_advection = CenteredSecondOrder()
momentum_advection = CenteredSecondOrder()

underlying_grid = RectilinearGrid(arch,
                                  size=(128, 64), halo=(3, 3), 
                                  y = (-1, 1),
                                  z = (-1, 0),
                                  topology=(Flat, Periodic, Bounded))

# A bump
h₀ = 0.5 # bump height
L = 0.25 # bump width
@inline h(y) = h₀ * exp(- y^2 / L^2)
@inline seamount(x, y) = - 1 + h(y)

seamount_field = Field{Center, Center, Nothing}(underlying_grid)
set!(seamount_field, seamount)
fill_halo_regions!(seamount_field)

minimum_fractional_Δz = 0.2
immersed_boundaries = [
                       PartialCellBottom(seamount_field.data;
                                         minimum_fractional_Δz),
                       GridFittedBottom(seamount_field.data)
                      ]

b = []
v = []

function progress(sim)
    vmax = maximum(abs, sim.model.velocities.v)
    @info @sprintf("Iter: %d, time: %.2e, max|v|: %.2e",
                   iteration(sim), time(sim), vmax)

    return nothing
end

for ib in immersed_boundaries
    grid = ImmersedBoundaryGrid(underlying_grid, ib)

    @show grid

    model = HydrostaticFreeSurfaceModel(; grid,
                                        tracer_advection,
                                        momentum_advection,
                                        coriolis = FPlane(f=0.1),
                                        tracers = :b,
                                        buoyancy = BuoyancyTracer())

    N² = 1
    bᵢ(x, y, z) = N² * z
    set!(model, b = bᵢ)

    simulation = Simulation(model; Δt=1e-3, stop_iteration=1000)
    simulation.callbacks[:p] = Callback(progress, IterationInterval(100))


    run!(simulation)

    push!(b, Array(interior(model.tracers.b, 1, :, :)))
    push!(v, Array(interior(model.velocities.v, 1, :, :)))
end

b_partial = b[1]
b_full    = b[2]
Δb = b_full .- b_partial

v_partial = v[1]
v_full    = v[2]
Δv = v_full .- v_partial

fig = Figure(resolution=(1200, 1800))

partial_cell_title = @sprintf("PartialCellBottom with ϵ = %.1f", minimum_fractional_Δz)
ax_bp = Axis(fig[1, 2], title=partial_cell_title)
ax_bf = Axis(fig[2, 2], title="GridFittedBottom")
ax_bd = Axis(fig[3, 2], title="Difference (GridFitted - PartialCell)")

# ax_vp = Axis(fig[1, 3])
# ax_vf = Axis(fig[2, 3])
# ax_vd = Axis(fig[3, 3])

color = (:black, 0.5)
linewidth = 3
levels = 15

hmbp = heatmap!(ax_bp, b_partial)
contour!(ax_bp, b_partial; levels, color, linewidth)
Colorbar(fig[1, 1], hmbp, label="Buoyancy", flipaxis=false)

hmbf = heatmap!(ax_bf, b_full)
contour!(ax_bf, b_full; levels, color, linewidth)
Colorbar(fig[2, 1], hmbf, label="Buoyancy", flipaxis=false)

hmbd = heatmap!(ax_bd, Δb)
Colorbar(fig[3, 1], hmbd, label="Buoyancy", flipaxis=false)

#=
hmvp = heatmap!(ax_vp, v_partial)
#contour!(ax_vp, v_partial, levels=15)
#Colorbar(fig[1, 1], hmvp)

hmvf = heatmap!(ax_vf, v_full)
#contour!(ax_vf, v_full, levels=15)
#Colorbar(fig[2, 1], hmvf)

hmvd = heatmap!(ax_vd, Δv)
#Colorbar(fig[3, 1], hmvd)
=#

display(fig)

@navidcy
Copy link
Collaborator

navidcy commented Nov 4, 2022 •
edited
Loading

A more minimal example:

using Oceananigans
using Oceananigans.ImmersedBoundaries: ImmersedBoundaryGrid, PartialCellBottom

arch = CPU()

underlying_grid = RectilinearGrid(arch,
                                  size=(128, 64), halo=(3, 3), 
                                  y = (-1, 1),
                                  z = (-1, 0),
                                  topology=(Flat, Periodic, Bounded))

@inline seamount(x, y) = - 1 + 0.5 * exp(- y^2 / 0.25^2)

minimum_fractional_Δz = 0.2

grid = ImmersedBoundaryGrid(underlying_grid, PartialCellBottom(seamount; minimum_fractional_Δz))

tracer_advection = CenteredSecondOrder()
momentum_advection = CenteredSecondOrder()

model = HydrostaticFreeSurfaceModel(; grid,
                                      tracer_advection,
                                      momentum_advection,
                                      coriolis = FPlane(f=0.1),
                                      tracers = :b,
                                      buoyancy = BuoyancyTracer())
ERROR: TaskFailedException

    nested task error: TaskFailedException

        nested task error: BoundsError: attempt to access 1×134 OffsetArray(::Matrix{Float64}, 1:1, -2:131) with eltype Float64 with indices 1:1×-2:131 at index [0, 1]
        Stacktrace:
          [1] throw_boundserror(A::OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, I::Tuple{Int64, Int64})
            @ Base ./abstractarray.jl:703
          [2] overdub
            @ ~/.julia/packages/KernelAbstractions/3ZHln/src/compiler.jl:51 [inlined]
          [3] overdub
            @ ./abstractarray.jl:668 [inlined]
          [4] getindex(::OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, ::Int64, ::Int64)
            @ ~/.julia/packages/OffsetArrays/WvkHl/src/OffsetArrays.jl:420 [inlined]
          [5] overdub
            @ ~/.julia/packages/OffsetArrays/WvkHl/src/OffsetArrays.jl:420 [inlined]
          [6] overdub
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/partial_cell_immersed_boundaries.jl:49 [inlined]
          [7] immersed_cell(::Int64, ::Int64, ::Int64, ::RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, ::PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64})
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/grid_fitted_immersed_boundaries.jl:184 [inlined]
          [8] overdub
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/grid_fitted_immersed_boundaries.jl:184 [inlined]
          [9] immersed_cell(::Int64, ::Int64, ::Int64, ::ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU})
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/ImmersedBoundaries.jl:166 [inlined]
         [10] overdub
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/ImmersedBoundaries.jl:166 [inlined]
         [11] inactive_cell(::Int64, ::Int64, ::Int64, ::ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU})
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/ImmersedBoundaries.jl:205 [inlined]
         [12] overdub
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/ImmersedBoundaries.jl:205 [inlined]
         [13] peripheral_node(::Int64, ::Int64, ::Int64, ::ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, ::Face, ::Center, ::Center)
            @ ~/Research/OC2.jl/src/Grids/inactive_node.jl:115 [inlined]
         [14] overdub
            @ ~/Research/OC2.jl/src/Grids/inactive_node.jl:115 [inlined]
         [15] scalar_mask(::Int64, ::Int64, ::Int64, ::ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, ::PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, ::Face, ::Center, ::Center, ::Float64, ::Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}})
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/mask_immersed_field.jl:51 [inlined]
         [16] overdub
            @ ~/Research/OC2.jl/src/ImmersedBoundaries/mask_immersed_field.jl:51 [inlined]
         [17] overdub
            @ ~/.julia/packages/KernelAbstractions/3ZHln/src/macros.jl:266 [inlined]
         [18] __thread_run(tid::Int64, len::Int64, rem::Int64, obj::KernelAbstractions.Kernel{KernelAbstractions.CPU, KernelAbstractions.NDIteration.StaticSize{(1, 128)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 64)}, typeof(Oceananigans.ImmersedBoundaries.cpu__mask_immersed_field!)}, ndrange::Nothing, iterspace::KernelAbstractions.NDIteration.NDRange{3, KernelAbstractions.NDIteration.StaticSize{(1, 1, 64)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 1)}, Nothing, Nothing}, args::Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Tuple{Face, Center, Center}, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Float64}, dynamic::KernelAbstractions.NDIteration.NoDynamicCheck)
            @ KernelAbstractions ~/.julia/packages/KernelAbstractions/3ZHln/src/cpu.jl:157
         [19] (::KernelAbstractions.var"#45#46"{KernelAbstractions.Kernel{KernelAbstractions.CPU, KernelAbstractions.NDIteration.StaticSize{(1, 128)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 64)}, typeof(Oceananigans.ImmersedBoundaries.cpu__mask_immersed_field!)}, Nothing, KernelAbstractions.NDIteration.NDRange{3, KernelAbstractions.NDIteration.StaticSize{(1, 1, 64)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 1)}, Nothing, Nothing}, Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Tuple{Face, Center, Center}, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Float64}, KernelAbstractions.NDIteration.NoDynamicCheck, Int64})()
            @ KernelAbstractions ./threadingconstructs.jl:258

    ...and 5 more exceptions.

    Stacktrace:
     [1] sync_end(c::Channel{Any})
       @ Base ./task.jl:436
     [2] macro expansion
       @ ./task.jl:455 [inlined]
     [3] __run(obj::KernelAbstractions.Kernel{KernelAbstractions.CPU, KernelAbstractions.NDIteration.StaticSize{(1, 128)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 64)}, typeof(Oceananigans.ImmersedBoundaries.cpu__mask_immersed_field!)}, ndrange::Nothing, iterspace::KernelAbstractions.NDIteration.NDRange{3, KernelAbstractions.NDIteration.StaticSize{(1, 1, 64)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 1)}, Nothing, Nothing}, args::Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Tuple{Face, Center, Center}, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Float64}, dynamic::KernelAbstractions.NDIteration.NoDynamicCheck)
       @ KernelAbstractions ~/.julia/packages/KernelAbstractions/3ZHln/src/cpu.jl:132
     [4] (::KernelAbstractions.var"#37#38"{Tuple{KernelAbstractions.NoneEvent}, Nothing, typeof(KernelAbstractions.__run), Tuple{KernelAbstractions.Kernel{KernelAbstractions.CPU, KernelAbstractions.NDIteration.StaticSize{(1, 128)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 64)}, typeof(Oceananigans.ImmersedBoundaries.cpu__mask_immersed_field!)}, Nothing, KernelAbstractions.NDIteration.NDRange{3, KernelAbstractions.NDIteration.StaticSize{(1, 1, 64)}, KernelAbstractions.NDIteration.StaticSize{(1, 128, 1)}, Nothing, Nothing}, Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Tuple{Face, Center, Center}, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Float64}, KernelAbstractions.NDIteration.NoDynamicCheck}})()
       @ KernelAbstractions ~/.julia/packages/KernelAbstractions/3ZHln/src/cpu.jl:22
Stacktrace:
  [1] wait(cpu::KernelAbstractions.CPU, ev::KernelAbstractions.MultiEvent{Tuple{KernelAbstractions.CPUEvent, KernelAbstractions.CPUEvent, KernelAbstractions.CPUEvent}}, progress::Nothing)
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/3ZHln/src/cpu.jl:59
  [2] wait(cpu::KernelAbstractions.CPU, ev::KernelAbstractions.MultiEvent{Tuple{KernelAbstractions.CPUEvent, KernelAbstractions.CPUEvent, KernelAbstractions.CPUEvent}})
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/3ZHln/src/cpu.jl:33
  [3] masking_actions!(model::HydrostaticFreeSurfaceModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64, NamedTuple{(:u, :v, :η, :b), Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, Nothing}, Nothing, CPU, ImplicitFreeSurface{Field{Center, Center, Face, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Float64, NamedTuple{(:u, :v), Tuple{Field{Face, Center, Nothing, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Nothing, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, Oceananigans.Models.HydrostaticFreeSurfaceModels.MatrixImplicitFreeSurfaceSolver{Oceananigans.Solvers.HeptadiagonalIterativeSolver{ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Vector{Int64}, Vector{Int64}, Vector{Float64}}, Vector{Float64}, SparseArrays.SparseMatrixCSC{Float64, Int64}, IncompleteLU.ILUFactorization{Float64, Int64}, Symbol, NamedTuple{(:τ,), Tuple{Float64}}, typeof(IterativeSolvers.cg!), IterativeSolvers.CGStateVariables{Float64, Vector{Float64}}, Float64, Float64}, Vector{Float64}, Vector{Float64}}, Symbol, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Float64, NamedTuple{(:momentum, :b), Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Buoyancy{BuoyancyTracer, Oceananigans.Grids.ZDirection}, FPlane{Float64}, NamedTuple{(:u, :v, :η, :b), NTuple{4, typeof(Oceananigans.Forcings.zeroforcing)}}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:b,), Tuple{Field{Center, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:pHY′,), Tuple{Field{Center, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, Nothing, NamedTuple{(), Tuple{}}})
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Research/OC2.jl/src/Models/HydrostaticFreeSurfaceModels/update_hydrostatic_free_surface_model_state.jl:44
  [4] #apply_regionally!#35
    @ ~/Research/OC2.jl/src/Utils/multi_region_transformation.jl:96 [inlined]
  [5] apply_regionally!
    @ ~/Research/OC2.jl/src/Utils/multi_region_transformation.jl:93 [inlined]
  [6] macro expansion
    @ ~/Research/OC2.jl/src/Utils/multi_region_transformation.jl:158 [inlined]
  [7] update_state!(model::HydrostaticFreeSurfaceModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64, NamedTuple{(:u, :v, :η, :b), Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, Nothing}, Nothing, CPU, ImplicitFreeSurface{Field{Center, Center, Face, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Float64, NamedTuple{(:u, :v), Tuple{Field{Face, Center, Nothing, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Nothing, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, Oceananigans.Models.HydrostaticFreeSurfaceModels.MatrixImplicitFreeSurfaceSolver{Oceananigans.Solvers.HeptadiagonalIterativeSolver{ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Vector{Int64}, Vector{Int64}, Vector{Float64}}, Vector{Float64}, SparseArrays.SparseMatrixCSC{Float64, Int64}, IncompleteLU.ILUFactorization{Float64, Int64}, Symbol, NamedTuple{(:τ,), Tuple{Float64}}, typeof(IterativeSolvers.cg!), IterativeSolvers.CGStateVariables{Float64, Vector{Float64}}, Float64, Float64}, Vector{Float64}, Vector{Float64}}, Symbol, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Float64, NamedTuple{(:momentum, :b), Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Buoyancy{BuoyancyTracer, Oceananigans.Grids.ZDirection}, FPlane{Float64}, NamedTuple{(:u, :v, :η, :b), NTuple{4, typeof(Oceananigans.Forcings.zeroforcing)}}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:b,), Tuple{Field{Center, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:pHY′,), Tuple{Field{Center, Center, Center, Nothing, ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, Float64, FieldBoundaryConditions{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing}}}}, Nothing, NamedTuple{(), Tuple{}}}, callbacks::Vector{Any}) (repeats 2 times)
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Research/OC2.jl/src/Models/HydrostaticFreeSurfaceModels/update_hydrostatic_free_surface_model_state.jl:22
  [8] HydrostaticFreeSurfaceModel(; grid::ImmersedBoundaryGrid{Float64, Flat, Periodic, Bounded, RectilinearGrid{Float64, Flat, Periodic, Bounded, Float64, Float64, Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, PartialCellBottom{OffsetArrays.OffsetMatrix{Float64, Matrix{Float64}}, Float64}, CPU}, clock::Clock{Float64}, momentum_advection::Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, tracer_advection::Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, buoyancy::BuoyancyTracer, coriolis::FPlane{Float64}, free_surface::ImplicitFreeSurface{Nothing, Float64, Nothing, Nothing, Symbol, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, forcing::NamedTuple{(), Tuple{}}, closure::Nothing, boundary_conditions::NamedTuple{(), Tuple{}}, tracers::Symbol, particles::Nothing, velocities::Nothing, pressure::Nothing, diffusivity_fields::Nothing, auxiliary_fields::NamedTuple{(), Tuple{}})
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Research/OC2.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_model.jl:189
  [9] top-level scope
    @ REPL[10]:1
 [10] top-level scope
    @ ~/.julia/packages/CUDA/DfvRa/src/initialization.jl:52

@fadaie91
Copy link
Contributor

fadaie91 commented Nov 4, 2022 •
edited
Loading

Yes and the error only happens in the case of PartialCellBottom and it is fine with GridFittedBottom.

@navidcy
Copy link
Collaborator

navidcy commented Nov 5, 2022

cc @simone-silvestri

@navidcy navidcy added bug 🐞 Even a perfect program still has bugs immersed boundaries ⛰️ Less Ocean, more anigans question 💭 No such thing as a stupid question and removed bug 🐞 Even a perfect program still has bugs labels Nov 5, 2022
@glwagner
Copy link
Member

glwagner commented Nov 7, 2022

Do we get an error if we aren't using Flat?

@navidcy
Copy link
Collaborator

navidcy commented Nov 7, 2022

Just changing the grid to

underlying_grid = RectilinearGrid(arch,
                                  size=(1, 128, 64), halo=(3, 3, 3), 
                                  x = (-1, 1),
                                  y = (-1, 1),
                                  z = (-1, 0),
                                  topology=(Periodic, Periodic, Bounded))

the model is constructed just fine!

julia> model = HydrostaticFreeSurfaceModel(; grid,
                                             tracer_advection,
                                             momentum_advection,
                                             coriolis = FPlane(f=0.1),
                                             tracers = :b,
                                             buoyancy = BuoyancyTracer())
HydrostaticFreeSurfaceModel{CPU, ImmersedBoundaryGrid}(time = 0 seconds, iteration = 0)
├── grid: 1×128×64 ImmersedBoundaryGrid{Float64, Periodic, Periodic, Bounded} on CPU with 3×3×3 halo
├── timestepper: QuasiAdamsBashforth2TimeStepper
├── tracers: b
├── closure: Nothing
├── buoyancy: BuoyancyTracer with -ĝ = ZDirection
├── free surface: ImplicitFreeSurface with gravitational acceleration 9.80665 m s⁻²
│   └── solver: MatrixImplicitFreeSurfaceSolver
└── coriolis: FPlane{Float64}

Does this gives us a clue @glwagner?

@francispoulin
Copy link
Collaborator Author

I guess we can avoid using Flat for the moment and that looks to fix the problem. @fadaie91 , did you want to try this with the example and your original problem?

@glwagner
Copy link
Member

glwagner commented Nov 7, 2022

I guess there's an issue with Flat, but not sure what. Could be related to #2301.

@navidcy navidcy mentioned this issue Nov 7, 2022
Closed
@navidcy navidcy changed the title problem with validation example: resting_stratified_bumpy_ocean.jl Problem using immersed boundary with PartialCellBottom and Flat horizontal dimensions Nov 12, 2022
@simone-silvestri
Copy link
Collaborator

Fixed by #3682

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
immersed boundaries ⛰️ Less Ocean, more anigans question 💭 No such thing as a stupid question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@navidcy @francispoulin @glwagner @simone-silvestri @fadaie91