Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement directionally-averaged, scale-invariant dynamic Smagorinsky closure #3642

Draft
wants to merge 27 commits into
base: main
Choose a base branch
from

Conversation

tomchor
Copy link
Collaborator

@tomchor tomchor commented Jun 28, 2024

Closes #3637

This PR implements the scale-invariant dynamic Smagorinsky. My main reference for this was Bou-Zeid et al. (2005), just because it's nicely summarized there.

As a refresher, the coefficient in this case is calculated as

$$c_s^2 = \frac{\langle L_{ij} M_{ij}\rangle}{\langle M_{ij} M_{ij} \rangle},$$

where $M_{ij}$ and $L_{ij}$ are obtained by applying the Germano identity and minimizing the error. $\langle \cdot \rangle$ is an average which usually is implemented as planar averaging. I'm implementing it here with an arbitrary DirectionalAveraging procedure that can average in whatever direction the user wants. Following a suggestion by @glwagner in #3637, I'm also starting to put the pieces in place for a more general averaging procedure so that eventually we can add Lagrangian averaging as an option for this model.

Note that this PR is related to #3638. The rationale for having this PR in addition to #3638 is that for most LES are doubly periodic, in which case there's not much advantage in having the Lagrangian averaging that's implemented in #3638. In these cases I'd argue the model implemented here is more efficient, so having both types of models is probably desirable.

Another note is that, once this PR is merged, it should be very straightforward to implement a scale-dependent version, which works better close to boundaries.

The model seems to be working. Here's an animation of an unforced 3D turbulence simulation on a 32³ grid. Left is vorticity and right is the strain rate modulus squared:

two_dimensional_turbulence.mp4

Note that the value of the calculated Smag coefficient $c_s$ (which I'm showing at the top of the plots above) is about 0.17, which is very close to the theoretical value obtained by Lilly of 0.16.

I'm opening this as a draft PR for now because there are some things that need doing:

  • Generalize the filter for stretched grids. For now it assumes a regular grid for simplicity, but it's trivial to generalize.
  • Optimize the calculation of the coefficient. At the moment I'm creating four extra fields in order to calculate the Smag coefficient: LM, MM are 3D fields; and LM_avg and MM_avg are 1D or 2D. What I'm doing is to first calculate $L_{ij} M_{ij}$ and $M_{ij} M_{ij}$ pointwise, and then LM_avg and MM_avg receive their averages. We should be able to calculate everything without needing LM, MM are 3D fields, I just couldn't figure out how yet :)
  • Write docs
  • Write tests
  • Validate that model is working as intended

CC @glwagner @simone-silvestri @xkykai @whitleyv

Feel free to add more things to the to-do list that I may have forgotten.

Comment on lines 81 to 87
@inline directional_average!(averaged_field, field, ::XDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=1)))
@inline directional_average!(averaged_field, field, ::YDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=2)))
@inline directional_average!(averaged_field, field, ::ZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=3)))
@inline directional_average!(averaged_field, field, ::XYDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(1, 2))))
@inline directional_average!(averaged_field, field, ::XZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(1, 3))))
@inline directional_average!(averaged_field, field, ::YZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(2, 3))))
@inline directional_average!(averaged_field, field, ::XYZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(1, 2, 3))))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part definitely needs to be changed. I just haven't been able to come up an appropriate kernel that works.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What kernel? If you want to compute an average then you shouldn't write your own kernel. You probably want to preallocate the result though. The code will be a lot simpler since then you just call compute!.

using Oceananigans.Operators: volume
using Oceananigans.Grids: XBoundedGrid, YBoundedGrid, ZBoundedGrid, XYBoundedGrid, XZBoundedGrid, YZBoundedGrid, XYZBoundedGrid

abstract type AveragingProcedure end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

abstract types are always written with the prefix Abstract, eg AbstractAveragingProcedure


@inline ℱx²ᵟ(i, j, k, grid::AG{FT}, f::F, args...) where {FT, F<:Function} = FT(0.5) * f(i, j, k, grid, args...) + FT(0.25) * (f(i-1, j, k, grid, args...) + f(i+1, j, k, grid, args...))
@inline ℱy²ᵟ(i, j, k, grid::AG{FT}, f::F, args...) where {FT, F<:Function} = FT(0.5) * f(i, j, k, grid, args...) + FT(0.25) * (f(i, j-1, k, grid, args...) + f(i, j+1, k, grid, args...))
@inline ℱz²ᵟ(i, j, k, grid::AG{FT}, f::F, args...) where {FT, F<:Function} = FT(0.5) * f(i, j, k, grid, args...) + FT(0.25) * (f(i, j, k-1, grid, args...) + f(i, j, k+1, grid, args...))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these all just double interpolations?

It might compile faster if you re-use interpolation operators. Not sure.

@kernel function compute_LM_MM!(LM, MM, grid, closure, velocities)
i, j, k = @index(Global, NTuple)
@inbounds LM[i, j, k] = LᵢⱼMᵢⱼ_ccc(i, j, k, grid, velocities...)
@inbounds MM[i, j, k] = MᵢⱼMᵢⱼ_ccc(i, j, k, grid, velocities...)
Copy link
Member

@glwagner glwagner Jun 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just use a KernelFunctionOperation?

struct XYDirectionalAveraging <: DirectionalAveraging end
struct XZDirectionalAveraging <: DirectionalAveraging end
struct YZDirectionalAveraging <: DirectionalAveraging end
struct XYZDirectionalAveraging <: DirectionalAveraging end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may not need all these types. They're just dims so you only need

struct DimensionalAveraging{D}
    dims :: D
end

@glwagner
Copy link
Member

Not sure if I am missing something but first I think the user can just write

averaging=(1, 2)

which would mean average over xy. Or if you want to be more verbose then

averaging=DimensionAveraging(dims=(1, 2))

or something.

Next for the average itself it seems you need

LM_op = KernelFunctionOperation{Center, Center, Center}(LᵢⱼMᵢⱼ_ccc, grid, u, v, w)
LM_avg = Field(Average(LM_op, averaging.dims))

In the constructor. Then you can just call compute!(LM_avg) without needing lots of additional dispatch.

@tomchor
Copy link
Collaborator Author

tomchor commented Jun 28, 2024

Not sure if I am missing something but first I think the user can just write

averaging=(1, 2)

I considered taking that approach but I felt it made it a bit awkward to implement Lagrangian averaging in the future. That said, I do agree that my currently implementation is probably unnecessarily verbose. I think your suggestion below of DimensionAveraging(dims=(1, 2)) is a good compromise.

which would mean average over xy. Or if you want to be more verbose then

averaging=DimensionAveraging(dims=(1, 2))

or something.

Next for the average itself it seems you need

LM_op = KernelFunctionOperation{Center, Center, Center}(LᵢⱼMᵢⱼ_ccc, grid, u, v, w)
LM_avg = Field(Average(LM_op, averaging.dims))

In the constructor. Then you can just call compute!(LM_avg) without needing lots of additional dispatch.

Ah, that's so simple! I'll implement that.

directional_average!(diffusivity_fields.LM_avg, diffusivity_fields.LM, closure.averaging)
directional_average!(diffusivity_fields.MM_avg, diffusivity_fields.MM, closure.averaging)
LM_avg = Field(Average(LM_op, dims=closure.averaging.dims))
MM_avg = Field(Average(MM_op, dims=closure.averaging.dims))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you want to preallocate these in DiffusivityFields, and just compute! them in compute_diffusivities!.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thanks. I realized yesterday that preallocating them in DiffusivityFields would require creating some new methods for DiffusivityFields in which we can pass model. So I was trying to see if it made sense to not modify the methods and instead do this. Surprisingly, I didn't see much of a performance decrease in this current version, although I've only tried for small models on CPUs.

I'll preallocate them in DiffusivityFields as you suggested though!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would require creating some new methods for DiffusivityFields in which we can pass model.

Why is that?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah because they depend on velocities. That is a little puzzle...

One option is to pass velocities into DiffusivityFields. This is a fairly large change that might affect a lot of code, but could be a positive change perhaps. Maybe, we want to save that for after this PR.

You can also preallocate the field you're averaging into, but not the kernel function operation. Then you can compute the average with something like

mean!(field, kernel_function_operation)

(instead of compute!)

That could be a stop gap that will still let you do high performance simulation while we figure out the right pattern for this

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow your last suggestion. What I understood you're suggesting is something like the following, but which doesn't seem to work:

julia> op = Average(model.velocities.u^2, dims=(1, 2)) # Stand-in for the Lij and Mij functions
Average of BinaryOperation at (Face, Center, Center) over dims (1, 2)
└── operand: BinaryOperation at (Face, Center, Center)
    └── grid: 32×32×32 RectilinearGrid{Float64, Periodic, Periodic, Periodic} on CPU with 3×3×3 halo

julia> f = Field{Nothing, Nothing, Center}(grid)
1×1×32 Field{Nothing, Nothing, Center} reduced over dims = (1, 2) on RectilinearGrid on CPU
├── grid: 32×32×32 RectilinearGrid{Float64, Periodic, Periodic, Periodic} on CPU with 3×3×3 halo
├── boundary conditions: FieldBoundaryConditions
│   └── west: Nothing, east: Nothing, south: Nothing, north: Nothing, bottom: Periodic, top: Periodic, immersed: ZeroFlux
└── data: 1×1×38 OffsetArray(::Array{Float64, 3}, 1:1, 1:1, -2:35) with eltype Float64 with indices 1:1×1:1×-2:35
    └── max=0.0, min=0.0, mean=0.0

julia> mean!(f, op)
ERROR: MethodError: no method matching mean!(::Field{…}, ::Reduction{…})

Closest candidates are:
  mean!(::Union{Oceananigans.Fields.AbstractField{Nothing}, Oceananigans.Fields.AbstractField{<:Any, Nothing}, Oceananigans.Fields.AbstractField{<:Any, <:Any, Nothing}}, ::AbstractArray; kwargs...)
   @ Oceananigans ~/repos/Oceananigans.jl2/src/Fields/field.jl:744
  mean!(::AbstractArray, ::AbstractArray)
   @ Statistics ~/bin/julia-1.10.2/share/julia/stdlib/v1.10/Statistics/src/Statistics.jl:140

Or maybe I did understand correctly, but we'd need to implement the relevant method for reductions?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you need mean!(f, op.operand). Like here:

s.scan!(field, s.operand)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I think it works now

tomchor and others added 2 commits July 1, 2024 11:07
@glwagner
Copy link
Member

glwagner commented Jul 1, 2024

Not sure if I am missing something but first I think the user can just write

averaging=(1, 2)

I considered taking that approach but I felt it made it a bit awkward to implement Lagrangian averaging in the future. That said, I do agree that my currently implementation is probably unnecessarily verbose. I think your suggestion below of DimensionAveraging(dims=(1, 2)) is a good compromise.

My suggestion is an API thing. If the user writes averaging = (1, 2) then you can interpret this as DimensionAveraging((1, 2)). Saves some characters.

Implementing this requires writing a function like

averaging(dims::Tuple) = DimenisionAveraging(dims)
averaging(other_averaging) = other_averaging

Also just brainstorming... maybe a constructor

DimensionAveraging(dims...) = DimensionAveraging(tuple(dims))
DimensionAveraging(dims::Colon) = DimensionAveraging(:)

will save some characters because arguably DimensionAveraging(1, 2) is easier to read than DimensionAveraging((1, 2)).

I think we are a little keyword-happy in the API, sometimes its nice to user positional arguments when meanings are obvious...

LM_avg = Field(Average(LM_op, dims=closure.averaging.dims))
MM_avg = Field(Average(MM_op, dims=closure.averaging.dims))
mean!(diffusivity_fields.LM_avg, LM_op.operand)
mean!(diffusivity_fields.MM_avg, MM_op.operand)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this only works on uniform grids, but hopefully that's good enough for preliminary testing

Comment on lines +312 to +313
LM_avg = directionally_averaged_field(grid, Val(closure.averaging))
MM_avg = directionally_averaged_field(grid, Val(closure.averaging))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use something like this to make things a little simpler:

# TODO: add `velocities` as argument to `DiffusivityFields` to simplify this
MM_avg = Field(Average(νₑ, dims=closure.averaging.dims))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but it doesn't matter

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I see your point. Since the important part is just to allocate memory, that already does the trick. That does kind of imply that MM_avg will later get the average of νₑ, which may confuse readers of the code.

How about

MM_avg = Field(Average(Field{Center, Center, Center}(grid), dims=closure.averaging.dims))

?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, my suggestion merely aimed to avoid allocating the additional 3D temporary array.

Hopefully this code is temporary. It's just a vehicle for you to validate this closure so it can be used as soon as possible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also use an empty / placeholder operation. But don't worry too much let's move on to validation...

@tomchor
Copy link
Collaborator Author

tomchor commented Jul 7, 2024

Here's some validation. The animation below is from a triply-periodic 3D simulation with an initial reasonably-resolved initial condition without any forcings. Note that the IC looks a bit weird right now (a couple of straight lines in strain) because I'm taking a shortcut in defining a well-resolved random IC, but I can easily change it later if necessary.

Left panel shows the strain rate squared for the SmagorinskyLilly closure, while the one on the right shows results for the ScaleInvarianteSmagorinsky closure, implemented in this PR. The bottom panel shows the evolution of the dynamically-calculated Smagorinsky coefficient for the latter, in comparison with the constant value of 0.16 imposed on the former.

Important here is that the value of 0.16 was analytically derived by Lilly by assuming an isotropic 3D turbulence, a Kolmogorovian energy cascade, and further assuming that the cut-off filter is in the inertial range. I think all these assumptions are valid in this simulation, so we expect the dynamically-calculated value of the Smagorinsky coefficient (the black line) to approach the value 0.16 as time goes on.

3d_turbulence_smagorinsky.mp4

Although the match is not exact (the value it approaches is ~0.17), I think this is close enough. That said, I'm planning on also implementing a boundary layer validation along similar lines, which we can use to validate the model in the same fashion as Bou-Zeid et al. (2005).

One thing to note is that the current implementation appears to be very slow. While the simulation with the SmagorinskyLilly closure runs on my laptop in 10 seconds, it takes 4 minutes for the simulation with the ScaleInvariantSmagorinsky. I know the dynamic model will be slower given the extra computations, but such a difference seems large to me, so I'm hoping something can be changed here to improve performance:

┌ Info: Running
└   closure = SmagorinskyLilly: C=0.16, Cb=1.0, Pr=1.0
[ Info: Initializing simulation...
[ Info:     ... simulation initialization complete (2.738 seconds)
[ Info: Executing initial time step...
[ Info:     ... initial time step complete (4.852 seconds).
[ Info: Simulation is stopping after running for 11.657 seconds.
[ Info: Simulation time 1.333 minutes equals or exceeds stop time 1.333 minutes.
┌ Info: Running
└   closure = ScaleInvariantSmagorinsky: averaging=Averaging over directions Colon(), Pr=1.0
[ Info: Initializing simulation...
[ Info:     ... simulation initialization complete (2.195 seconds)
[ Info: Executing initial time step...
[ Info:     ... initial time step complete (2.812 seconds).
[ Info: Simulation is stopping after running for 3.735 minutes.
[ Info: Simulation time 1.333 minutes equals or exceeds stop time 1.333 minutes.

@tomchor
Copy link
Collaborator Author

tomchor commented Jul 8, 2024

It's also worth noting that right now many calculations are done more than once in each timestep. For example for each component of $M_{ij}$ I'm calculating the whole strain rate tensor modulus in addition to the strain rate tensor component needed:

@inline var"|S̄|S̄₁₁ᶜᶜᶜ"(i, j, k, grid, u, v, w) = (Σ̄ᵢⱼΣ̄ᵢⱼᶜᶜᶜ(i, j, k, grid, u, v, w)) * Σ̄₁₁(i, j, k, grid, u, v, w) # ccc
@inline var"|S̄|S̄₂₂ᶜᶜᶜ"(i, j, k, grid, u, v, w) = (Σ̄ᵢⱼΣ̄ᵢⱼᶜᶜᶜ(i, j, k, grid, u, v, w)) * Σ̄₂₂(i, j, k, grid, u, v, w) # ccc
@inline var"|S̄|S̄₃₃ᶜᶜᶜ"(i, j, k, grid, u, v, w) = (Σ̄ᵢⱼΣ̄ᵢⱼᶜᶜᶜ(i, j, k, grid, u, v, w)) * Σ̄₃₃(i, j, k, grid, u, v, w) # ccc
@inline var"|S̄|S̄₁₂ᶜᶜᶜ"(i, j, k, grid, u, v, w) = (Σ̄ᵢⱼΣ̄ᵢⱼᶠᶠᶜ(i, j, k, grid, u, v, w)) * Σ̄₁₂(i, j, k, grid, u, v, w) # ffc
@inline var"|S̄|S̄₁₃ᶜᶜᶜ"(i, j, k, grid, u, v, w) = (Σ̄ᵢⱼΣ̄ᵢⱼᶠᶜᶠ(i, j, k, grid, u, v, w)) * Σ̄₁₃(i, j, k, grid, u, v, w) # fcf
@inline var"|S̄|S̄₂₃ᶜᶜᶜ"(i, j, k, grid, u, v, w) = (Σ̄ᵢⱼΣ̄ᵢⱼᶜᶠᶠ(i, j, k, grid, u, v, w)) * Σ̄₂₃(i, j, k, grid, u, v, w) # cff
@inline Δᶠ(i, j, k, grid) = volume(i, j, k, grid, Center(), Center(), Center())
@inline M₁₁ᶜᶜᶜ(i, j, k, grid, u, v, w, α, β) = 2*Δᶠ(i, j, k, grid)^2 * (var"⟨|S|S₁₁⟩ᶜᶜᶜ"(i, j, k, grid, u, v, w) - α^2*β * var"|S̄|S̄₁₁ᶜᶜᶜ"(i, j, k, grid, u, v, w))
@inline M₂₂ᶜᶜᶜ(i, j, k, grid, u, v, w, α, β) = 2*Δᶠ(i, j, k, grid)^2 * (var"⟨|S|S₂₂⟩ᶜᶜᶜ"(i, j, k, grid, u, v, w) - α^2*β * var"|S̄|S̄₂₂ᶜᶜᶜ"(i, j, k, grid, u, v, w))
@inline M₃₃ᶜᶜᶜ(i, j, k, grid, u, v, w, α, β) = 2*Δᶠ(i, j, k, grid)^2 * (var"⟨|S|S₃₃⟩ᶜᶜᶜ"(i, j, k, grid, u, v, w) - α^2*β * var"|S̄|S̄₃₃ᶜᶜᶜ"(i, j, k, grid, u, v, w))
@inline M₁₂ᶜᶜᶜ(i, j, k, grid, u, v, w, α, β) = 2*Δᶠ(i, j, k, grid)^2 * (var"⟨|S|S₁₂⟩ᶜᶜᶜ"(i, j, k, grid, u, v, w) - α^2*β * var"|S̄|S̄₁₂ᶜᶜᶜ"(i, j, k, grid, u, v, w))
@inline M₁₃ᶜᶜᶜ(i, j, k, grid, u, v, w, α, β) = 2*Δᶠ(i, j, k, grid)^2 * (var"⟨|S|S₁₃⟩ᶜᶜᶜ"(i, j, k, grid, u, v, w) - α^2*β * var"|S̄|S̄₁₃ᶜᶜᶜ"(i, j, k, grid, u, v, w))
@inline M₂₃ᶜᶜᶜ(i, j, k, grid, u, v, w, α, β) = 2*Δᶠ(i, j, k, grid)^2 * (var"⟨|S|S₂₃⟩ᶜᶜᶜ"(i, j, k, grid, u, v, w) - α^2*β * var"|S̄|S̄₂₃ᶜᶜᶜ"(i, j, k, grid, u, v, w))

This is done for legibility of the code, but it may be necessary to forfeit that in favor of doing fewer calculations. (Also note that I'm using a weird way to define function names here, so lmk if you guys think I should change it.)

Another thing to note that it's common to update dynamic Smagorinsky coefficients once every 5 or so time-steps only, since they can be pretty expensive. afaik this is generally done for the scale-dependent versions, which have two test filters instead of the one needed in this PR, but I wouldn't be surprised if it's occasionally necessary for the scale-invariant versions as well.

@tomchor
Copy link
Collaborator Author

tomchor commented Jul 21, 2024

One thing to note is that the current implementation appears to be very slow. While the simulation with the SmagorinskyLilly closure runs on my laptop in 10 seconds, it takes 4 minutes for the simulation with the ScaleInvariantSmagorinsky. I know the dynamic model will be slower given the extra computations, but such a difference seems large to me, so I'm hoping something can be changed here to improve performance:

Avoided recomputation of the strain rate at ccc and sped things up a bit more. Now it runs in 2.9 minutes. A lot more to go...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

More / more flexible Smagorinsky models
2 participants