-
Notifications
You must be signed in to change notification settings - Fork 190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement directionally-averaged, scale-invariant dynamic Smagorinsky closure #3642
base: main
Are you sure you want to change the base?
Conversation
@inline directional_average!(averaged_field, field, ::XDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=1))) | ||
@inline directional_average!(averaged_field, field, ::YDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=2))) | ||
@inline directional_average!(averaged_field, field, ::ZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=3))) | ||
@inline directional_average!(averaged_field, field, ::XYDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(1, 2)))) | ||
@inline directional_average!(averaged_field, field, ::XZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(1, 3)))) | ||
@inline directional_average!(averaged_field, field, ::YZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(2, 3)))) | ||
@inline directional_average!(averaged_field, field, ::XYZDirectionalAveraging) = averaged_field .= compute!(Field(Average(field, dims=(1, 2, 3)))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This part definitely needs to be changed. I just haven't been able to come up an appropriate kernel that works.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What kernel? If you want to compute an average then you shouldn't write your own kernel. You probably want to preallocate the result though. The code will be a lot simpler since then you just call compute!
.
…gans.jl into tc/invariant-dynamic-smag
using Oceananigans.Operators: volume | ||
using Oceananigans.Grids: XBoundedGrid, YBoundedGrid, ZBoundedGrid, XYBoundedGrid, XZBoundedGrid, YZBoundedGrid, XYZBoundedGrid | ||
|
||
abstract type AveragingProcedure end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
abstract types are always written with the prefix Abstract
, eg AbstractAveragingProcedure
src/TurbulenceClosures/turbulence_closure_implementations/scale_invariant_smagorinsky.jl
Outdated
Show resolved
Hide resolved
|
||
@inline ℱx²ᵟ(i, j, k, grid::AG{FT}, f::F, args...) where {FT, F<:Function} = FT(0.5) * f(i, j, k, grid, args...) + FT(0.25) * (f(i-1, j, k, grid, args...) + f(i+1, j, k, grid, args...)) | ||
@inline ℱy²ᵟ(i, j, k, grid::AG{FT}, f::F, args...) where {FT, F<:Function} = FT(0.5) * f(i, j, k, grid, args...) + FT(0.25) * (f(i, j-1, k, grid, args...) + f(i, j+1, k, grid, args...)) | ||
@inline ℱz²ᵟ(i, j, k, grid::AG{FT}, f::F, args...) where {FT, F<:Function} = FT(0.5) * f(i, j, k, grid, args...) + FT(0.25) * (f(i, j, k-1, grid, args...) + f(i, j, k+1, grid, args...)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are these all just double interpolations?
It might compile faster if you re-use interpolation operators. Not sure.
src/TurbulenceClosures/turbulence_closure_implementations/scale_invariant_smagorinsky.jl
Outdated
Show resolved
Hide resolved
@kernel function compute_LM_MM!(LM, MM, grid, closure, velocities) | ||
i, j, k = @index(Global, NTuple) | ||
@inbounds LM[i, j, k] = LᵢⱼMᵢⱼ_ccc(i, j, k, grid, velocities...) | ||
@inbounds MM[i, j, k] = MᵢⱼMᵢⱼ_ccc(i, j, k, grid, velocities...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just use a KernelFunctionOperation
?
struct XYDirectionalAveraging <: DirectionalAveraging end | ||
struct XZDirectionalAveraging <: DirectionalAveraging end | ||
struct YZDirectionalAveraging <: DirectionalAveraging end | ||
struct XYZDirectionalAveraging <: DirectionalAveraging end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You may not need all these types. They're just dims
so you only need
struct DimensionalAveraging{D}
dims :: D
end
Not sure if I am missing something but first I think the user can just write averaging=(1, 2) which would mean average over xy. Or if you want to be more verbose then averaging=DimensionAveraging(dims=(1, 2)) or something. Next for the average itself it seems you need LM_op = KernelFunctionOperation{Center, Center, Center}(LᵢⱼMᵢⱼ_ccc, grid, u, v, w)
LM_avg = Field(Average(LM_op, averaging.dims)) In the constructor. Then you can just call |
I considered taking that approach but I felt it made it a bit awkward to implement Lagrangian averaging in the future. That said, I do agree that my currently implementation is probably unnecessarily verbose. I think your suggestion below of
Ah, that's so simple! I'll implement that. |
src/TurbulenceClosures/turbulence_closure_implementations/scale_invariant_smagorinsky.jl
Outdated
Show resolved
Hide resolved
directional_average!(diffusivity_fields.LM_avg, diffusivity_fields.LM, closure.averaging) | ||
directional_average!(diffusivity_fields.MM_avg, diffusivity_fields.MM, closure.averaging) | ||
LM_avg = Field(Average(LM_op, dims=closure.averaging.dims)) | ||
MM_avg = Field(Average(MM_op, dims=closure.averaging.dims)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you want to preallocate these in DiffusivityFields
, and just compute!
them in compute_diffusivities!
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, thanks. I realized yesterday that preallocating them in DiffusivityFields
would require creating some new methods for DiffusivityFields
in which we can pass model
. So I was trying to see if it made sense to not modify the methods and instead do this. Surprisingly, I didn't see much of a performance decrease in this current version, although I've only tried for small models on CPUs.
I'll preallocate them in DiffusivityFields
as you suggested though!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would require creating some new methods for DiffusivityFields in which we can pass model.
Why is that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah because they depend on velocities. That is a little puzzle...
One option is to pass velocities
into DiffusivityFields
. This is a fairly large change that might affect a lot of code, but could be a positive change perhaps. Maybe, we want to save that for after this PR.
You can also preallocate the field you're averaging into, but not the kernel function operation. Then you can compute the average with something like
mean!(field, kernel_function_operation)
(instead of compute!
)
That could be a stop gap that will still let you do high performance simulation while we figure out the right pattern for this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I follow your last suggestion. What I understood you're suggesting is something like the following, but which doesn't seem to work:
julia> op = Average(model.velocities.u^2, dims=(1, 2)) # Stand-in for the Lij and Mij functions
Average of BinaryOperation at (Face, Center, Center) over dims (1, 2)
└── operand: BinaryOperation at (Face, Center, Center)
└── grid: 32×32×32 RectilinearGrid{Float64, Periodic, Periodic, Periodic} on CPU with 3×3×3 halo
julia> f = Field{Nothing, Nothing, Center}(grid)
1×1×32 Field{Nothing, Nothing, Center} reduced over dims = (1, 2) on RectilinearGrid on CPU
├── grid: 32×32×32 RectilinearGrid{Float64, Periodic, Periodic, Periodic} on CPU with 3×3×3 halo
├── boundary conditions: FieldBoundaryConditions
│ └── west: Nothing, east: Nothing, south: Nothing, north: Nothing, bottom: Periodic, top: Periodic, immersed: ZeroFlux
└── data: 1×1×38 OffsetArray(::Array{Float64, 3}, 1:1, 1:1, -2:35) with eltype Float64 with indices 1:1×1:1×-2:35
└── max=0.0, min=0.0, mean=0.0
julia> mean!(f, op)
ERROR: MethodError: no method matching mean!(::Field{…}, ::Reduction{…})
Closest candidates are:
mean!(::Union{Oceananigans.Fields.AbstractField{Nothing}, Oceananigans.Fields.AbstractField{<:Any, Nothing}, Oceananigans.Fields.AbstractField{<:Any, <:Any, Nothing}}, ::AbstractArray; kwargs...)
@ Oceananigans ~/repos/Oceananigans.jl2/src/Fields/field.jl:744
mean!(::AbstractArray, ::AbstractArray)
@ Statistics ~/bin/julia-1.10.2/share/julia/stdlib/v1.10/Statistics/src/Statistics.jl:140
Or maybe I did understand correctly, but we'd need to implement the relevant method for reductions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess you need mean!(f, op.operand)
. Like here:
Oceananigans.jl/src/Fields/scans.jl
Line 69 in e7bbb1b
s.scan!(field, s.operand) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I think it works now
…e_invariant_smagorinsky.jl Co-authored-by: Gregory L. Wagner <[email protected]>
My suggestion is an API thing. If the user writes Implementing this requires writing a function like averaging(dims::Tuple) = DimenisionAveraging(dims)
averaging(other_averaging) = other_averaging Also just brainstorming... maybe a constructor
will save some characters because arguably I think we are a little keyword-happy in the API, sometimes its nice to user positional arguments when meanings are obvious... |
LM_avg = Field(Average(LM_op, dims=closure.averaging.dims)) | ||
MM_avg = Field(Average(MM_op, dims=closure.averaging.dims)) | ||
mean!(diffusivity_fields.LM_avg, LM_op.operand) | ||
mean!(diffusivity_fields.MM_avg, MM_op.operand) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note this only works on uniform grids, but hopefully that's good enough for preliminary testing
LM_avg = directionally_averaged_field(grid, Val(closure.averaging)) | ||
MM_avg = directionally_averaged_field(grid, Val(closure.averaging)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would use something like this to make things a little simpler:
# TODO: add `velocities` as argument to `DiffusivityFields` to simplify this
MM_avg = Field(Average(νₑ, dims=closure.averaging.dims))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but it doesn't matter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I see your point. Since the important part is just to allocate memory, that already does the trick. That does kind of imply that MM_avg
will later get the average of νₑ
, which may confuse readers of the code.
How about
MM_avg = Field(Average(Field{Center, Center, Center}(grid), dims=closure.averaging.dims))
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's fine, my suggestion merely aimed to avoid allocating the additional 3D temporary array.
Hopefully this code is temporary. It's just a vehicle for you to validate this closure so it can be used as soon as possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can also use an empty / placeholder operation. But don't worry too much let's move on to validation...
…gans.jl into tc/invariant-dynamic-smag
Here's some validation. The animation below is from a triply-periodic 3D simulation with an initial reasonably-resolved initial condition without any forcings. Note that the IC looks a bit weird right now (a couple of straight lines in strain) because I'm taking a shortcut in defining a well-resolved random IC, but I can easily change it later if necessary. Left panel shows the strain rate squared for the Important here is that the value of 0.16 was analytically derived by Lilly by assuming an isotropic 3D turbulence, a Kolmogorovian energy cascade, and further assuming that the cut-off filter is in the inertial range. I think all these assumptions are valid in this simulation, so we expect the dynamically-calculated value of the Smagorinsky coefficient (the black line) to approach the value 0.16 as time goes on. 3d_turbulence_smagorinsky.mp4Although the match is not exact (the value it approaches is ~0.17), I think this is close enough. That said, I'm planning on also implementing a boundary layer validation along similar lines, which we can use to validate the model in the same fashion as Bou-Zeid et al. (2005). One thing to note is that the current implementation appears to be very slow. While the simulation with the ┌ Info: Running
└ closure = SmagorinskyLilly: C=0.16, Cb=1.0, Pr=1.0
[ Info: Initializing simulation...
[ Info: ... simulation initialization complete (2.738 seconds)
[ Info: Executing initial time step...
[ Info: ... initial time step complete (4.852 seconds).
[ Info: Simulation is stopping after running for 11.657 seconds.
[ Info: Simulation time 1.333 minutes equals or exceeds stop time 1.333 minutes.
┌ Info: Running
└ closure = ScaleInvariantSmagorinsky: averaging=Averaging over directions Colon(), Pr=1.0
[ Info: Initializing simulation...
[ Info: ... simulation initialization complete (2.195 seconds)
[ Info: Executing initial time step...
[ Info: ... initial time step complete (2.812 seconds).
[ Info: Simulation is stopping after running for 3.735 minutes.
[ Info: Simulation time 1.333 minutes equals or exceeds stop time 1.333 minutes. |
It's also worth noting that right now many calculations are done more than once in each timestep. For example for each component of Lines 241 to 257 in 25cc34e
This is done for legibility of the code, but it may be necessary to forfeit that in favor of doing fewer calculations. (Also note that I'm using a weird way to define function names here, so lmk if you guys think I should change it.) Another thing to note that it's common to update dynamic Smagorinsky coefficients once every 5 or so time-steps only, since they can be pretty expensive. afaik this is generally done for the scale-dependent versions, which have two test filters instead of the one needed in this PR, but I wouldn't be surprised if it's occasionally necessary for the scale-invariant versions as well. |
src/TurbulenceClosures/turbulence_closure_implementations/scale_invariant_smagorinsky.jl
Outdated
Show resolved
Hide resolved
…gans.jl into tc/invariant-dynamic-smag
Avoided recomputation of the strain rate at |
Closes #3637
This PR implements the scale-invariant dynamic Smagorinsky. My main reference for this was Bou-Zeid et al. (2005), just because it's nicely summarized there.
As a refresher, the coefficient in this case is calculated as
where$M_{ij}$ and $L_{ij}$ are obtained by applying the Germano identity and minimizing the error. $\langle \cdot \rangle$ is an average which usually is implemented as planar averaging. I'm implementing it here with an arbitrary
DirectionalAveraging
procedure that can average in whatever direction the user wants. Following a suggestion by @glwagner in #3637, I'm also starting to put the pieces in place for a more general averaging procedure so that eventually we can add Lagrangian averaging as an option for this model.Note that this PR is related to #3638. The rationale for having this PR in addition to #3638 is that for most LES are doubly periodic, in which case there's not much advantage in having the Lagrangian averaging that's implemented in #3638. In these cases I'd argue the model implemented here is more efficient, so having both types of models is probably desirable.
Another note is that, once this PR is merged, it should be very straightforward to implement a scale-dependent version, which works better close to boundaries.
The model seems to be working. Here's an animation of an unforced 3D turbulence simulation on a 32³ grid. Left is vorticity and right is the strain rate modulus squared:
two_dimensional_turbulence.mp4
Note that the value of the calculated Smag coefficient$c_s$ (which I'm showing at the top of the plots above) is about 0.17, which is very close to the theoretical value obtained by Lilly of 0.16.
I'm opening this as a draft PR for now because there are some things that need doing:
LM
,MM
are 3D fields; andLM_avg
andMM_avg
are 1D or 2D. What I'm doing is to first calculateLM_avg
andMM_avg
receive their averages. We should be able to calculate everything without needingLM
,MM
are 3D fields, I just couldn't figure out how yet :)CC @glwagner @simone-silvestri @xkykai @whitleyv
Feel free to add more things to the to-do list that I may have forgotten.