Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load balance based on reaction term cost during AMR regrid #809

Open
BenWibking opened this issue Dec 8, 2024 · 0 comments
Open

load balance based on reaction term cost during AMR regrid #809

BenWibking opened this issue Dec 8, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@BenWibking
Copy link
Collaborator

BenWibking commented Dec 8, 2024

Describe the proposal
We can optionally load balance the boxes based on their cost, e.g. rhs calls for chemistry.

This can be done by overriding AmrCore::regrid() with a function that does the same regridding, except with a custom DistributionMapping created with per-box weights:
https://github.com/AMReX-Codes/amrex/blob/96db0a665ff1e6bbe638490fd02d3aafb9188f6b/Src/AmrCore/AMReX_AmrCore.cpp#L85

The tricky part is that to estimate the per-box weights, we have to keep track of a Multifab on each level that stores the cost per cell, since in general we have a completely different set of boxes after a regrid. Then we have to sum up the values of this Multifab in each box to get the vector of per-box weights.

WarpX example usage:
https://github.com/ECP-WarpX/WarpX/blob/3f3c91d09a7f9d53f09419b56a68dac29e20407f/Source/Parallelization/WarpXRegrid.cpp#L109

(However, we can't directly copy the WarpX code, since that is implemented in a way that only works for static mesh refinement.)

Describe alternatives you've considered
Don't load balance based on chemistry cost.

Additional context
This will be useful for simulations with chemistry. This could also be useful for simulations with optically-thin cooling.

@BenWibking BenWibking added the enhancement New feature or request label Dec 8, 2024
@BenWibking BenWibking self-assigned this Dec 8, 2024
@BenWibking BenWibking changed the title load balance based on chemistry cost during AMR regrid load balance based on reaction term cost during AMR regrid Dec 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant