Assistance Needed with Initial Setup on Windows 10 Pro

[Niyasdeen]

Dear PeleC Community,

I am new to using PeleC and am currently trying to complete the initial setup on a Windows 10 Pro system. I have installed all necessary components using Cygwin and GCC, and I have also set the required environment variables. However, I am encountering an error when compiling:

sethostname: Use the Network Control Panel Applet to set hostname. hostname -s is not supported.

Any guidance on resolving this issue would be greatly appreciated.

Thanks

[jrood-nrel]

We have not tried building PeleC with Cygwin on Windows so I'm not sure what problems you might encounter. However, it looks like it's failing on the hostname command here: https://github.com/AMReX-Codes/amrex/blob/8eff86d32d17bfbfc6b6bf1091a45bdeb6dd5c86/Tools/GNUMake/Make.machines#L16

You can try removing the -s or commenting that line out. You should also check if the hostname command works in your terminal.

[marchdf]

Another option is to try the cmake build route and bypass AMReX's automatic build system stuff.

[Niyasdeen]

[@jrood-nrel]
PeleC.txt
Thank you for your message. I have commented out both lines in the link:
#host_name := $(shell hostname -f) #host_name_short := $(shell hostname -s)

While this resolved the previous error, I am now encountering a new issue. Kindly review the attached text file containing the terminal output. The error message states:

/usr/bin/sh: -c: line 1: syntax error near unexpected token('"
`

[Niyasdeen]

@marchdf CMake is somewhat more complex than GNU Make; therefore, I will initially give it a try using GNU Make.

[marchdf]

I am pretty sure you are going to keep hitting these issues with gnumake on windows. Basically you are relying on AMReX's build system to properly work on windows. I doubt they support it. Your best bet is cmake. You can see how they test amrex on windows by looking at their CI: https://github.com/AMReX-Codes/amrex/blob/development/.github/workflows/windows.yml. Looks pretty straightforward.

[baperry2]

You may also have better luck installing through Windows Subsystem for Linux rather than Cygwin, as WSL lets you work in a true linux environment. Although I haven't tried that route so I'm not sure if there would be any pitfalls.

[baperry2]

I tried it and can confirm that I was able to compile and run PeleC through WSL (version 2.1.5) on Windows 11 after installing relevant compilers, etc.

[Niyasdeen]

@baperry2 I installed WSL and successfully executed the commands make TPLrealclean && make realclean && make TPL.

However, I encountered several errors when running make -j (refer to attachment 1).

Post installation of WSL, I appended these lines to the bashrc file (refer to attachment 2). Is there any additional configuration required before using make -j?

Are there any specific configurations needed for this library?

Thanks

[baperry2]

Based on those errors it looks like you may somehow have an incompatible sundials version (7.0 instead of 6.6). Are you using the Sundials obtained through git submodules (PeleC/Submodules/PelePhysics/Submodules/sundials) or an external version?

Note, it should not be necessary to export those environment variables if you are using the default dependencies obtained as git submodules.

[Niyasdeen]

@baperry2 Thank you for your advice. I was using Sundials version 7.0 instead of 6.6, so I changed it and recompiled everything, and now it shows SUCCESS.

I tried the Shu-Osher problem on my PC, and it works well. Now, I want to move towards openMP simulation. Could you please guide me on where to find more details about OpenMP simulation with PeleC?

I am planning to try OpenMP-based simulation on my PC.

[baperry2]

To use OpenMP, compile with USE_OMP = TRUE in your GNUmakefile.

[drummerdoc]

Adding to Bruce's reply above, you'll need to set the OMP_NUM_THREADS environment variable before running the OMP-enabled executable, or it will default to using just one thread. Also, note that AMReX has a strategy for using OMP threads that is based on logically splitting grid boxes into virtual patches to work on simultaneously across threads, so be sure that the cases you run have large enough boxes that this splitting results in enough work for the cores to do. Read up on the documentation to be sure you are getting the most out of this option.

[Niyasdeen]

@drummerdoc Thanks. I am reviewing the documentation at https://amrex-combustion.github.io/PeleC/index.html, , but I cannot find any information regarding the OMP_NUM_THREADS setting. Could you please guide me to the relevant section in the documentation?

Another question pertains to the Shu-Osher problem. I successfully reproduced the tutorial case, but when I try to increase the cell count to amr.n_cell = 512 4 4 ,

I encounter the following error:

Could you please explain what this error means?

I am also attaching the Backtrace.0 file here:
Backtrace.txt

Thank you

[drummerdoc]

With "ncell = 512 4 4" your problo and probhi needs to define a physical domain to be equal in size in the y and z direction and 128 times larger than those in x, so that on level zero the code will calculate exactly the same values for dx, dy and dz (the grid spacing in each coordinate direction). That is not a requirement of all AMReX applications, but rather a limitation of the some of the operations specific to Pele, such as the embedded boundary redistribution and in the low Mach code, for the linear solvers.

As for OMP_NUM_THREADS, this environment variable thing is a standard approach in Unix environments to control how many threads per core are launched in the OpenMP environment when an OMP-enabled executable is run. It may be that this part of the Pele documentation needs a bit of beefing up though. I'll have a look at what we've written and make sure it complete enough. However, for now when you run any AMReX-based application compiled with the USE_OMP=TRUE open, the first thing that is written to stdout (if amrex.v=1 (the default value)) is the number of OMP threads per core that will be used for that run - use that to verify that the code was properly compiled for OMP and that you are using the number of threads per core that you intend. I believe that most systems that support OMP builds will default to 1 thread per core if that variable is not set explicitly, and most will support any integer number you set it to. However, each real CPU has a fixed number of hardware threads and beyond that the OS will supply virtual threads to fulfill the request. It depends heavily on the specific CPU architecture and system configuration whether it is a net benefit to ask for more threads per CPU than the number of hardware threads available. And again, in most Unix systems, the method to select this number is via the environment variable, OMP_NUM_THREADS. You can set this on the command line as

$> OMP_NUM_THREADS=4 ./amr3d.OMP.ex inputs

for example, or with a separate "export OMP_NuM_THREADS=4" prior to calling the executable. This is usually done in one's batch script when using a batch system.

Finally, as mentioned, we should add some documentation (or reference those parts of the AMReX documentation) that discuss the impact of blocking factor and max_grid_size on how work and memory is distributed in parallel with MPI, OMP and over GPUs. For now, please look closely at the OMP-relevant parts of the AMReX documentation (specifically associated with the "logical tiling" operations), and feel free to ask me for clarifications.

[Niyasdeen]

@drummerdoc Thanks. I resolved the grid issue.

For running OMP, after setting OMP=TRUE, I added the following lines in the command line:
export OMP_NUM_THREADS=4 ./PeleC3d.gnu.ex example.inp
However, the number of threads utilized is still only one. See the attached Task Manager details.

Is there another way to enable the use of multiple threads?

Thanks

[drummerdoc]

The executable you ran does not have OMP as part of its name and is therefore not the one you built with USE_OMP=TRUE. Do you have more than 1 executable in that folder and did you accidentally call the wrong one? :)

[Niyasdeen]

@drummerdoc I followed your advice, and OMP works perfectly.

I am planning to study the AMR dependency on the Shu-Osher problem. While I understand that changing the amr.max_level will achieve this, I need more clarity on the blocking factor.

Could you please help me with the following questions?

What is amr.blocking_factor?
What is amr.max_grid_size?
Besides the documented numerical schemes (WENO 7th, 5th, 3rd), are there any other schemes available, such as MUSCL?

Thanks.

[marchdf]

Those OMP variables are being removed in #811.

[drummerdoc]

Check the PeleC docs for other options and associated switches for the advection stencils. Note that there is a balance of robustness and accuracy in these things, so a single test case may not capture all that is interesting. Specifically, there are issues with strong shocks, non-coordinate-aligned physics, mass fractions needing to sum to 1 and stay positive, interactions with the embedded boundary, etc. So, be careful to fully test for the processes you care most about, and be careful not to over-generalize the summary of your results. That said, I'm curious to find out more about exactly what you are testing for, and how it goes.

The grid parameters are described in the AMReX documentation. Every box generated in the AMReX infrastructure has a number of cells that is divisible evenly by the blocking_factor, which can be specified as a scalar or per dimension and also per AMR level. No box generated has a number of cells in any dimension larger than max_grid_size. Thus, for example, if blocking_factor is 16 and max_grid_size is 32, all boxes will be generated such that they have either 16 or 32 cells in each direction, while still covering all cells that have been tagged for refinement. blocking_factor is typically used to guarantee that the box arrays at each AMR can be coarsened a minimum number of times in generating a multigrid hierarchy as part of the linear solvers. max_grid_size is often used to chop generated boxes up for work and data distribution control across parallel machines. With all these (plus n_error_buf, and others) there are many tradeoffs and the user needs to experiment to find combinations that achieve the best performance for a given problem and as measured by a given metric (e.g., wall clock run time, memory usage, total resources used, etc). For example a too large block_factor will result in too much of the domain being refined and wasting resources. A max_grid_size too small will result in excessive communication. Again, I refer you to the AMReX documentation for help in understanding these very important tradeoffs that each user needs to understand in order to get the best out of their AMR setup.

[Niyasdeen]

@drummerdoc Thank you for the details. I will look into the AMReX documentation for more information.

My goal is to study the capabilities of PeleC-AMReX for detonation simulations, similar to the one titled "Detonation Propagation and Failure" on this page: https://amrex-codes.github.io/amrex/gallery.html

Have you come across any such detonation studies in AMReX before?

Since OMP is working fine, I am planning to install PeleC on our HPC cluster. Is the installation procedure the same as for WSL on Linux?

[Niyasdeen]

@drummerdoc Any comments on my previous question would be greatly appreciated. Thank you.

[drummerdoc]

Regarding detonations: have you tried to contact the folks mentioned on that AMReX gallery page?

"installing" PeleC on any unix machine involves downloading/cloning from GitHub and locally building the executable. Any case that you want to run where you need OMP, you'd build with USE_OMP=TRUE, you'd set the NUM_OMP_THREADS environment variable and make sure you use the executable with OMP in its name. Cloning and building are described on the PeleC README, and lots of information about builds is included in the online documentation at https://amrex-combustion.github.io/PeleC/

Let me know if you run into difficulties.

[Niyasdeen]

@drummerdoc, sorry for the late response.
I did not contact them but I looked into their website. https://multiphasereactiveflowgroup.wordpress.com/. They are using Amrex with their in-house solver called Hyburn.

Regarding the PeleC installation on the cluster, I am facing some issues.

1. The GNU compiler version is attached:
   

This GNU version is below the requirement for PeleC installation. There are multiple codes running on our cluster, and my colleagues suggested not to install/update anything on the cluster. Instead, they recommended trying to bring a pre-compiled solver.

So, my question is, how can I bring the pre-compiled PeleC solver to the cluster?

Thanks

[drummerdoc]

Well, you'd find another machine that has a later version of the compiler, git-clone PeleC and its dependencies there, do a "make USE_OMP=TRUE" command that hopefully would complete and result in an executable, PeleC.XXXXX.OMP.ex and you would scp that executable over to your cluster. However, you would have to make sure that all the libraries linked in the build were static rather than dynamic (search the link like for any libraries ending with .so or .dll) otherwise you'd need to bring those over as well since they won't be linked into the final executable. It all is a hack though.

Perhaps a better solution is to locally install a package manager, such as spack or homebrew, into your own user space on that cluster, and add a more modern compiler in your own space. Then you'd be independent of the system versions.

[jrood-nrel]

Binary compatibility is very difficult between systems in my opinion. Your best option would be to obtain a newer compiler to build the code. The first version of GCC 4.8 was released 11 years ago, so not updating anything on the cluster seems like a serious impediment to running modern applications. As Marc said, one option would be to install your own GCC using Spack.