Initial condition multi-thread issue

[ChenxiSSS]

Hi! I am just recently started to use 21cmFAST. I am noticing that only one thread was used for the majority of the time when generating the initial condition. Is this behavior expected, or did I set anything wrong?

Treads info:

• Set 200 Threads to my HII_DIM = 2048 BOX_LEN = 1600 test run
• Set 20 Threads to my HII_DIM = 1024 BOX_LEN = 1600 test run
• Set 20 Threads to my HII_DIM = 512 BOX_LEN = 1600 test run

They all behave in a similar way. Using multiple threads for some short period of time and using a single thread most of the time.

[qyx268]

The scalability doesn't behave well because, as you've spotted, only a fraction of the calculation is done using multiple threads. If you check the JOSS paper (https://arxiv.org/pdf/2010.15121.pdf), you will see the timing (for running a full lightcone, i.e. initial condition + other calculations) becomes even longer (w.r.t. single thread) when using >48 threads.

[ChenxiSSS]

Okay, now I see that. Does this mean that it is recommended to use less than 48 threads when possible? I am using a shared-memory cluster, no worries on the core count to memory ratio.

[steven-murray]

Yes, regardless of memory, using too many threads will slow things down. You don't really get much of a speedup after NTHREADS~4, unfortunately (though this might be somewhat system dependent).

…

On Tue, Nov 16, 2021 at 5:46 AM Chenxi Shan ***@***.***> wrote: Okay, now I see that. Does this mean that it is recommended to use less than 48 threads when possible? I am using a shared-memory cluster, no worries on the core count to memory ratio. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/21cmfast/21cmFAST/discussions/263*discussioncomment-1651196__;Iw!!IKRxdwAv5BmarQ!JETHQecFB3dMbQRGFOa3Ap23Nv2cXeYweT-K_Ez5FVI6T8_t3ZceoAieOuPrIg_5e5_-$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAJWRXU7V3PZAYHUUCE5WCLUMJHCLANCNFSM5IEANNGQ__;!!IKRxdwAv5BmarQ!JETHQecFB3dMbQRGFOa3Ap23Nv2cXeYweT-K_Ez5FVI6T8_t3ZceoAieOuPrIrQfp5sV$> .

[ChenxiSSS]

I have done a quick threading test with a small box with the same parameters besides thread number. It does seem that on my system, the IC run time is declining although not scaling well, and the coeval run time fluctuates.

Cells	Box Len (Mpc)	# of Treads	Initial Condition Time (s)	Coevl Time (s)
256^3 IC	1000	1	678.58	120.60
256^3 IC	1000	4	446.56	71.87
256^3 IC	1000	8	400.51	55.12
256^3 IC	1000	16	386.78	63.87
256^3 IC	1000	24	373.09	51.70
256^3 IC	1000	48	364.12	56.28

[qyx268]

yeah, 48 is perhaps the best you can get...