Repository navigation
Conversation
Julia threads: on import, PySR set `PYTHON_JULIACALL_THREADS=auto` whenever that variable was unset. juliacall only falls back to `JULIA_NUM_THREADS` when `PYTHON_JULIACALL_THREADS` is unset, and passes `--threads=auto` to Julia, which overrides `JULIA_NUM_THREADS`. So the standard Julia setting was silently ignored: with `JULIA_NUM_THREADS=2` on a 22-CPU machine, `import pysr` started Julia with 22 threads. This oversubscribes nodes shared by several jobs, each of which limits its threads this way. Only default to `auto` when no thread count was requested through `-X juliacall-threads`, `PYTHON_JULIACALL_THREADS` or `JULIA_NUM_THREADS`. GC threads: the `JULIA_NUM_GC_THREADS` default is now derived from that same requested count, rather than from `PYTHON_JULIACALL_THREADS` alone (e.g. `-X juliacall-threads=4` on a 128-core node gave 64 GC threads). It is no longer computed at all when the variable is already set, and a thread count it cannot parse is left for juliacall and Julia to report, instead of raising a bare `int()` error inside PySR. `PYTHON_JULIACALL_OPTLEVEL` is removed: juliacall has no such option (the setting is `PYTHON_JULIACALL_OPTIMIZE`), so it never had an effect. Enabling `-O3` as intended would also make Julia reject package images precompiled at the default `-O2`. Worker processes: `parallelism="multiprocessing"` defaulted `procs` to `multiprocessing.cpu_count()`, which counts every CPU on the node, even inside a Slurm or container allocation restricted to a few of them. Use the CPUs this process may run on instead (`os.process_cpu_count()` or the affinity mask). Also warn when `procs` is passed without `parallelism`, where it was dropped silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PySR no longer overrides JULIA_NUM_THREADS with PYTHON_JULIACALL_THREADS=auto; an empty JULIA_NUM_THREADS still falls back to auto. Default GC threads are sized from the requested count, including -X juliacall-threads. The warning for a non-auto PYTHON_JULIACALL_THREADS is removed. With parallelism='multiprocessing', the default procs is the CPU affinity count. Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JULIA_NUM_THREADSwas ignoredOn import, PySR set
PYTHON_JULIACALL_THREADS=autowhenever that variable was unset. juliacall 0.9.36 only falls back toJULIA_NUM_THREADSwhenPYTHON_JULIACALL_THREADSis unset (option('threads', default=os.getenv('JULIA_NUM_THREADS', '1'))). It passes--threads=autoto Julia, which takes precedence overJULIA_NUM_THREADS. Measured with juliacall 0.9.36 and Julia 1.13.1 on a 22-CPU machine:JULIA_NUM_THREADS=2JULIA_NUM_THREADS=2+PYTHON_JULIACALL_THREADS=auto(whatimport pysrset)So
JULIA_NUM_THREADS=2 python -c "import pysr"started Julia with 22 threads. This oversubscribes nodes shared by several jobs that each limit their threads this way. PySR now defaults toautoonly when no thread count was requested through-X juliacall-threads,PYTHON_JULIACALL_THREADSorJULIA_NUM_THREADS, following juliacall's precedence.GC threads
The
JULIA_NUM_GC_THREADSdefault is now derived from that same requested count, rather than fromPYTHON_JULIACALL_THREADSalone. Previously,-X juliacall-threads=4on a 128-core node gave 64 GC threads. It is also no longer computed at all whenJULIA_NUM_GC_THREADSis already set. A thread count it cannot parse is now left for juliacall and Julia to report, instead of raising a bareint()error inside PySR.PYTHON_JULIACALL_OPTLEVELremovedjuliacall has no
optleveloption; the setting isoptimize/PYTHON_JULIACALL_OPTIMIZE. So the variable never had any effect. Enabling-O3as intended would also make Julia reject package images precompiled at the default-O2, so the dead setting is removed rather than renamed.Worker processes
parallelism="multiprocessing"defaultedprocstomultiprocessing.cpu_count(). That counts every CPU on the node, even inside a Slurm job or container restricted to a few cores, and each worker is a full Julia process. It now uses the CPUs this process may run on:os.process_cpu_count()on Python 3.13+, otherwise the affinity mask. A positiveprocspassed withoutparallelism, where it was silently dropped, now produces a warning.procs=0stays silent, as the test suite uses it.Tests
test_julia_num_threads_is_respected(startup):JULIA_NUM_THREADS=2gives exactly 2 Julia threads. On master it givesthreads=22;on this machine.test_parallelism_defaults: theprocswarning, and the default worker count coming from the available CPUs.The existing startup tests (
test_bad_startup_options,test_autoload_extension_env_precedence) pass.🤖 Generated with Claude Code