allow to specify thread counts (sync,build) #201
Conversation
when DOS_MAX_THREADS_REPO and/or DOS_MAX_THREADS_BUILD are defined in init.sh they will be used. otherwise the max threads are calculated based on the cpu cores count. if that calculation fails (depends on nproc) a default fallback of: - 4 is used for repo sync (which reflects the current state for repo sync) - `<empty>` is used for make - which yields to no limitation (which reflects the current state for mka / make) to avoid issues regarding being rate limited a max value for repo sync processes will overwrite DOS_MAX_THREADS_REPO when set greater than MAX_THREADS_REPO_RATE (8). Signed-off-by: steadfasterX <[email protected]>
| @@ -34,7 +34,7 @@ buildDevice() { | |||
| cd "$DOS_BUILD_BASE"; | |||
| export OTA_KEY_OVERRIDE_DIR="$DOS_SIGNING_KEYS/$1"; | |||
| pkill java && sleep 10; #XXX: ugly hack | |||
| breakfast "lineage_$1-user" && mka target-files-package otatools && processRelease $1 true $2; | |||
| breakfast "lineage_$1-user" && mka -j${DOS_MAX_THREADS_BUILD} target-files-package otatools && processRelease $1 true $2; | |||
There was a problem hiding this comment.
mka already defaults to all cores +2
There was a problem hiding this comment.
Yes and this change allows it to decrease it. It's not needed when you want to use all cores but this not always wanted - at least for me
| @@ -23,7 +23,7 @@ export -f startPatcher; | |||
|
|
|||
| resetWorkspace() { | |||
| umask 0022; | |||
| repo forall -c 'git add -A && git reset --hard' && rm -rf out DOS_PATCHED_FLAG && repo sync -j8 --force-sync --detach; | |||
| repo forall -j${DOS_MAX_THREADS_REPO} -c 'git add -A && git reset --hard' && rm -rf out DOS_PATCHED_FLAG && repo sync -j${DOS_MAX_THREADS_REPO} --force-sync --detach; | |||
There was a problem hiding this comment.
using too many threads on sync will get you rate limited by google and temporarily blocked
using many threads on reset may cause thrashing, and already completes quickly
There was a problem hiding this comment.
That's why there's a max allowed value which avoids being rate limited. It uses your current used value.
Again this is more to limit (e.g using 1 helps for debug but using other values can help speeding up the process - yes even for reset)
| export MAX_THREADS_REPO_RATE=8; #to avoid being rate limited we never go above this for syncing | ||
| export FALLBACK_MAX_THREADS_BUILD="nolimit"; #if nothing specified we will use all CPU power available | ||
|
|
||
| calcThreads(){ |
There was a problem hiding this comment.
this is needless complexity
There was a problem hiding this comment.
I don't see any needless conplexity here but more flexibility. You might not need it which is ok but i do (and maybe I'm not the only one) ;)
That being said I'm fine if you wanna close this pr
when DOS_MAX_THREADS_REPO and/or DOS_MAX_THREADS_BUILD are defined in init.sh they will be used. otherwise the max threads are calculated based on the cpu cores count.
if that calculation fails (depends on nproc) a default fallback of:
<empty>is used for make - which yields to no limitation (which reflects the current state for mka / make)to avoid issues regarding being rate limited a max value for repo sync processes will overwrite DOS_MAX_THREADS_REPO when set greater than MAX_THREADS_REPO_RATE (8).