Group-level session sharing, how to increase the number of parallel tasks #5155
-
|
If ''spark.dynamicAllocation.maxExecutors'' is set to 400, then only 400 tasks can be run at the same time, and other newly submitted SQL will be blocked. How to solve this concurrency problem |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Increasing executor cores, typically, |
Beta Was this translation helpful? Give feedback.
-
|
What I mean is that as long as a job that takes up a lot of resources fills up the executor, other jobs can only wait for the resources to be released. |
Beta Was this translation helpful? Give feedback.
-
|
https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application |
Beta Was this translation helpful? Give feedback.
https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application