[Enhancement] run auto analyze job with priority #55446

murphyatwork · 2025-01-26T03:25:05Z

Why I'm doing:

In case of a database with 10K+ tables, the auto analyze job can sustain for a few hours, so the job will not stop even if out of the running window.

What I'm doing:

To address this issue, we introduce two optimizations:

Check the running window before analyze each table
Sort jobs with priority, considering the stats healthy and stats staleness

Out of scope:

Analyzing external tables do not support priority
If analyze a single table can take a long time, it cannot be interrupted immediately

Fixes #55447

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

2. interrupt the job if out of running window Signed-off-by: Murphy <[email protected]>

Signed-off-by: Murphy <[email protected]>

sonarqubecloud · 2025-01-26T05:46:10Z

Quality Gate failed

Failed conditions
B Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

github-actions · 2025-01-26T08:44:09Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-01-26T08:44:45Z

[FE Incremental Coverage Report]

✅ pass : 38 / 43 (88.37%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/statistic/ExternalAnalyzeJob.java	1	2	50.00%	[199]
🔵	com/starrocks/statistic/NativeAnalyzeJob.java	1	2	50.00%	[227]
🔵	com/starrocks/statistic/StatisticAutoCollector.java	4	5	80.00%	[110]
🔵	com/starrocks/statistic/StatisticsCollectJobFactory.java	12	14	85.71%	[519, 521]
🔵	com/starrocks/statistic/StatisticsCollectJob.java	20	20	100.00%	[]

github-actions · 2025-01-26T08:44:59Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-01-26T09:27:01Z

@Mergifyio backport branch-3.4

github-actions · 2025-01-26T09:27:02Z

@Mergifyio backport branch-3.3

github-actions · 2025-01-26T09:27:03Z

@Mergifyio backport branch-3.2

mergify · 2025-01-26T09:27:09Z

backport branch-3.4

✅ Backports have been created

#55457 [Enhancement] run auto analyze job with priority (backport #55446) has been created for branch branch-3.4 but encountered conflicts

mergify · 2025-01-26T09:27:10Z

backport branch-3.3

✅ Backports have been created

#55456 [Enhancement] run auto analyze job with priority (backport #55446) has been created for branch branch-3.3 but encountered conflicts

mergify · 2025-01-26T09:27:12Z

backport branch-3.2

✅ Backports have been created

#55458 [Enhancement] run auto analyze job with priority (backport #55446) has been created for branch branch-3.2 but encountered conflicts

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 4f8f233) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticsCollectJobFactory.java # fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 4f8f233) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/statistic/ExternalAnalyzeJob.java # fe/fe-core/src/main/java/com/starrocks/statistic/NativeAnalyzeJob.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticAutoCollector.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticsCollectJob.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticsCollectJobFactory.java # fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 4f8f233) Signed-off-by: Murphy <[email protected]> # Conflicts: # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticsCollectJobFactory.java # fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 4f8f233) Signed-off-by: Murphy <[email protected]> # Conflicts: # fe/fe-core/src/main/java/com/starrocks/statistic/ExternalAnalyzeJob.java # fe/fe-core/src/main/java/com/starrocks/statistic/NativeAnalyzeJob.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticAutoCollector.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticsCollectJob.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticsCollectJobFactory.java # fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java

…55456) Co-authored-by: Murphy <[email protected]>

…55458) Co-authored-by: Murphy <[email protected]>

…55457) Co-authored-by: Murphy <[email protected]>

1. sort jobs with priority

8c71746

2. interrupt the job if out of running window Signed-off-by: Murphy <[email protected]>

murphyatwork requested a review from a team as a code owner January 26, 2025 03:25

mergify bot assigned murphyatwork Jan 26, 2025

github-actions bot added 3.4 3.3 3.2 labels Jan 26, 2025

murphyatwork added 2 commits January 26, 2025 12:06

consider the stats staleness in priority

788ea5a

Signed-off-by: Murphy <[email protected]>

fix external table

7f0d6f0

Signed-off-by: Murphy <[email protected]>

Seaven approved these changes Jan 26, 2025

View reviewed changes

kangkaisen approved these changes Jan 26, 2025

View reviewed changes

kangkaisen enabled auto-merge (squash) January 26, 2025 06:11

kangkaisen merged commit 4f8f233 into StarRocks:main Jan 26, 2025
48 of 50 checks passed

github-actions bot removed the 3.4 label Jan 26, 2025

github-actions bot removed 3.3 3.2 labels Jan 26, 2025

mergify bot mentioned this pull request Jan 26, 2025

[Enhancement] run auto analyze job with priority (backport #55446) #55456

Merged

18 tasks

mergify bot mentioned this pull request Jan 26, 2025

[Enhancement] run auto analyze job with priority (backport #55446) #55457

Merged

18 tasks

mergify bot mentioned this pull request Jan 26, 2025

[Enhancement] run auto analyze job with priority (backport #55446) #55458

Merged

18 tasks

wanpengfei-git pushed a commit that referenced this pull request Jan 26, 2025

[Enhancement] run auto analyze job with priority (backport #55446) (#…

80d9b59

…55456) Co-authored-by: Murphy <[email protected]>

github-actions bot added the 3.3-merged label Jan 26, 2025

wanpengfei-git pushed a commit that referenced this pull request Jan 26, 2025

[Enhancement] run auto analyze job with priority (backport #55446) (#…

9738e2a

…55458) Co-authored-by: Murphy <[email protected]>

github-actions bot added the 3.2-merged label Jan 26, 2025

wanpengfei-git pushed a commit that referenced this pull request Jan 27, 2025

[Enhancement] run auto analyze job with priority (backport #55446) (#…

9705fc4

…55457) Co-authored-by: Murphy <[email protected]>

github-actions bot added the 3.4-merged label Jan 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] run auto analyze job with priority #55446

[Enhancement] run auto analyze job with priority #55446

murphyatwork commented Jan 26, 2025 •

edited

Loading

sonarqubecloud bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

mergify bot commented Jan 26, 2025 •

edited

Loading

mergify bot commented Jan 26, 2025 •

edited

Loading

mergify bot commented Jan 26, 2025 •

edited

Loading

[Enhancement] run auto analyze job with priority #55446

[Enhancement] run auto analyze job with priority #55446

Conversation

murphyatwork commented Jan 26, 2025 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarqubecloud bot commented Jan 26, 2025

Quality Gate failed

github-actions bot commented Jan 26, 2025

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Jan 26, 2025

[FE Incremental Coverage Report]

file detail

github-actions bot commented Jan 26, 2025

[BE Incremental Coverage Report]

github-actions bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

github-actions bot commented Jan 26, 2025

mergify bot commented Jan 26, 2025 • edited Loading

✅ Backports have been created

mergify bot commented Jan 26, 2025 • edited Loading

✅ Backports have been created

mergify bot commented Jan 26, 2025 • edited Loading

✅ Backports have been created

murphyatwork commented Jan 26, 2025 •

edited

Loading

mergify bot commented Jan 26, 2025 •

edited

Loading

mergify bot commented Jan 26, 2025 •

edited

Loading

mergify bot commented Jan 26, 2025 •

edited

Loading