Skip to content

fix(ci): add matrix sharding to import profiler workflow - #17779

Open
hebaalazzeh wants to merge 6 commits into
mainfrom
fix/import-profiler-warnings
Open

fix(ci): add matrix sharding to import profiler workflow#17779
hebaalazzeh wants to merge 6 commits into
mainfrom
fix/import-profiler-warnings

Conversation

@hebaalazzeh

@hebaalazzeh hebaalazzeh commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR implements static matrix sharding for the import-profiler workflow to run modified packages in parallel across 8 shards, speeding up CI execution time on multi-package PRs.

Key Changes

  • Parallel Sharding: Updated .github/workflows/import-profiler.yml to split modified packages under packages/ across 8 GitHub Actions runner matrix shards using modulo logic (idx % TOTAL_SHARDS).
  • Unified Status Check: Added an all-import-profiles aggregation job that depends on the matrix run (needs: import-profile) to serve as the single required GitHub status check for branch protection and merge queue rules.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the --fail-threshold value from 5000 to 120000 in ci/run_single_test.sh when executing the profiler script. The feedback suggests double-quoting the ${PROFILER_SCRIPT} and ${PACKAGE_NAME} variables to prevent potential word splitting and globbing issues.

Comment thread ci/run_single_test.sh Outdated
Comment thread ci/run_single_test.sh Outdated
@hebaalazzeh
hebaalazzeh marked this pull request as ready for review July 20, 2026 18:30
@hebaalazzeh
hebaalazzeh requested a review from a team as a code owner July 20, 2026 18:30
@hebaalazzeh
hebaalazzeh force-pushed the fix/import-profiler-warnings branch from 2e75bfa to 0b2d9fd Compare July 20, 2026 18:30
@hebaalazzeh hebaalazzeh self-assigned this Jul 20, 2026
@hebaalazzeh
hebaalazzeh force-pushed the fix/import-profiler-warnings branch 4 times, most recently from 37448ee to 99f6a31 Compare July 20, 2026 19:14
@hebaalazzeh
hebaalazzeh force-pushed the fix/import-profiler-warnings branch from 99f6a31 to c13a5e7 Compare July 20, 2026 19:16

@parthea parthea left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create an example PR which includes changes to packages so we can see the results? Right now we only see No packages assigned to Shard 0. which is expected, but it would be great to see an example

@hebaalazzeh

Copy link
Copy Markdown
Contributor Author

Can you create an example PR which includes changes to packages so we can see the results? Right now we only see No packages assigned to Shard 0. which is expected, but it would be great to see an example

see: #17936

@parthea parthea assigned parthea and unassigned hebaalazzeh Aug 4, 2026
@parthea

parthea commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

In the demo PR, I see these errors. If that's expected, can we make the error more user friendly?

https://github.com/googleapis/google-cloud-python/actions/runs/30484508360/job/90686692668?pr=17936

/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
/home/runner/work/_temp/1fd2f340-51ce-4adb-b3d4-044836d51fd7.sh: line 12: [: SHARD_INDEX: integer expression expected
No packages assigned to Shard 4.

@parthea parthea assigned hebaalazzeh and unassigned parthea Aug 5, 2026
@hebaalazzeh hebaalazzeh changed the title fix(ci): increase import profiler absolute threshold fix(ci): add matrix sharding to import profiler workflow Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants