Trt 2389 test summaries #4868
Trt 2389 test summaries #4868
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
WalkthroughAdd TestSummaryByPeriodRow and a BigQuery-backed ListTestSummaryByPeriod (with retry and mock support); add a "tests" data-type path in the historical analyzer that fetches, formats, and writes test summaries; add related helpers and CLI validation. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes ✨ Finishing touches
Comment |
|
/hold |
openshift-ci
bot
added
the
do-not-merge/hold
openshift-ci
bot
added
the
approved
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go (1)
25-52: Newdata-type=testsflow is cleanly separated; consider future configurabilityThe early branch in
RunandrunTestsDataTypecleanly bypass the comparison logic fordata-type=tests, fetching summaries once and writing them directly viaformatTestOutput, without disturbing the alerts/disruptions path. The hardcodedsuiteName,daysBack, andminTestCountare reasonable defaults, but if this grows beyond a single use case, promoting them to flags or config would improve long‑term flexibility.Also applies to: 100-132
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (7)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go(1 hunks)pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go(2 hunks)pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go(1 hunks)pkg/jobrunaggregator/jobrunaggregatorlib/util.go(1 hunks)pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go(2 hunks)pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go(2 hunks)pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**
⚙️ CodeRabbit configuration file
-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.
Files:
pkg/jobrunaggregator/jobrunaggregatorlib/util.gopkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.gopkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.gopkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.gopkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.gopkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.gopkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go
🧬 Code graph analysis (3)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-20)
pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-20)
pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (2)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-20)pkg/api/types.go (1)
Release(452-466)
🔇 Additional comments (6)
pkg/jobrunaggregator/jobrunaggregatorlib/util.go (1)
39-58: BigQuery label addition is consistent and low‑riskThe new
bigQueryLabelValueTestSummaryByPeriodconstant matches existing naming/usage patterns for query labels; no issues from a maintainability or behavior standpoint.pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go (1)
29-49:data-type=testswiring and validation look coherentAdding
"tests"tosupportedDataTypesand relaxing only the--current/--previous-releaserequirements (while still enforcing BigQuery auth/coords when--newis empty) matches the new tests path, which fetches directly from BigQuery without doing comparisons. No obvious correctness or UX regressions here.Also applies to: 51-77
pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (1)
170-185: Test-summary formatting helper is straightforward and consistentThe in-place sort by release, failure count (desc), then test name mirrors the query’s ordering and existing
formatOutputpatterns, andjson.MarshalIndentkeeps output stable for diffs. Behavior for empty input is defensive and safe.pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)
188-196: Retry wrapper mirrors existing client behavior
ListTestSummaryByPeriodon the retrying client follows the sameretry.OnError(slowBackoff, isReadQuotaError, …)pattern as the other methods, so retry semantics stay consistent and localized to quota-style failures.If you haven’t already, run a quick integration test hitting the real BigQuery-backed implementation to confirm retry behavior and error wrapping are as expected.
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
1-20: Struct shape correctly matches the BigQuery query schema
TestSummaryByPeriodRow’s field names, types, andbigquerytags line up with theListTestSummaryByPeriodSELECT clause (SUMs asint64, ratios/durations asfloat64, dates ascivil.Date), so decoding should be straightforward.Please confirm via a small end‑to‑end query (or unit test with a fake
RowIterator) that BigQuery scanning into this struct works without field/tag mismatches.pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
70-76: Test-summary query is well‑structured and consistent with existing CIDataClient patternsThe new
ListTestSummaryByPeriodmethod is documented clearly on the interface and the implementation’s SQL matches theTestSummaryByPeriodRowschema (field aliases ↔ struct tags). The aggregation logic (SUM/SAFE_DIVIDE/AVG, window viaCURRENT_DATE()anddaysBack,HAVING SUM(test_count) > @min_test_count) and labeling (bigQueryLabelValueTestSummaryByPeriod) align with how other queries in this client are written, so it should be straightforward to operate and debug.If possible, run this against a representative dataset to confirm:
- Parameter binding for
days_back/min_test_countis accepted by BigQuery.- The
DATE_SUB(CURRENT_DATE(), INTERVAL @days_back DAY)filter yields the expected window.- The decoded
TestSummaryByPeriodRowvalues (includingFailureRateandPeriodStart/End) match manual query results.Also applies to: 1107-1173
neisw
force-pushed
the
TRT-2389-test-summaries
branch
from
c5366f4 to
cee2982
Compare
|
/test images |
1 similar comment
|
/test images |
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
1107-1107: Consider validating input parameters.Adding validation for
daysBackandminTestCountwould prevent unexpected behavior from negative or extreme values.🛡️ Suggested validation
func (c *ciDataClient) ListTestSummaryByPeriod(ctx context.Context, suiteName, releaseName string, daysBack, minTestCount int) ([]jobrunaggregatorapi.TestSummaryByPeriodRow, error) { + if daysBack < 1 { + return nil, fmt.Errorf("daysBack must be at least 1, got %d", daysBack) + } + if minTestCount < 0 { + return nil, fmt.Errorf("minTestCount must be non-negative, got %d", minTestCount) + } + // Query to summarize test results for a specific suite and release over a time period
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (2)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.gopkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go
🧰 Additional context used
📓 Path-based instructions (1)
**
⚙️ CodeRabbit configuration file
-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.
Files:
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.gopkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go
🧬 Code graph analysis (1)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-21)
🔇 Additional comments (2)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
70-76: LGTM! Well-documented interface addition.The new method signature is clear and follows existing patterns. The parameter documentation helps callers understand the expected inputs.
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
1-21: LGTM! Clean data structure definition.The struct is well-designed with:
- Appropriate field types (
civil.Datefor date-only values,int64for counts,float64for rates/averages)- Correct BigQuery tags matching the query fields
- Clear documentation including the purpose of
FirstSampleDate
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go:
- Around line 1107-1195: The HAVING clause in ListTestSummaryByPeriod (function
ciDataClient.ListTestSummaryByPeriod) currently uses a strict greater-than check
SUM(main.test_count) > @min_test_count which excludes tests with exactly
minTestCount runs; change the HAVING condition to use >= (SUM(main.test_count)
>= @min_test_count) so the comparison is inclusive of minTestCount to match the
parameter intent and documentation.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (1)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go
🧰 Additional context used
📓 Path-based instructions (1)
**
⚙️ CodeRabbit configuration file
-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.
Files:
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go
🧬 Code graph analysis (1)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-21)
🔇 Additional comments (1)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
70-76: Clear interface documentation.The method signature and documentation are well-structured, making the purpose and parameters easy to understand.
| func (c *ciDataClient) ListTestSummaryByPeriod(ctx context.Context, suiteName, releaseName string, daysBack, minTestCount int) ([]jobrunaggregatorapi.TestSummaryByPeriodRow, error) { | ||
| // Query to summarize test results for a specific suite and release over a time period | ||
| // Groups by release and test_name only (no infrastructure dimensions) | ||
| // Calculates total test_count, failure_count, flake_count, failure_rate, and avg_duration_ms | ||
| // Filters results to only include tests with sufficient test runs | ||
| // Also includes first_seen_date: the earliest date each test_name appeared across all releases for this suite | ||
| queryString := c.dataCoordinates.SubstituteDataSetLocation(` | ||
| WITH earliest_test_dates AS ( | ||
| -- Pre-compute the earliest date each test was seen across all releases for this suite | ||
| -- Look back max(100, @days_back) days to ensure we capture historical context | ||
| SELECT | ||
| test_name, | ||
| MIN(date) AS first_sample_date | ||
| FROM | ||
| DATA_SET_LOCATION.TestsSummaryByDate | ||
| WHERE | ||
| suite = @suite_name | ||
| AND date >= DATE_SUB(CURRENT_DATE(), INTERVAL GREATEST(100, @days_back) DAY) | ||
| AND date <= CURRENT_DATE() | ||
| GROUP BY | ||
| test_name | ||
| ) | ||
| SELECT | ||
| main.release, | ||
| main.test_name, | ||
| SUM(main.test_count) AS total_test_count, | ||
| SUM(main.failure_count) AS total_failure_count, | ||
| SUM(main.flake_count) AS total_flake_count, | ||
| SAFE_DIVIDE(SUM(main.failure_count), SUM(main.test_count)) AS failure_rate, | ||
| AVG(main.avg_duration_ms) AS avg_duration_ms, | ||
| MIN(main.date) AS period_start, | ||
| MAX(main.date) AS period_end, | ||
| COUNT(DISTINCT main.date) AS days_with_data, | ||
| earliest.first_sample_date | ||
| FROM | ||
| DATA_SET_LOCATION.TestsSummaryByDate AS main | ||
| LEFT JOIN | ||
| earliest_test_dates AS earliest | ||
| ON | ||
| main.test_name = earliest.test_name | ||
| WHERE | ||
| main.suite = @suite_name | ||
| AND main.release = @release_name | ||
| AND main.date >= DATE_SUB(CURRENT_DATE(), INTERVAL @days_back DAY) | ||
| AND main.date <= CURRENT_DATE() | ||
| GROUP BY | ||
| main.release, | ||
| main.test_name, | ||
| earliest.first_sample_date | ||
| HAVING | ||
| SUM(main.test_count) > @min_test_count | ||
| ORDER BY | ||
| main.release, | ||
| total_failure_count DESC, | ||
| main.test_name | ||
| `) | ||
|
|
||
| query := c.client.Query(queryString) | ||
| query.Labels = map[string]string{ | ||
| bigQueryLabelKeyApp: bigQueryLabelValueApp, | ||
| bigQueryLabelKeyQuery: bigQueryLabelValueTestSummaryByPeriod, | ||
| } | ||
| query.QueryConfig.Parameters = []bigquery.QueryParameter{ | ||
| {Name: "suite_name", Value: suiteName}, | ||
| {Name: "release_name", Value: releaseName}, | ||
| {Name: "days_back", Value: daysBack}, | ||
| {Name: "min_test_count", Value: minTestCount}, | ||
| } | ||
|
|
||
| rows, err := query.Read(ctx) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to query generic test summary by period with %q: %w", queryString, err) | ||
| } | ||
|
|
||
| results := []jobrunaggregatorapi.TestSummaryByPeriodRow{} | ||
| for { | ||
| row := &jobrunaggregatorapi.TestSummaryByPeriodRow{} | ||
| err = rows.Next(row) | ||
| if err == iterator.Done { | ||
| break | ||
| } | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| results = append(results, *row) | ||
| } | ||
|
|
||
| return results, nil | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go | sed -n '1107,1195p'Repository: openshift/ci-tools
Length of output: 3664
🏁 Script executed:
# Search for test cases or usage of ListTestSummaryByPeriod
rg "ListTestSummaryByPeriod" -A 5 -B 5Repository: openshift/ci-tools
Length of output: 8755
🏁 Script executed:
# Look for any documentation about minTestCount parameter
rg "minTestCount|min_test_count" -B 3 -A 3 -iRepository: openshift/ci-tools
Length of output: 8276
🏁 Script executed:
# Check if there are similar patterns with comparison operators in other queries
rg "HAVING.*COUNT|HAVING.*SUM" -A 2 -B 2Repository: openshift/ci-tools
Length of output: 44
Change HAVING clause to use >= instead of >.
Line 1157: The HAVING clause uses SUM(main.test_count) > @min_test_count (strict greater than), which excludes tests with exactly minTestCount runs. The parameter documentation states "Minimum number of test executions required to include a test in results" and the parameter name minTestCount both suggest inclusive comparison. Change to >= to match the documented intent.
🤖 Prompt for AI Agents
In @pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go around lines 1107
- 1195, The HAVING clause in ListTestSummaryByPeriod (function
ciDataClient.ListTestSummaryByPeriod) currently uses a strict greater-than check
SUM(main.test_count) > @min_test_count which excludes tests with exactly
minTestCount runs; change the HAVING condition to use >= (SUM(main.test_count)
>= @min_test_count) so the comparison is inclusive of minTestCount to match the
parameter intent and documentation.
|
/retest-required |
openshift-ci
bot
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: neisw, stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Scheduling required tests: Scheduling tests matching the |
|
@neisw: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
Adds
testsdata typeProduces test_summaries.json
Need to review the importance of this if we are pivoting to openshift/origin#30604