Skip to content

[SDTEST-2055] Add documentation for AI flaky test categorization #30589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 22, 2025

Conversation

ManuelPalenzuelaDD
Copy link
Contributor

@ManuelPalenzuelaDD ManuelPalenzuelaDD commented Jul 21, 2025

What does this PR do? What is the motivation?

We've recently released a new feature for the Flaky Test Management product which categorizes your flaky tests with AI. We want to document its behavior.

Merge instructions

Merge readiness:

  • Ready for merge

Copy link
Contributor

github-actions bot commented Jul 21, 2025

✅ Documentation Team Review

The documentation team has approved this pull request. Thank you for your contribution!

Copy link
Contributor

Preview links (active after the build_preview check completes)

Modified Files

@ManuelPalenzuelaDD ManuelPalenzuelaDD marked this pull request as ready for review July 21, 2025 11:27
@ManuelPalenzuelaDD ManuelPalenzuelaDD requested review from a team as code owners July 21, 2025 11:27
Copy link
Contributor

@TovRudyy TovRudyy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to update the Flaky Test Management docs adding the new tag (here) and enable the tag in the metric (FF)

@joepeeples joepeeples self-assigned this Jul 21, 2025
@joepeeples joepeeples added the editorial review Waiting on a more in-depth review label Jul 21, 2025
@joepeeples
Copy link
Contributor

joepeeples commented Jul 21, 2025

Opened DOCS-11573 and assigned myself to follow up with editorial review.

Copy link
Contributor

@joepeeples joepeeples left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with a couple small comments, thanks!


Flaky Test Management uses AI to automatically assign a root cause category to each flaky test based on execution patterns and error signals. This helps you filter, triage, and prioritize flaky tests more effectively.

<div class="alert alert-info"><strong>Note:</strong> A test must have at least one failed execution that includes both <code>@error.message</code> and <code>@error.stack</code> tags to be eligible for categorization. If the test was recently detected, categorization may take some time to complete.</div>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to give a more specific time estimate? Something like "may take several minutes/hours" or "up to x minutes" so customers know better what to expect.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed 👍

@ManuelPalenzuelaDD ManuelPalenzuelaDD merged commit e535510 into master Jul 22, 2025
15 checks passed
@ManuelPalenzuelaDD ManuelPalenzuelaDD deleted the manuel.palenzuela/SDTEST-2055-add-docs branch July 22, 2025 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
editorial review Waiting on a more in-depth review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants