Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/content/docs/ci-insights.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ hubAccent: true
import Docset from '../../components/DocsetGrid/Docset.astro';
import DocsetGrid from '../../components/DocsetGrid/DocsetGrid.astro';

CI Insights helps you monitor your pipelines, detect flaky tests and jobs, and optimize
CI Insights helps you monitor your pipelines, detect flaky tests and jobs, and improve
your CI efficiency. This guide walks you through enabling CI Insights via
GitHub and covers basic configuration steps.

> First time here? Follow these 3 steps to see data fast:
> 1. Install the Mergify GitHub App on a repo with existing CI.
> 2. Ensure your tests produce a supported format (JUnit XML, pytest, Jest, etc.) pick a framework below if needed.
> 2. Ensure your tests produce a supported format (JUnit XML, pytest, Jest, etc.); pick a framework below if needed.
> 3. Re-run a workflow (or push a trivial commit) then refresh this page to view jobs & tests.

## Understanding CI Insights
Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/ci-insights/auto-retry.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ re-running all failed jobs while preserving successful ones.
- **Data-Driven Insights**: Track which jobs are frequently retried to identify
systematic issues

- **Time Savings**: Reduce manual retry actions and faster feedback loops
- **Time Savings**: Fewer manual retry actions and faster feedback loops

## Configuration

Expand Down Expand Up @@ -89,7 +89,7 @@ failures. Each rule specifies:

Conditions let you control which job failures should trigger a retry. Each
condition matches against a specific attribute of the failed job. You can
combine multiple conditions in a single rule all conditions must match for
combine multiple conditions in a single rule; all conditions must match for
the rule to apply.

The following condition attributes are available:
Expand All @@ -113,7 +113,7 @@ The following condition attributes are available:
- **Log pattern**: Match against the content of the job logs and GitHub
Actions check-run annotations using regular expressions. Each pattern is
evaluated against individual log lines. You can specify multiple
patterns a retry is triggered if any of the patterns match. For
patterns; a retry is triggered if any of the patterns match. For
example, `timeout|connection refused` would match any log line
containing either "timeout" or "connection refused". The rule builder
includes a **Test your log patterns** panel to check your log-pattern
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/ci-insights/flaky-test-detection.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ pipeline.

## Setting Up Flaky Test Detection

See CIspecific setup guides:
See CI-specific setup guides:

<DocsetGrid>
<Docset title="GitHub Actions"
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/ci-insights/setup/buildkite.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ every 12 hours on weekdays.
the most impact

- **Weekday Schedule**: Run Monday to Friday to avoid running when no changes
are made on the code
are made to the code

- **Job name**: Ensure the step label matches the one running tests on your
pull request, allowing CI Insights to identify them. It can be overridden
Expand Down
10 changes: 5 additions & 5 deletions src/content/docs/ci-insights/setup/github-actions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Actions and how to configure repeated test executions for flaky test detection.
page](https://dashboard.mergify.com/integrations/github)
([docs](/integrations/github)).

2. In order to upload your test results, you'll need to [configure your
2. To upload your test results, you'll need to [configure your
test](#test-framework-configuration). Before doing so, create an application
key with `ci` scope as described in the [API Usage documentation](/api/usage)
and set it as `MERGIFY_TOKEN` in your GitHub Actions secrets.
Expand Down Expand Up @@ -54,14 +54,14 @@ Before setting up flaky test detection, ensure you have:

2. Created an application key with `ci` scope and set it as `MERGIFY_TOKEN` in GitHub Actions secrets

3. Configured test integration (e.g., `pytest-mergify` or `mergify cli upload`)
3. Configured test integration (e.g., `pytest-mergify` or JUnit report upload)
for your test framework

### GitHub Actions Setup

The recommended approach is to use a scheduled workflow that runs your test
suite multiple times on the default branch. Here's an example configuration
that runs tests twice daily with 5 parallel executions:
that runs tests twice daily with 5 repeated executions:

- Example with native integration (e.g. `pytest-mergify`)

Expand Down Expand Up @@ -100,7 +100,7 @@ jobs:
exit $failed
```

- Examples with `mergify cli upload` integration
- Example with JUnit report upload via `gha-mergify-ci`

```yaml
name: CI
Expand Down Expand Up @@ -152,7 +152,7 @@ jobs:
the most impact

- **Weekday Schedule**: The example runs Monday to Friday (`1-5`) to avoid
running when no changes are made on the code
running when no changes are made to the code

- **Job name**: Ensure the job name is the same as the one running tests on your pull request,
allowing CI Insights to identify them. It can be overridden with `MERGIFY_TEST_JOB_NAME` if needed.
Expand Down
14 changes: 7 additions & 7 deletions src/content/docs/ci-insights/setup/jenkins.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,14 @@ Before setting up flaky test detection, ensure you have:

2. Created an application key with `ci` scope and set it as `MERGIFY_TOKEN` in Jenkins credentials.

3. Configured test integration (e.g., `pytest-mergify` or `mergify cli upload`)
for your test framework
3. Configured test integration (e.g., `pytest-mergify` or JUnit report upload
with `mergify ci junit-process`) for your test framework

### Jenkins job Setup
### Jenkins Job Setup

The recommended approach is to use a scheduled job that runs your test
suite multiple times on the default branch. Here's an example configuration
that runs tests twice daily with 5 parallel executions:
that runs tests twice daily with 5 repeated executions:

- Example with native integration (e.g. `pytest-mergify`)

Expand Down Expand Up @@ -117,7 +117,7 @@ pipeline {
}
```

- Examples with `mergify cli upload` integration
- Example with JUnit report upload via `mergify ci junit-process`

```groovy
pipeline {
Expand All @@ -142,7 +142,7 @@ pipeline {
# npm test -- --reporters=default --reporters=jest-junit --outputFile=test-results-$i.xml
done
set -e
# Quote the glob so Mergify expands it rather than the shell
# Quote the glob so Mergify expands it rather than the shell;
# recommended when many reports match.
mergify ci junit-process 'test-results-*.xml'
'''
Expand All @@ -161,7 +161,7 @@ pipeline {
the most impact

- **Weekday Schedule**: The example runs Monday to Friday (`1-5`) to avoid
running when no changes are made on the code
running when no changes are made to the code

- **Job name**: Ensure the job name is the same as the one running tests on your pull request,
allowing CI Insights to identify them. It can be overridden with `MERGIFY_TEST_JOB_NAME` if needed.
Expand Down
20 changes: 11 additions & 9 deletions src/content/docs/monorepo-ci.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@ title: Monorepo CI
description: Run the right tests for the right pull requests by combining scopes with your CI system.
---

Monorepos supercharge collaboration, but they also make continuous integration
expensive. Every change potentially touches hundreds of packages or services,
and running the full test matrix for all of them at every pull request update
quickly becomes unsustainable.
Monorepos make continuous integration expensive. Every change potentially
touches hundreds of packages or services, and running the full test matrix for
all of them at every pull request update quickly becomes unsustainable.

Mergify Monorepo CI brings precision testing to your existing CI pipelines by
tagging each pull request with **scopes**. Scopes are named slices of your
Expand All @@ -22,8 +21,10 @@ With scopes in place you can:
- Run only the CI jobs that matter for those scopes.
- Reuse those scopes later in the merge queue to build smarter batches.

Today, GitHub Actions is the first CI platform with native support through our
[`gha-mergify-ci`](https://github.com/Mergifyio/gha-mergify-ci) action.
Native support is available for GitHub Actions through our
[`gha-mergify-ci`](https://github.com/Mergifyio/gha-mergify-ci) action and for
Buildkite through the
[`mergify-ci` plugin](https://github.com/Mergifyio/mergify-ci-buildkite-plugin).

## How it works

Expand Down Expand Up @@ -73,9 +74,9 @@ strict digraph {

1. **Define scopes** in your `.mergify.yml` file by mapping the areas of your repository (via file
patterns) to scope names that matter for your CI.
2. **Collect scopes in CI.** A helper (currently
[`gha-mergify-ci`](https://github.com/Mergifyio/gha-mergify-ci)) inspects the

2. **Collect scopes in CI.** A helper such as
[`gha-mergify-ci`](https://github.com/Mergifyio/gha-mergify-ci) inspects the
pull request diff and dispatches only the jobs whose scopes are touched.

3. **Drive your jobs conditionally.** Use the reported scopes to decide which
Expand Down Expand Up @@ -109,6 +110,7 @@ Once those scopes exist, pick the workflow that matches your setup:

- [Scope configuration reference](/merge-queue/scopes)
- [GitHub Actions setup](/monorepo-ci/github-actions)
- [Buildkite setup](/monorepo-ci/buildkite)
- [Merge queue monorepo overview](/merge-queue/monorepo)

Looking for a different CI provider? [Let us know](mailto:support@mergify.com); more platforms are on our roadmap.
2 changes: 1 addition & 1 deletion src/content/docs/monorepo-ci/github-actions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ jobs:
inspects the pull request diff and returns a JSON map of scopes set to `true`
or `false`.

- Each job checks the scope it cares about before running, dramatically reducing
- Each job checks the scope it cares about before running, reducing
redundant builds.

- The final `ci-gate` job ensures that the aggregated status reflects the actual
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/prevention.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ reliability.
Prevention requires a test framework plugin that instruments test runs, and it
is opt-in per repository. Once a plugin is installed, set a rerun budget on the
[Prevention page](https://dashboard.mergify.com/test-insights/prevention) in the
dashboard to enable it — until then, Mergify collects test results but does not
dashboard to enable it. Until then, Mergify collects test results but does not
rerun tests.

See the [test framework configuration](/test-insights#test-framework-configuration)
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/test-insights/quarantine.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ To ensure the correct setup, refer to [the documentation for your test framework

## Adding and removing tests from quarantine

### From The Mergify Dashboard
### From the Mergify Dashboard

The [Mitigation page](https://dashboard.mergify.com/test-insights/mitigation) on the dashboard contains the list of
tests that were retrieved from the CI Insights integration.
From there, you can add or remove a test from quarantine.

### From The Mergify CLI
### From the Mergify CLI

The [Mergify CLI](/cli/tests) can manage quarantines from the terminal,
which is handy for scripting or for letting an AI coding agent quarantine a flaky test it just hit:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<li>The CLI provides information about the upload. Check the logs in GitHub Actions.</li>
<li>File Paths: Double-check that the output file matches the path used in `report_path`.</li>
<li>Permissions: Make sure the `MERGIFY_TOKEN` is valid and setup in your GitHub Actions secrets [as explained in the docs](/ci-insights#enabling-ci-insights-for-github).</li>
<li>Permissions: Make sure the `MERGIFY_TOKEN` is valid and setup in your GitHub Actions secrets [as explained in the docs](/ci-insights/setup/github-actions#enabling-ci-insights-for-github-actions).</li>
<li>Workflow Conditions: If your step is not running, confirm the if condition is actually triggered in your job.</li>
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
In order to benefit from Test Insights Quarantine, you need to add `continue-on-error: true`
To benefit from Test Insights Quarantine, you need to add `continue-on-error: true`
in your GitHub Actions step that executes your tests and generates the JUnit file.
The step running the `gha-mergify-ci` action will determine the success or failure conclusion,
considering quarantined tests.
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/test-insights/test-frameworks/cypress.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ JUnit reports and upload them to Test Insights using your CI workflow.
By default, Cypress doesn't generate JUnit-format test results. You need to
enable it in your `cypress.config.ts` file.

This is [documenteded on Cypress
This is [documented on the Cypress
website](https://docs.cypress.io/app/tooling/reporters).

The best way to do this is to leverage `cypress-multi-reporters` so you can
The best way to do this is to use `cypress-multi-reporters` so you can
both have JUnit generated while keeping the standard output.

1. Add `cypress-multi-reporters` to your package file, e.g.,
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/test-frameworks/golang.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ gotestsum --junitfile junit.xml
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to Test
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down
7 changes: 3 additions & 4 deletions src/content/docs/test-insights/test-frameworks/jest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,8 @@ them to **Test Insights** using your CI workflow.

## Generate a JUnit Report with Jest

Jest has built-in support for JUnit XML reports. You can configure Jest to
output JUnit reports by updating your Jest configuration or using command-line
options.
Jest can output JUnit XML reports through the `jest-junit` reporter. You can
enable it in your Jest configuration or with command-line options.

### Using Jest Configuration

Expand Down Expand Up @@ -68,7 +67,7 @@ JEST_JUNIT_OUTPUT_DIR=. JEST_JUNIT_OUTPUT_NAME=junit.xml npx jest --reporters=de
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to CI
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/test-frameworks/mstest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ dotnet test --logger "console;verbosity=detailed" --logger "junit;LogFilePath=ju
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to CI
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/test-frameworks/nunit.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ dotnet test --settings test.runsettings
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to CI
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/test-frameworks/pest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ TestSuite::getInstance()->log('junit', 'junit.xml');
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to CI
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/test-frameworks/phpunit.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ composer test
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to CI
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/test-frameworks/rust.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ cargo nextest run --message-format junit --output-file junit.xml
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to CI
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/test-insights/test-frameworks/testng.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ java -cp "your-classpath" org.testng.TestNG -listener org.testng.reporters.JUnit
<CIInsightsSetupNote />

After generating the JUnit report, add a step to upload the results to CI
Insights using the mergifyio/gha-mergify-ci action.
Insights using the `mergifyio/gha-mergify-ci` action.

For example, in your workflow file:

Expand Down