chore(reporter): report per-shard durations #38623

Skn0tt · 2025-12-19T11:11:50Z

See #38604. Adds per-shard duration reporting to the merge reporter, and a visualisation of that data to HTML. In a next step, i'd like to add the heuristic from #38635 (comment), too.

docs/src/test-reporter-api/class-reporter.md

docs/src/test-reporter-api/class-testresult.md

packages/playwright/src/reporters/merge.ts

packages/html-reporter/src/speedboard.tsx

dgozman · 2026-01-05T13:03:57Z

packages/html-reporter/src/barchart.tsx

+  const width = 800;
+
+  // Calculate left margin based on longest group name
+  // Rough estimate: 7 pixels per character at fontSize 12


I wonder whether you should make it two columns, let the titles column take as much space as it needs (say, up to 50%) and the bars column take the rest?

I'm not exactly sure what you mean, but I allowed the titles to take up to 50% now.

packages/html-reporter/src/barchart.tsx

dgozman · 2026-01-05T13:06:33Z

packages/html-reporter/src/barchart.tsx

+            const y = groupY + barIndex * (barHeight + barSpacing);
+            barIndex++;
+
+            const colors = ['var(--color-scale-yellow-3)', 'var(--color-scale-orange-4)'];


Perhaps we should have more? Have you tried this with 5 shards?

i've added one more colour. it's hard to find colours in our swatch that match well

dgozman · 2026-01-05T13:07:43Z

packages/html-reporter/src/barchart.tsx

+  return `${minutes}m ${seconds}s`;
+};
+
+export const GroupedBarChart = ({


This reminds me of timeline.tsx quite a lot!

Maybe that was part of the training data 😁

dgozman · 2026-01-06T13:46:12Z

docs/src/test-reporter-api/class-reporter.md

  - `duration` <[int]> Test run duration in milliseconds.
+  - `shards` <[Array]<[Object]>> Only present on merged reports
+    - `shardIndex` ?<[int]> The index of the shard, one-based.
+    - `tag` ?<[Array]<[string]>> Bot [`property: TestConfig.tag`] that differentiates CI environments


Suggested change

- `tag` ?<[Array]<[string]>> Bot [`property: TestConfig.tag`] that differentiates CI environments

- `tag` ?<[Array]<[string]>> Global [`property: TestConfig.tag`] that differentiates CI environments

dgozman · 2026-01-06T13:48:15Z

packages/html-reporter/src/speedboard.tsx

+  const bots: Record<string, number[]> = {};
+  for (const shard of shards) {
+    const botName = shard.tag.map(t => {
+      if (t.startsWith('@'))


Why remove @? I think we should keep it to make things familiar.

dgozman · 2026-01-06T13:52:49Z

packages/playwright/src/isomorphic/teleReceiver.ts

  workerIndex: number;
  parallelIndex: number;
-  shardIndex: number;
+  shardIndex?: number;


Shouldn't we also update TeleTestResult.shardIndex definition to be undefined by default?

dgozman · 2026-01-06T13:54:11Z

utils/generate_types/overrides-testReporter.d.ts

+    shardIndex?: number;
+
+    /**
+     * Bot tag that differentiates CI environments.


Suggested change

* Bot tag that differentiates CI environments.

* Global tag that differentiates CI environments.

github-actions · 2026-01-06T14:39:47Z

Test results for "MCP"

2822 passed, 116 skipped

Merge workflow run.

github-actions · 2026-01-06T14:43:33Z

Test results for "tests 1"

15 failed
❌ [default] › debug-tests.spec.ts:58 › should debug one test and pause at end @vscode-extension
❌ [default] › debug-tests.spec.ts:172 › should debug one test and pause on error @vscode-extension
❌ [default] › debug-tests.spec.ts:380 › should run global setup before debugging @vscode-extension
❌ [default] › debug-tests.spec.ts:465 › should debug multiple tests and stop on first failure @vscode-extension
❌ [default] › debug-tests.spec.ts:516 › should not pause at the end of a setup test @vscode-extension
❌ [default-reuse] › debug-tests.spec.ts:58 › should debug one test and pause at end @vscode-extension
❌ [default-reuse] › debug-tests.spec.ts:172 › should debug one test and pause on error @vscode-extension
❌ [default-reuse] › debug-tests.spec.ts:380 › should run global setup before debugging @vscode-extension
❌ [default-reuse] › debug-tests.spec.ts:465 › should debug multiple tests and stop on first failure @vscode-extension
❌ [default-reuse] › debug-tests.spec.ts:516 › should not pause at the end of a setup test @vscode-extension
❌ [default-trace] › debug-tests.spec.ts:58 › should debug one test and pause at end @vscode-extension
❌ [default-trace] › debug-tests.spec.ts:172 › should debug one test and pause on error @vscode-extension
❌ [default-trace] › debug-tests.spec.ts:380 › should run global setup before debugging @vscode-extension
❌ [default-trace] › debug-tests.spec.ts:465 › should debug multiple tests and stop on first failure @vscode-extension
❌ [default-trace] › debug-tests.spec.ts:516 › should not pause at the end of a setup test @vscode-extension

5 flaky

⚠️ [installation tests] › playwright-cli.spec.ts:21 › cli should work `@package-installations-macos-latest`
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:1082 › cli codegen › should not throw csp directive violation errors `@firefox-ubuntu-22.04-node20`
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104 › should work with strict CSP policy `@firefox-ubuntu-22.04-node20`
⚠️ [playwright-test] › ui-mode-trace.spec.ts:183 › should show snapshots for steps `@windows-latest-node20`
⚠️ [playwright-test] › ui-mode-trace.spec.ts:796 › should update state on subsequent run `@windows-latest-node20`

34396 passed, 696 skipped

Merge workflow run.

chore(reporter): report per-shard durations

8067d6e

Skn0tt requested review from pavelfeldman and yury-s December 19, 2025 11:11

Skn0tt self-assigned this Dec 19, 2025

This comment has been minimized.

Sign in to view

Skn0tt added 2 commits January 5, 2026 13:03

Merge branch 'main' into shard-diagram

dafa348

fix md

82e4f92

Skn0tt force-pushed the shard-diagram branch from 9d76a38 to 82e4f92 Compare January 5, 2026 12:21

Skn0tt requested a review from dgozman January 5, 2026 12:23

This comment has been minimized.

Sign in to view

fix separately

de7a8ec

This comment has been minimized.

Sign in to view

dgozman reviewed Jan 5, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

Skn0tt added 2 commits January 5, 2026 17:01

address

7764723

more

fb8dd70

This comment has been minimized.

Sign in to view

Skn0tt added 6 commits January 6, 2026 09:55

take up to 50%

ecaf185

add test

892b295

print durations

937c8c8

more colours

e4c5891

improve aria

7e8cec4

move css

57b591b

Skn0tt requested a review from dgozman January 6, 2026 09:35

This comment has been minimized.

Sign in to view

dgozman approved these changes Jan 6, 2026

View reviewed changes

address feedback

1930d0a

Skn0tt merged commit 5579232 into microsoft:main Jan 6, 2026
32 of 35 checks passed

	- `tag` ?<[Array]<[string]>> Bot [`property: TestConfig.tag`] that differentiates CI environments
	- `tag` ?<[Array]<[string]>> Global [`property: TestConfig.tag`] that differentiates CI environments

	* Bot tag that differentiates CI environments.
	* Global tag that differentiates CI environments.

chore(reporter): report per-shard durations #38623

chore(reporter): report per-shard durations #38623

Conversation

Skn0tt commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Skn0tt Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Skn0tt Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 6, 2026

Test results for "MCP"

Uh oh!

github-actions bot commented Jan 6, 2026

Test results for "tests 1"

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Skn0tt commented Dec 19, 2025 •

edited

Loading

Skn0tt Jan 6, 2026 •

edited

Loading

Skn0tt Jan 5, 2026 •

edited

Loading