Quick-watch: Catch early bulk operation errors without `--watch` #6687

ericlee878 · 2025-12-04T20:10:18Z

Resolves: https://github.com/orgs/shop/projects/208/views/34?filterQuery=&pane=issue&itemId=140129246&issue=shop%7Cissues-api-foundations%7C1123

Inspired by: https://github.com/shop/issues-api-foundations/issues/1123#issuecomment-3603641001

Background

When users run bulk operations with --variables or --variable-file, invalid JSON or malformed inputs aren't validated upfront. Without the --watch flag, users would:

Submit a potentially invalid job
Walk away thinking it's running
Come back hours later to find errors in the result file

This is a frustrating experience, especially for long-running bulk operations.

We considered validating JSON on the client before sending to the API, but:

There are too many ways for Core to reject input (incorrect query, incorrect variables, incorrect data format, etc.). There could also be errors we cannot catch from client side, making it possible for there to sometimes be errors caught on the client side and sometimes given in the output file (potentially causing confusion).
It's impractical to catch every type of error at a validation stage.
Full validation of large imports would hurt performance.

The solution: Quick-watch
Instead of duplicating validation logic, we leverage the fact that errors tend to surface in the first 2-3 seconds of a bulk operation. By polling briefly after creation, we can catch most "obvious issues" (like malformed JSON) and present them immediately—without requiring users to run a second command or wait for hours.

This approach:

Catches early errors without duplicating Core's validation logic
Doesn't significantly impact performance (only 3 seconds of polling)
Works for the common case where validation errors appear quickly

Implementation Details

Detecting mutation-level errors in results
When a bulk operation completes (status: COMPLETED), we check the downloaded results for userErrors using a simple string check: results.includes('"userErrors":[{').

This catches cases where the operation "succeeds" but individual mutations failed (e.g., malformed JSON variables). Instead of showing a misleading "Bulk operation succeeded" message, we show:

renderWarning with "Bulk operation completed with errors."
If --output-file was provided: includes the file path so users know where to check

Why .includes() instead of ?.userErrors?.length: like in line 62 of execute-bulk-operation.ts?
The result file is a raw JSONL string, not a JavaScript object. Parsing every line would be expensive for large files — a simple string search is O(n) with no allocations. Checking for "userErrors":[{ only matches non-empty arrays ("userErrors":[] won't match).

Flag behavior
Quick watch only happens when the --watch flag isn't given. This decision is explained here.

Polling configuration

QUICK_WATCH_TIMEOUT_MS = 3000 - Total time to poll
QUICK_WATCH_POLL_INTERVAL_MS = 300 - Poll every 300ms (~10 polls max)

No null checks in quickWatchBulkOperation:
We use bulkOperation! (non-null assertion) because:

executeBulkOperation already checks userErrors before calling quick-watch
If bulkOperation is null, userErrors would exist and we'd return early

Tophatting

For catching errors early:

pnpm shopify:run app bulk execute \
  --path <PATH_TO_TEST_APP> \
  --output-file ./results.jsonl \
  -q 'mutation productUpdate($input: ProductInput!) { productUpdate(input: $input) { product { id tags } userErrors { field message } } }' \
  -v '{"input": invalid json}'

This is what my results.jsonl looked like:

{"data":{"productUpdate":{"userErrors":[{"message":"unexpected character: 'invalid' at line 1 column 11"}]}},"__lineNumber":0}

For super quickly finished bulk operations that successfully finish within the quick-watch:

pnpm shopify:run app bulk execute \
  --path <PATH_TO_TEST_APP>\
  -q '{ products(first: 1) { edges { node { id title } } } }'

For bulk operations that do not finish in quick-watch:

Create a variable file:

cat > /tmp/bulk-products.jsonl << 'EOF'
{"input":{"title":"Test Product 1","descriptionHtml":"<p>This is test product 1</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 2","descriptionHtml":"<p>This is test product 2</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 3","descriptionHtml":"<p>This is test product 3</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 4","descriptionHtml":"<p>This is test product 4</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 5","descriptionHtml":"<p>This is test product 5</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 6","descriptionHtml":"<p>This is test product 6</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 7","descriptionHtml":"<p>This is test product 7</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 8","descriptionHtml":"<p>This is test product 8</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 9","descriptionHtml":"<p>This is test product 9</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 10","descriptionHtml":"<p>This is test product 10</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
EOF

Run the long bulk mutation:

pnpm shopify:run app bulk execute \
  --path ./coffee-and-more \
  --variable-file /tmp/bulk-products.jsonl \
  -q 'mutation productCreate($input: ProductInput!) {
    productCreate(input: $input) {
      product {
        id
        title
        metafields(first: 10) {
          edges {
            node {
              id
              namespace
              key
              value
            }
          }
        }
      }
      userErrors {
        field
        message
      }
    }
  }'

ericlee878 · 2025-12-04T20:10:38Z

Add user display during short poll #6700
Quick-watch: Catch early bulk operation errors without --watch #6687 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2025-12-05T00:24:14Z

We detected some changes at packages/*/src and there are no updates in the .changeset.
If the changes are user-facing, run pnpm changeset add to track your changes and include them in the next release CHANGELOG.

Caution

DO NOT create changesets for features which you do not wish to be included in the public changelog of the next CLI release.

github-actions · 2025-12-05T00:30:13Z

Coverage report

St.^❔	Category	Percentage	Covered / Total
🟡	Statements	79.5% (+0.27% 🔼)	14167/17821
🟡	Branches	73.57% (+0.47% 🔼)	6955/9453
🟡	Functions	79.64% (+0.27% 🔼)	3634/4563
🟡	Lines	79.87% (+0.29% 🔼)	13393/16769

Show new covered files 🐣

St.^❔	File	Statements	Branches	Functions	Lines
🟢	`...` / admin-as-app.ts	100%	100%	100%	100%
🟢	`...` / metafield_definitions.ts	100%	100%	100%	100%
🟢	`...` / metaobject_definitions.ts	100%	100%	100%	100%
🟢	`...` / bulk-operation-run-mutation.ts	100%	100%	100%	100%
🟢	`...` / bulk-operation-run-query.ts	100%	100%	100%	100%
🟢	`...` / get-bulk-operation-by-id.ts	100%	100%	100%	100%
🟢	`...` / list-bulk-operations.ts	100%	100%	100%	100%
🟢	`...` / staged-uploads-create.ts	100%	100%	100%	100%
🔴	`...` / execute.ts	0%	0%	0%	0%
🔴	`...` / status.ts	0%	0%	0%	0%
🔴	`...` / pull.ts	0%	100%	0%	0%
🟢	`...` / execute-operation.ts	92.86%	80%	100%	92.31%
🔴	`...` / pull.ts	0%	0%	0%	0%
🟢	`...` / bulk-operation-status.ts	96.61%	92.11%	100%	100%
🟢	`...` / download-bulk-operation-results.ts	100%	100%	100%	100%
🟢	`...` / execute-bulk-operation.ts	92.75%	87.23%	100%	93.94%
🟢	`...` / format-bulk-operation-status.ts	100%	100%	100%	100%
🟢	`...` / run-mutation.ts	100%	100%	100%	100%
🟢	`...` / run-query.ts	100%	100%	100%	100%
🟡	`...` / stage-file.ts	73.53%	62.5%	85.71%	72.73%
🟢	`...` / watch-bulk-operation.ts	100%	100%	100%	100%
🟢	`...` / declarative-definitions.ts	98.54%	93.18%	100%	98.51%
🟢	`...` / common.ts	96.15%	93.33%	100%	95.45%
🟢	`...` / execute-command-helpers.ts	100%	100%	100%	100%
🔴	`...` / promiseWithResolvers.ts	33.33%	50%	50%	33.33%

Show files with reduced coverage 🔻

St.^❔	File	Statements	Branches	Functions	Lines
🔴	`...` / execute.ts	0%	0% (-100% 🔻)	0%	0%
🟢	`...` / extension-instance.ts	84.8% (+0.23% 🔼)	77.6% (-0.91% 🔻)	92.06% (+0.13% 🔼)	85.11% (+0.24% 🔼)
🟡	`...` / specification.ts	69.09%	75.61% (+2.44% 🔼)	76.47% (-1.31% 🔻)	68.75%
🟢	`...` / ui_extension.ts	85.38% (-9.44% 🔻)	72.34% (-8.91% 🔻)	84% (-16% 🔻)	88% (-8.46% 🔻)
🟢	`...` / developer-platform-client.ts	84.62% (-1.5% 🔻)	71.43% (+0.84% 🔼)	81.82% (+1.82% 🔼)	93.75% (+0.42% 🔼)
🟢	`...` / api.ts	87.07% (-0.43% 🔻)	76.71% (-0.1% 🔻)	100%	86.49% (-0.43% 🔻)
🟢	`...` / SingleTask.tsx	84.21% (-15.79% 🔻)	50% (-50% 🔻)	80% (-20% 🔻)	84.21% (-15.79% 🔻)
🔴	`...` / environment.ts	35% (-5% 🔻)	41.18%	40% (-10% 🔻)	36.84% (-5.26% 🔻)
🔴	`...` / ui.tsx	50.82% (-0.79% 🔻)	42.86% (-5.53% 🔻)	54.55% (+1.42% 🔼)	50% (-0.82% 🔻)
🟢	`...` / console.ts	81.82% (+15.15% 🔼)	75% (-25% 🔻)	100% (+33.33% 🔼)	81.82% (+15.15% 🔼)
🔴	`...` / dev.ts	14.29% (+0.95% 🔼)	3.13% (+0.18% 🔼)	50% (-7.14% 🔻)	14.29% (+0.95% 🔼)
🟢	`...` / init.ts	88% (-0.89% 🔻)	71.43% (+4.76% 🔼)	86.67% (+4.85% 🔼)	88% (-0.89% 🔻)
🟢	`...` / storefront-renderer.ts	90.2% (-0.54% 🔻)	78.95%	81.82% (-1.52% 🔻)	90.2% (-0.54% 🔻)
🟡	`...` / theme-polling.ts	67.12% (-0.93% 🔻)	68.75%	78.57%	66.67% (-0.98% 🔻)

Test suite run success

3544 tests passing in 1414 suites.

Report generated by 🧪jest coverage report action from 910f6b0

packages/app/src/cli/services/bulk-operations/watch-bulk-operation.ts

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts

jordanverasamy

I wonder how much the extra polling (seems like right now we poll 10 times in a 3 second period) actually helps. Would be simple and nice to just e.g. wait 1 second and fetch once, instead of doing all of that work, and I bet it'd cover most of the same common cases anyways. What are your thoughts on the tradeoffs there?

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts

ericlee878 · 2025-12-05T22:13:06Z

Are there any impactful consequences with extra polling? Does it slow anything down or put too much pressure on the platform or something? Tbh, I don't think I know enough about the speed of the API, the user/agent's needs, or the impact of the extra polling to decide this. Happy to refactor either way though!

packages/app/src/cli/services/bulk-operations/watch-bulk-operation.ts

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts

packages/app/src/cli/services/bulk-operations/watch-bulk-operation.ts

jordanverasamy · 2025-12-09T00:27:43Z

Are there any impactful consequences with extra polling? Does it slow anything down or put too much pressure on the platform or something? Tbh, I don't think I know enough about the speed of the API, the user/agent's needs, or the impact of the extra polling to decide this. Happy to refactor either way though!

Not really, it's not that many requests in the grand scheme of things. IMO the real cost there is code complexity and developer brainpower, not platform pressure. But that said I'm happy with keeping it and like where we landed, so don't worry :)

gonzaloriestra

Nice solution! It works well.

But I agree with @jordanverasamy that the current polling is maybe excessive, even if it won't cause problem. I think a 1s polling interval should be enough and consistent with #6695

ericlee878 changed the title ~~quick-watch-for-cli~~ Quick-watch: Catch early bulk operation errors without --watch Dec 4, 2025

ericlee878 changed the title ~~Quick-watch: Catch early bulk operation errors without --watch~~ Quick-watch: Catch early bulk operation errors without --watch Dec 4, 2025

ericlee878 marked this pull request as ready for review December 4, 2025 20:19

ericlee878 requested a review from a team as a code owner December 4, 2025 20:19

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch 3 times, most recently from abd87e4 to fe2d0bb Compare December 5, 2025 00:24

ericlee878 requested a review from jordanverasamy December 5, 2025 00:24

allancalix reviewed Dec 5, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/watch-bulk-operation.ts Outdated Show resolved Hide resolved

allancalix reviewed Dec 5, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/watch-bulk-operation.ts Outdated Show resolved Hide resolved

allancalix reviewed Dec 5, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts Outdated Show resolved Hide resolved

ericlee878 requested a review from allancalix December 5, 2025 00:49

jordanverasamy reviewed Dec 5, 2025

View reviewed changes

ericlee878 commented Dec 5, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts Show resolved Hide resolved

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch 2 times, most recently from e27a952 to cf26356 Compare December 5, 2025 21:30

ericlee878 requested a review from jordanverasamy December 5, 2025 22:05

jordanverasamy reviewed Dec 5, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/watch-bulk-operation.ts Outdated Show resolved Hide resolved

jordanverasamy reviewed Dec 5, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts Outdated Show resolved Hide resolved

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch 2 times, most recently from fa74da9 to b0b505d Compare December 8, 2025 22:42

jordanverasamy reviewed Dec 8, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts Show resolved Hide resolved

jordanverasamy reviewed Dec 8, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/execute-bulk-operation.ts Outdated Show resolved Hide resolved

jordanverasamy reviewed Dec 8, 2025

View reviewed changes

packages/app/src/cli/services/bulk-operations/watch-bulk-operation.ts Outdated Show resolved Hide resolved

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch 2 times, most recently from 5f8926e to d0fe8d1 Compare December 9, 2025 00:12

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch from d0fe8d1 to 84d9954 Compare December 9, 2025 00:21

jordanverasamy approved these changes Dec 9, 2025

View reviewed changes

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch from 84d9954 to fb8280d Compare December 9, 2025 00:50

ericlee878 requested a review from gonzaloriestra December 9, 2025 17:57

ericlee878 mentioned this pull request Dec 10, 2025

Add user display during short poll #6700

Merged

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch 3 times, most recently from 65302ac to 4b4ce44 Compare December 11, 2025 19:19

gonzaloriestra approved these changes Dec 12, 2025

View reviewed changes

quick-watch-for-cli

910f6b0

ericlee878 force-pushed the 12-04-cli-quick-watch-feature branch from 4b4ce44 to 910f6b0 Compare December 12, 2025 18:30

ericlee878 added this pull request to the merge queue Dec 12, 2025

Merged via the queue into main with commit 8f8b496 Dec 12, 2025
25 checks passed

ericlee878 deleted the 12-04-cli-quick-watch-feature branch December 12, 2025 18:54

ericlee878 added the #gsd: 47715 label Dec 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Quick-watch: Catch early bulk operation errors without `--watch` #6687

Quick-watch: Catch early bulk operation errors without `--watch` #6687

ericlee878 commented Dec 4, 2025 •

edited

Loading

Uh oh!

ericlee878 commented Dec 4, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 5, 2025

Uh oh!

github-actions bot commented Dec 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordanverasamy left a comment

Uh oh!

Uh oh!

ericlee878 commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordanverasamy commented Dec 9, 2025

Uh oh!

gonzaloriestra left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Quick-watch: Catch early bulk operation errors without --watch #6687

Quick-watch: Catch early bulk operation errors without --watch #6687

Conversation

ericlee878 commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Implementation Details

Tophatting

Uh oh!

ericlee878 commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 5, 2025

Uh oh!

github-actions bot commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Test suite run success

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordanverasamy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ericlee878 commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordanverasamy commented Dec 9, 2025

Uh oh!

gonzaloriestra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Quick-watch: Catch early bulk operation errors without `--watch` #6687

Quick-watch: Catch early bulk operation errors without `--watch` #6687

ericlee878 commented Dec 4, 2025 •

edited

Loading

ericlee878 commented Dec 4, 2025 •

edited

Loading

github-actions bot commented Dec 5, 2025 •

edited

Loading