fix(ci): make approve-test-queue tolerate races

[ko3n1g]

Claude summary

Why

The Approve Test Queue cron has been failing every 5 minutes today with KeyError: 'id' followed by exit 1. Example: https://github.com/NVIDIA-NeMo/NeMo/actions/runs/25600162235/job/75152606750

The actual failure mode wasn't a race — it was a wedged run. After a force-push on PR #15668, a CICD run on the discarded head was left in waiting status. GET /actions/runs/<id>/pending_deployments returned a single entry, but POST to approve it 422'd (No pending deployment requests to approve or reject). With MAX_CONCURRENCY=1 the cron picked the same orphan first every tick, crashed on a deployment['id'] lookup (no such key on /pending_deployments items), and never reached anything else.

What

Distinguish race from orphan, then remediate the orphan:

result = make_request(deployment_url, method="POST", data=status_data)
if result:
    total_workflows += 1
    continue

# POST failed. Re-GET to tell race vs orphan apart.
deployments_after = make_request(deployment_url) or []
if not deployments_after:
    # benign race — someone else approved between our GET and POST
    continue

# wedged: pending_deployments still lists it, but POST refuses
if cancel_run(workflow_id):
    print(f"Cancelled wedged run {workflow_id}")
else:
    had_unrecoverable = True

• New cancel_run() helper: POST /actions/runs/{id}/cancel + raise_for_status.
• If a wedged run can't be cancelled (e.g. token lacks scope), set had_unrecoverable and exit(1) after the loop, so other queue items still get processed.
• Initial GET that returns an empty list is also handled (skip silently — same race window, just earlier).
• KeyError: 'id' is gone; the only place we printed deployment['id'] referenced a key that does not exist on /pending_deployments items.

Verification

• The wedged run that triggered today's outage (25597959104) has been cancelled manually; the next 5-min cron tick will be the first reproduction-free one.
• After this PR lands, the same orphan scenario from a future force-push will be auto-cleared on the next tick.

[github-actions]

[🤖]: Hi @ko3n1g 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.