Skip to content

feat(doctor): nvcc's unreachable back-end, and the post-release verification that found it - #560

Merged
Sunrisepeak merged 4 commits into
mainfrom
docs/accel-post-release-verification
Sep 4, 2026
Merged

feat(doctor): nvcc's unreachable back-end, and the post-release verification that found it#560
Sunrisepeak merged 4 commits into
mainfrom
docs/accel-post-release-verification

Conversation

@Sunrisepeak

@Sunrisepeak Sunrisepeak commented Sep 4, 2026

Copy link
Copy Markdown
Member

Two things, in the order they happened.

1. A check: whether nvcc can reach its own back-end

Verifying the release inside a sandbox found the CUDA example failing where it
succeeds on the host, with:

sh: 1: cicc: not found

Single-variable isolation on the same nvcc command line:

environment result
sandbox, as-is sh: 1: cicc: not found
sandbox + PATH=/usr/lib/nvidia-cuda-toolkit/bin:$PATH produces the .o
sandbox + that PATH + NVVMIR_LIBRARY_DIR produces the .o

One PATH entry is the whole of the difference. nvcc runs cicc, cudafe++,
ptxas and fatbinary as bare names on a PATH it prepends itself from an
nvcc.profile beside its own binary; Debian-family packaging makes that
profile a symlink into /etc; a sandbox that replaces /etc removes it. The
toolkit is complete, nvcc is on PATH, crt/host_config.h is readable, and the
doctor's device section prints as usual — every obvious check passes, and the
message the user gets names neither nvcc nor the profile.

The plan is asked for rather than assumed. nvcc --dryrun prints the stages
and the PATH nvcc will use without compiling anything. parse_dryrun reads
that plan; the doctor resolves each bare name against the path nvcc states and
reports the first that does not:

$ mcpp self doctor
    Checking device toolkit
warning: nvcc cannot reach its own back-end: it invokes 'cicc' by name, and
         that name does not resolve on the search path it states.

This is the same principle as reading the host-compiler bound out of
crt/host_config.h instead of tabulating it: the vendor answers, mcpp does not
guess. A dryrun that yields no plan — there is no nvcc, or the output is not a
plan — produces no finding.

Measured on both sides

A check that is always silent and a check that is always loud look alike from
one run, so both were produced:

control occurrences of cannot reach its own back-end
working nvcc (profile readable) 0
nvcc copied to a directory holding no profile, placed first on PATH 1, naming cicc

It names cicc rather than the earlier cudafe++ because Debian ships
cudafe++, ptxas and fatbinary in /usr/bin and cicc only in the
toolkit's own directory — which is the stage the real failure names.

Coverage

No e2e. It would need nvcc, which CI does not have, and a # requires: nvcc
script skips and exits 0 on every shard, which is an assertion that never runs.
The coverage is five unit tests over two real --dryrun transcripts differing
in exactly that PATH line, plus the two-sided control above.

2. Section 15: what the released 2026.9.5.1 was measured to do

Section 14 was written at merge time, before the release. Section 15 records
the six criteria run against the published binary in a fresh subos:

# Criterion Result
A The artifact states its own version mcpp 2026.9.5.1
B The CN mirror is configurable Configured xlings mirror = CN
C self doctor reports the host/device compiler pairing gcc 13 exceeds the bound of 12 stated in /usr/include/crt/host_config.h
D The accel dimension refuses a mismatched artifact, naming the dimension, both sides, and a remedy on that axis the refusal contains accel, sm_90f, sm_86, --no-accel
E --no-accel admits it passes
F compat.cuda-runtime makes the driver reachable with no environment variable OK: driver reachable, cuInit=0x7a828447cbc0

Plus a control for F: removing the dependency and changing nothing else yields
CONTROL-OK: driver NOT reachable without the package, so F measures the
package rather than host leakage.

The first run of those criteria was a false green

Fitting the script into xlings subos use <name> --sandbox --cmd "$(...)" ran
into a quoting problem, and the workaround deleted the script's first line —
which was set -euo pipefail. Nothing was installed, every failure continued,
three sections asserted nothing, and the closing banner printed anyway. Two
rules follow: assertions carry their own || fail "..." rather than resting on
set -e, and a transport problem is fixed in the quoting, not by deleting a
line from the script.

One further finding

Inside the sandbox xlings install mcpp@2026.9.5.1 succeeds while the
bare-name shim still resolves to the previous release; xlings reports this
itself. Release artifacts are therefore verified through the store path. This
is the same directory as the shim-pruning defect in section 14.4
(openxlings/xlings#582).

Also

The comment added to the runtime.artifacts whitelist carried a warning emoji.
Emphasis is now carried by a capitalised phrase.

…i from a comment

Section 14 was written when the engine-side change was merged but not yet
released. Section 15 records what the released 2026.9.5.1 was measured to do:
six criteria run against the published binary in a fresh subos, and a control
establishing that the driver becomes reachable because of the index package
rather than because of host leakage.

It also records that the first run of those criteria was a false green. The
script was correct; the invocation removed its first line, which was
`set -euo pipefail`, in order to work around a quoting problem. Every
assertion then became advisory and the closing banner printed regardless. The
rule this yields is that assertions must not rest on `set -e` alone.

The comment added to the runtime.artifacts whitelist carried a warning emoji.
Emphasis is carried by a capitalised phrase instead.
Verifying the release inside a sandbox found the CUDA example failing where
it succeeds on the host. The difference is one line: nvcc runs cicc, cudafe++,
ptxas and fatbinary as bare names on a PATH it prepends itself from an
`nvcc.profile` beside its binary, Debian-family packaging makes that profile a
symlink into /etc, and a sandbox that replaces /etc removes it. nvcc then
keeps the ambient PATH and reports `sh: 1: cicc: not found`, which names
neither nvcc nor the profile. Nothing about the toolkit is missing, so every
obvious check passes.

The plan is asked for rather than assumed. `nvcc --dryrun` prints the stages
and the PATH nvcc will use without compiling anything; parse_dryrun reads that
plan, and the doctor resolves each bare name against the path nvcc states,
reporting the first that does not resolve. A dryrun that yields no plan --
there is no nvcc, or the output is not one -- produces no finding, on the same
principle as the host-compiler bound: a probe that reaches no answer must not
invent one.

Measured on both sides, because a check that is always silent and a check that
is always loud look alike from one run. With a working nvcc the warning does
not appear; with nvcc copied to a directory holding no profile it appears and
names `cicc` -- the same stage the real failure names, because Debian ships
cudafe++, ptxas and fatbinary in /usr/bin and cicc only in the toolkit's own
directory.

No e2e accompanies this. It would need nvcc, which CI does not have, and a
`# requires: nvcc` script skips and exits 0 on every shard. The coverage is
five unit tests over two real dryrun transcripts differing in exactly that
PATH line, plus the two-sided control recorded in the design's section 15.7.
@Sunrisepeak Sunrisepeak changed the title docs(accel): record the post-release verification of 2026.9.5.1 feat(doctor): nvcc's unreachable back-end, and the post-release verification that found it Sep 4, 2026
@Sunrisepeak
Sunrisepeak merged commit 4cd01a5 into main Sep 4, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants