fix(notarize): auto-retrieve notarytool log on status: Invalid#1251
Conversation
When Apple's notarization returns 'status: Invalid', the rejection reason is only available via 'xcrun notarytool log <UUID>'. Previously this required running the command manually after CI failed, adding a debugging round-trip. Now the script captures the submission output, extracts the UUID, and automatically fetches the rejection log from Apple's server. The full JSON log (which lists every rejected binary with the specific error) is printed directly in CI output, making the next failure self-diagnosing.
Greptile SummaryThis PR improves the notarization failure UX by automatically fetching Apple's rejection log when Confidence Score: 5/5Safe to merge — the change correctly improves failure diagnostics without touching the success path, and the two previously-flagged P1 concerns (output buffering and POSIX All remaining findings are P2 or already in prior threads. The No files require special attention beyond the pre-existing
|
| Filename | Overview |
|---|---|
| scripts/notarize.sh | Adds UUID extraction and automatic notarytool log fetch on status: Invalid; tee-based output streaming correctly preserves real-time CI progress; ` |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[run_notarytool dist] --> B[store-credentials]
B --> C["notarytool submit --wait\n(tee to tmpfile)"]
C --> D[capture submission_exit + output]
D --> E{"output contains\n'status: Invalid'?"}
E -- No --> F[return submission_exit]
E -- Yes --> G["extract uuid\n(grep + awk)"]
G --> H{"uuid non-empty?"}
H -- Yes --> I["notarytool log uuid\n|| true"]
I --> J[print rejection log]
J --> K[return 1]
H -- No --> K
Reviews (2): Last reviewed commit: "fix(notarize): stream notarytool output ..." | Re-trigger Greptile
scripts/notarize.sh
Outdated
| submission_output=$(xcrun notarytool submit $dist --keychain-profile $keychain_profile --wait 2>&1) | ||
| submission_exit=$? | ||
| echo "$submission_output" |
There was a problem hiding this comment.
Output buffering breaks real-time CI progress streaming
$() buffers all stdout/stderr until the subprocess exits — nothing is emitted to the CI console while notarytool submit --wait is polling Apple's servers (which can take minutes). The PR description says "This does not change the success path at all," but this buffering affects every run, successful or not. A CI step that silently produces no output for several minutes can trigger job cancellation or alarm on-call.
A common workaround is to tee to a temp file:
tmpfile=$(mktemp)
xcrun notarytool submit "$dist" --keychain-profile $keychain_profile --wait 2>&1 | tee "$tmpfile"
submission_exit=${PIPESTATUS[0]}
submission_output=$(cat "$tmpfile")
rm -f "$tmpfile"This keeps live output streaming while still making the full text available for UUID extraction.
scripts/notarize.sh
Outdated
| # On failure, retrieve the detailed rejection log from Apple's server. | ||
| # This avoids having to run 'notarytool log' manually after the fact. | ||
| if echo "$submission_output" | grep -q "status: Invalid"; then | ||
| uuid=$(echo "$submission_output" | grep '^\s*id:' | head -1 | awk '{print $NF}') |
There was a problem hiding this comment.
\s is not POSIX BRE; use grep -E or a literal space
\s is supported by GNU grep and macOS grep as an extension, but it is not defined in POSIX Basic Regular Expressions. If this script ever runs on a strict POSIX shell (or a future macOS toolchain change), the pattern '^\s*id:' will match a literal \s instead of whitespace, silently returning an empty $uuid and skipping the log fetch.
| uuid=$(echo "$submission_output" | grep '^\s*id:' | head -1 | awk '{print $NF}') | |
| uuid=$(echo "$submission_output" | grep -E '^\s*id:' | head -1 | awk '{print $NF}') |
| if [ -n "$uuid" ]; then | ||
| echo "" | ||
| echo "=== Notarization rejected (status: Invalid) — fetching rejection log for $uuid ===" | ||
| xcrun notarytool log "$uuid" --keychain-profile $keychain_profile 2>&1 || true |
There was a problem hiding this comment.
Silent failure on log fetch gives no diagnostic hint
|| true suppresses the exit code, so if notarytool log fails (network error, expired credentials, rate-limit, etc.) the CI log will show only the empty section markers with no indication of what went wrong. Since this fetch is specifically meant to aid debugging, a small notice on failure is more useful:
| xcrun notarytool log "$uuid" --keychain-profile $keychain_profile 2>&1 || true | |
| xcrun notarytool log "$uuid" --keychain-profile $keychain_profile 2>&1 || echo "Warning: 'notarytool log' failed — log may be unavailable" |
Addresses Greptile P1: the $() subshell was buffering all notarytool submit --wait output until completion (~minutes), silencing CI progress. Fix: pipe through tee to a temp file so output streams in real-time, then read the temp file for pattern matching. Use PIPESTATUS[0] to capture the true exit status through the pipe. Also fixes non-POSIX \s → [[:space:]] in grep pattern.
|
Fixed the P1 output-buffering issue in bbece57:
The |
|
@greptileai review |
|
PR is ready for merge: all CI passes, Greptile reviewed at 5/5 confidence. Waiting only on a maintainer click. |
…ndles PyInstaller copies Python.framework contents as separate files rather than symlinks — Python, Versions/Current/Python, and Versions/3.9/Python are distinct inodes inside each watcher bundle. The previous fallback only signed $fw/$fw_name (Python.framework/Python), leaving the Versions/ copies unsigned. Apple notarization then rejected the submission with: - "The signature of the binary is invalid." - "The signature does not include a secure timestamp." for every unsigend Versions/ copy (6 errors across 3 watcher bundles). Fix: when codesign reports "bundle format is ambiguous" on a framework, use `find "$fw" -type f | xargs file | grep Mach-O` to enumerate ALL Mach-O files inside it and sign each one via the temp-path copy workaround. This ensures Python.framework/Python, Versions/Current/Python, and Versions/3.9/Python are all signed with --options runtime --timestamp. Diagnosed from notarytool rejection log captured by ActivityWatch#1251 in CI run 24193329397.
Problem
When Apple's notarization returns
status: Invalid, the detailed rejection reason is only available viaxcrun notarytool log <UUID>. The current script prints the UUID but doesn't retrieve the log, requiring a manual out-of-band step after CI fails.This has been the main debugging bottleneck for the dev/nightly release work: each failure requires Erik to re-run
notarytool loglocally, adding a full cycle (CI run → manual diagnosis → fix → CI run) to each iteration.Fix
Capture the
notarytool submit --waitoutput, extract the submission UUID onstatus: Invalid, and automatically fetch the rejection log from Apple's server. The full JSON response (which lists every rejected binary with the specific reason) is printed directly in CI output.This does not change the success path at all — only adds a log fetch + explicit
return 1on the failure path.Before / After
Before: CI fails, UUID shown, maintainer must run locally:
→ manual:
xcrun notarytool log bb24c56f-... --keychain-profile ...After: CI fails, rejection log printed inline:
Fixes debugging latency for #546.