Skip to content

fix(notarize): auto-retrieve notarytool log on status: Invalid#1251

Merged
ErikBjare merged 2 commits intoActivityWatch:masterfrom
TimeToBuildBob:fix/notarize-log-on-failure
Apr 9, 2026
Merged

fix(notarize): auto-retrieve notarytool log on status: Invalid#1251
ErikBjare merged 2 commits intoActivityWatch:masterfrom
TimeToBuildBob:fix/notarize-log-on-failure

Conversation

@TimeToBuildBob
Copy link
Copy Markdown
Contributor

Problem

When Apple's notarization returns status: Invalid, the detailed rejection reason is only available via xcrun notarytool log <UUID>. The current script prints the UUID but doesn't retrieve the log, requiring a manual out-of-band step after CI fails.

This has been the main debugging bottleneck for the dev/nightly release work: each failure requires Erik to re-run notarytool log locally, adding a full cycle (CI run → manual diagnosis → fix → CI run) to each iteration.

Fix

Capture the notarytool submit --wait output, extract the submission UUID on status: Invalid, and automatically fetch the rejection log from Apple's server. The full JSON response (which lists every rejected binary with the specific reason) is printed directly in CI output.

This does not change the success path at all — only adds a log fetch + explicit return 1 on the failure path.

Before / After

Before: CI fails, UUID shown, maintainer must run locally:

status: Invalid
id: bb24c56f-ce26-4441-96ef-416f34c5dcf4

→ manual: xcrun notarytool log bb24c56f-... --keychain-profile ...

After: CI fails, rejection log printed inline:

=== Notarization rejected (status: Invalid) — fetching rejection log for bb24c56f-... ===
{"status":"Invalid","statusSummary":"Archive contains critical validation errors",...}
=== End of rejection log ===

Fixes debugging latency for #546.

When Apple's notarization returns 'status: Invalid', the rejection
reason is only available via 'xcrun notarytool log <UUID>'. Previously
this required running the command manually after CI failed, adding a
debugging round-trip.

Now the script captures the submission output, extracts the UUID, and
automatically fetches the rejection log from Apple's server. The full
JSON log (which lists every rejected binary with the specific error) is
printed directly in CI output, making the next failure self-diagnosing.
@greptile-apps
Copy link
Copy Markdown

greptile-apps bot commented Apr 9, 2026

Greptile Summary

This PR improves the notarization failure UX by automatically fetching Apple's rejection log when notarytool submit --wait returns status: Invalid, replacing the previous manual out-of-band step. It also addresses two previously-flagged issues: output buffering is fixed via tee to a temp file, and the grep pattern now uses POSIX-compliant [[:space:]] instead of \s.

Confidence Score: 5/5

Safe to merge — the change correctly improves failure diagnostics without touching the success path, and the two previously-flagged P1 concerns (output buffering and POSIX \s) are now resolved.

All remaining findings are P2 or already in prior threads. The || true on the log fetch is an acknowledged but non-blocking style issue (already raised in a previous thread). No new logic bugs are introduced; the tee-based streaming and UUID extraction are sound.

No files require special attention beyond the pre-existing run_stapler unconditional call in the main script body (lines 93, 101), which is out of scope for this PR.

Vulnerabilities

No security concerns identified. The keychain profile credentials follow the existing pattern; no new secrets are introduced or logged.

Important Files Changed

Filename Overview
scripts/notarize.sh Adds UUID extraction and automatic notarytool log fetch on status: Invalid; tee-based output streaming correctly preserves real-time CI progress; `

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[run_notarytool dist] --> B[store-credentials]
    B --> C["notarytool submit --wait\n(tee to tmpfile)"]
    C --> D[capture submission_exit + output]
    D --> E{"output contains\n'status: Invalid'?"}
    E -- No --> F[return submission_exit]
    E -- Yes --> G["extract uuid\n(grep + awk)"]
    G --> H{"uuid non-empty?"}
    H -- Yes --> I["notarytool log uuid\n|| true"]
    I --> J[print rejection log]
    J --> K[return 1]
    H -- No --> K
Loading

Reviews (2): Last reviewed commit: "fix(notarize): stream notarytool output ..." | Re-trigger Greptile

Comment on lines +19 to +21
submission_output=$(xcrun notarytool submit $dist --keychain-profile $keychain_profile --wait 2>&1)
submission_exit=$?
echo "$submission_output"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Output buffering breaks real-time CI progress streaming

$() buffers all stdout/stderr until the subprocess exits — nothing is emitted to the CI console while notarytool submit --wait is polling Apple's servers (which can take minutes). The PR description says "This does not change the success path at all," but this buffering affects every run, successful or not. A CI step that silently produces no output for several minutes can trigger job cancellation or alarm on-call.

A common workaround is to tee to a temp file:

tmpfile=$(mktemp)
xcrun notarytool submit "$dist" --keychain-profile $keychain_profile --wait 2>&1 | tee "$tmpfile"
submission_exit=${PIPESTATUS[0]}
submission_output=$(cat "$tmpfile")
rm -f "$tmpfile"

This keeps live output streaming while still making the full text available for UUID extraction.

# On failure, retrieve the detailed rejection log from Apple's server.
# This avoids having to run 'notarytool log' manually after the fact.
if echo "$submission_output" | grep -q "status: Invalid"; then
uuid=$(echo "$submission_output" | grep '^\s*id:' | head -1 | awk '{print $NF}')
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 \s is not POSIX BRE; use grep -E or a literal space

\s is supported by GNU grep and macOS grep as an extension, but it is not defined in POSIX Basic Regular Expressions. If this script ever runs on a strict POSIX shell (or a future macOS toolchain change), the pattern '^\s*id:' will match a literal \s instead of whitespace, silently returning an empty $uuid and skipping the log fetch.

Suggested change
uuid=$(echo "$submission_output" | grep '^\s*id:' | head -1 | awk '{print $NF}')
uuid=$(echo "$submission_output" | grep -E '^\s*id:' | head -1 | awk '{print $NF}')

if [ -n "$uuid" ]; then
echo ""
echo "=== Notarization rejected (status: Invalid) — fetching rejection log for $uuid ==="
xcrun notarytool log "$uuid" --keychain-profile $keychain_profile 2>&1 || true
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Silent failure on log fetch gives no diagnostic hint

|| true suppresses the exit code, so if notarytool log fails (network error, expired credentials, rate-limit, etc.) the CI log will show only the empty section markers with no indication of what went wrong. Since this fetch is specifically meant to aid debugging, a small notice on failure is more useful:

Suggested change
xcrun notarytool log "$uuid" --keychain-profile $keychain_profile 2>&1 || true
xcrun notarytool log "$uuid" --keychain-profile $keychain_profile 2>&1 || echo "Warning: 'notarytool log' failed — log may be unavailable"

Addresses Greptile P1: the $() subshell was buffering all notarytool
submit --wait output until completion (~minutes), silencing CI progress.

Fix: pipe through tee to a temp file so output streams in real-time,
then read the temp file for pattern matching. Use PIPESTATUS[0] to
capture the true exit status through the pipe.

Also fixes non-POSIX \s → [[:space:]] in grep pattern.
@TimeToBuildBob
Copy link
Copy Markdown
Contributor Author

Fixed the P1 output-buffering issue in bbece57:

  • Replaced $() subshell capture with tee to a temp file, so notarytool submit --wait streams output in real-time while still allowing post-hoc pattern matching
  • Used PIPESTATUS[0] to capture the true exit status through the pipe
  • Fixed \s[[:space:]] in the grep pattern (P2)

The || true on the notarytool log fetch is intentional — if the log fetch fails we still want to return 1 for the invalid submission status, which happens unconditionally on the return 1 two lines later.

@TimeToBuildBob
Copy link
Copy Markdown
Contributor Author

@greptileai review

@TimeToBuildBob
Copy link
Copy Markdown
Contributor Author

PR is ready for merge: all CI passes, Greptile reviewed at 5/5 confidence. Waiting only on a maintainer click.

@ErikBjare ErikBjare merged commit c94ac70 into ActivityWatch:master Apr 9, 2026
17 checks passed
TimeToBuildBob added a commit to TimeToBuildBob/activitywatch that referenced this pull request Apr 9, 2026
…ndles

PyInstaller copies Python.framework contents as separate files rather than
symlinks — Python, Versions/Current/Python, and Versions/3.9/Python are
distinct inodes inside each watcher bundle. The previous fallback only signed
$fw/$fw_name (Python.framework/Python), leaving the Versions/ copies unsigned.
Apple notarization then rejected the submission with:
  - "The signature of the binary is invalid."
  - "The signature does not include a secure timestamp."
for every unsigend Versions/ copy (6 errors across 3 watcher bundles).

Fix: when codesign reports "bundle format is ambiguous" on a framework, use
`find "$fw" -type f | xargs file | grep Mach-O` to enumerate ALL Mach-O files
inside it and sign each one via the temp-path copy workaround. This ensures
Python.framework/Python, Versions/Current/Python, and Versions/3.9/Python
are all signed with --options runtime --timestamp.

Diagnosed from notarytool rejection log captured by ActivityWatch#1251 in CI run 24193329397.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants