fix: nuclei template freshness + debug-upload visibility#159
Merged
Conversation
A failed install/update left the agent running with stale or missing templates, surfacing as per-template "file not found" errors mid-scan for every cloud-sent path absent locally. Exit on failure instead of silently degrading.
Long-lived agents kept whatever templates they pulled at boot, so a template added after startup never landed and the cloud's path for it failed mid-scan. Refresh once per scan in processJetStreamScan before chunk processing. Serialized across concurrent scans and non-fatal: boot already guaranteed a usable set, so a transient failure proceeds on the existing templates.
Whether the cloud issues a presigned debug-upload URL was invisible until upload time, and only at Debug level. Surface it on the NATS credentials line at startup/refresh and before the shutdown upload so it's clear why a DB upload did or didn't happen.
ShubhamRasal
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related changes to the nuclei template lifecycle and debug-upload observability.
1. Make template install/update failure fatal at boot
ensureNucleiTemplateslogged and returned on failure, so the agent kept running with a stale or incompletenuclei-templatesdir — surfacing later as per-templateFailed to read template file ... no such file or directorymid-scan. Both failure paths (can't determine dir, update errors) nowos.Exit(1), matching theprereq.EnsureAll()check right above the call site. A scanner with no usable templates should fail loud at startup, not silently degrade.2. Refresh templates at the start of each vuln scan
Long-lived agents kept whatever templates they pulled at boot, so a template added upstream after startup never landed and the cloud's path for it failed mid-scan.
processJetStreamScannow refreshes once per scan, before chunk processing. Serialized across concurrent scans (ScanParallelism > 1) and non-fatal — boot already guaranteed a usable set, so a transient refresh failure proceeds on the existing templates rather than dropping the scan. Enumeration doesn't touch templates, so it's untouched.3. Log debug-upload URL availability
Whether the cloud issues a presigned debug-upload URL was invisible until upload time, and only at Debug level. Now surfaced as
debug_upload_url_available=true|falseon the NATS credentials line (startup + every refresh) and before the shutdown upload — so it's clear why a DB upload did or didn't happen.Known gap (not addressed here)
A present-but-incomplete templates dir whose
.templates-config.jsonversion stamp is current still slips through —UpdateIfOutdatedtrusts the stamp and never reconciles files against the stored checksum. This is mostly a dev-box hygiene case (prod containers cold-start fresh). The correct fix is upstream in the nuclei SDK (reconcile the existing.checksumeven when the version is current); tracking separately rather than adding brittle corruption heuristics here.