From 17fafefd6385220c903d8b80b6797b5d9376687b Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Thu, 18 Jun 2026 21:47:51 -0400 Subject: [PATCH 01/10] docs(monitoring): add Search monitoring section (the search target type) Co-Authored-By: Claude Opus 4.8 (1M context) --- features/monitoring.mdx | 195 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 194 insertions(+), 1 deletion(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index 1e0c1740..f1ffd69b 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -273,15 +273,208 @@ The check detail response includes `estimatedCredits`, `actualCredits`, summary +## Search monitoring + +Scrape and crawl monitors watch URLs you name. A **search** monitor watches the whole web. Instead of diffing pages you already know about, you give it a `goal`, and Firecrawl turns that goal into search queries, runs them on every check, and alerts on **new** results that match. It's discovery rather than diffing. + +Each check runs the same cycle: take the goal and its `queries`, apply a recency window, run the search, dedupe the results by fingerprint (canonical URL plus title plus snippet), let the optional AI judge decide which new results are meaningful to the goal, and alert through the same webhook and email channels as scrape and crawl monitors. Scheduling, goals, judging, and notifications all work exactly as described above. + + + Page and crawl monitors diff content on URLs you name; search monitors discover new results across the web. Same scheduling, judge, and notifications underneath. + + +### Search target + +A search target uses `type: "search"` and replaces `urls` with the queries to run and how to score the results: + +```json Search target +{ + "type": "search", + "queries": ["open source AI coding assistant launch"], + "searchWindow": "24h", + "maxResults": 10, + "depth": "deep", + "alertMode": "first_match", + "includeDomains": ["news.ycombinator.com"], + "excludeDomains": ["example-spam.com"] +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `type` | `"search"` | Selects the search target. | +| `queries` | `string[]` | Search queries to run on each check. | +| `searchWindow` | `"5m" \| "15m" \| "1h" \| "6h" \| "24h" \| "7d"` | Recency filter — only consider results published within this window. | +| `maxResults` | `number` | Results to evaluate per check, `1`–`50`. Defaults to `10`. | +| `depth` | `"raw" \| "standard" \| "deep"` | How thoroughly each result is evaluated. Defaults to `deep`. | +| `alertMode` | `"first_match" \| "every_new_result" \| "material_dev"` | How often a real-world event is allowed to alert. | +| `includeDomains` | `string[]` | Optional. Restrict results to these domains. | +| `excludeDomains` | `string[]` | Optional. Drop results from these domains. | + +The monitor-level `goal` and `judgeEnabled` apply just as in [Goals and judging](#goals-and-judging). Firecrawl expands the goal into queries when you don't supply your own, and the judge scores each new result against the goal. + +### Depth modes + +`depth` controls how much work each check does per result, trading cost for thoroughness: + +| Depth | What it does | Cost profile | +|-------|--------------|--------------| +| `raw` | Search results only, deduped. No AI. | Cheapest, fully deterministic. | +| `standard` | AI judges each result's title and snippet. | Search plus a judge call per result. | +| `deep` | Scrapes each result page, then runs full AI judging on the content. | Most thorough. Each scraped result is billed like a scrape. | + + + With `judgeEnabled: false`, a search monitor returns deduped results with no AI judging — equivalent to `raw`, just the new SERP hits. + + +### Statuses and dedup + +Search results are recorded as `new` or `already_seen`, rather than scrape and crawl's `same`, `changed`, and `removed`. A result alerts once when it first appears as `new`. If its title or snippet changes on re-index it fingerprints differently and can fire again. With `alertMode: "first_match"`, one real-world event spread across many articles alerts only once. + +### Create a search monitor + +A search monitor is created the same way as a scrape monitor — the only differences are the target (`type: "search"` with `queries`, `searchWindow`, `maxResults`, `depth`, and optional domain filters) and that there are no URLs: + + +```python Python +from firecrawl import Firecrawl + +firecrawl = Firecrawl( + # No API key needed to get started — add one for higher rate limits: + # api_key="fc-YOUR-API-KEY", +) + +monitor = firecrawl.create_monitor( + name="AI coding assistant launches", + schedule={"text": "every 30 minutes", "timezone": "UTC"}, + goal="Alert when a new open-source AI coding assistant is announced. Ignore funding rounds and unrelated AI news.", + targets=[ + { + "type": "search", + "queries": ["open source AI coding assistant launch"], + "searchWindow": "24h", + "maxResults": 10, + "depth": "deep", + } + ], + notification={ + "email": { + "enabled": True, + "recipients": ["alerts@example.com"], + "includeDiffs": True, + } + }, +) + +print(monitor.id) +``` + +```js Node +import Firecrawl from "@mendable/firecrawl-js"; + +const firecrawl = new Firecrawl({ + // No API key needed to get started — add one for higher rate limits: + // apiKey: "fc-YOUR-API-KEY", +}); + +const monitor = await firecrawl.createMonitor({ + name: "AI coding assistant launches", + schedule: { text: "every 30 minutes", timezone: "UTC" }, + goal: + "Alert when a new open-source AI coding assistant is announced. Ignore funding rounds and unrelated AI news.", + notification: { + email: { + enabled: true, + recipients: ["alerts@example.com"], + includeDiffs: true, + }, + }, + targets: [ + { + type: "search", + queries: ["open source AI coding assistant launch"], + searchWindow: "24h", + maxResults: 10, + depth: "deep", + }, + ], +}); + +console.log(monitor.id); +``` + +```bash cURL +curl -s -X POST "https://api.firecrawl.dev/v2/monitor" \ + -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "AI coding assistant launches", + "schedule": { + "text": "every 30 minutes", + "timezone": "UTC" + }, + "goal": "Alert when a new open-source AI coding assistant is announced. Ignore funding rounds and unrelated AI news.", + "notification": { + "email": { + "enabled": true, + "recipients": ["alerts@example.com"], + "includeDiffs": true + } + }, + "targets": [ + { + "type": "search", + "queries": ["open source AI coding assistant launch"], + "searchWindow": "24h", + "maxResults": 10, + "depth": "deep" + } + ] + }' +``` + + +When a new matching result is discovered, the `monitor.page` webhook reports it with status `new` and, when judging ran, a `judgment` describing why it matters: + +```json monitor.page +{ + "success": true, + "type": "monitor.page", + "id": "019df960-5f2a-75fb-a98b-bd2d32ca67d4", + "webhookId": "f1e2d3c4-0000-0000-0000-000000000000", + "data": [ + { + "monitorId": "019df960-06e7-7383-9d89-82c0113dc31a", + "checkId": "019df960-5f2a-75fb-a98b-bd2d32ca67d4", + "url": "https://news.ycombinator.com/item?id=40000000", + "status": "new", + "error": null, + "isMeaningful": true, + "judgment": { + "meaningful": true, + "confidence": "high", + "reason": "A new open-source AI coding assistant was announced, which matches the monitor goal." + } + } + ], + "metadata": { + "environment": "production" + } +} +``` + ## Pricing -Monitors don't introduce a separate per-monitor fee. Each check pays for the underlying scrape or crawl it performs, plus an optional credit per changed page when meaningful-change judging is enabled. +Monitors don't introduce a separate per-monitor fee. Each check pays for the underlying scrape, crawl, or search it performs, plus an optional credit per changed page when meaningful-change judging is enabled. | Component | Credits | |-----------|---------| | Scrape monitor | 1 credit per URL per check | | Crawl monitor | 1 credit per discovered page per check | +| Search monitor | ~2 credits per 10 results per check | +| Search monitor (`deep`) | Each scraped result is billed like a [scrape](/features/scrape), in addition to the search call | | Meaningful change enabled | 1 additional credit per changed page that the judge validates | +| Meaningful-change judging (search `standard` / `deep`) | LLM billed at cost for each result judged | | Format add-ons (JSON, PDF, question, enhanced mode, etc.) | Same as standalone [scrape](/features/scrape) | From 614cbb495d422391af0aa0258dfdefa9087d0570 Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Thu, 18 Jun 2026 22:03:22 -0400 Subject: [PATCH 02/10] Update monitoring.mdx --- features/monitoring.mdx | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index f1ffd69b..cabfbbfe 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -293,8 +293,6 @@ A search target uses `type: "search"` and replaces `urls` with the queries to ru "queries": ["open source AI coding assistant launch"], "searchWindow": "24h", "maxResults": 10, - "depth": "deep", - "alertMode": "first_match", "includeDomains": ["news.ycombinator.com"], "excludeDomains": ["example-spam.com"] } @@ -306,34 +304,22 @@ A search target uses `type: "search"` and replaces `urls` with the queries to ru | `queries` | `string[]` | Search queries to run on each check. | | `searchWindow` | `"5m" \| "15m" \| "1h" \| "6h" \| "24h" \| "7d"` | Recency filter — only consider results published within this window. | | `maxResults` | `number` | Results to evaluate per check, `1`–`50`. Defaults to `10`. | -| `depth` | `"raw" \| "standard" \| "deep"` | How thoroughly each result is evaluated. Defaults to `deep`. | -| `alertMode` | `"first_match" \| "every_new_result" \| "material_dev"` | How often a real-world event is allowed to alert. | | `includeDomains` | `string[]` | Optional. Restrict results to these domains. | | `excludeDomains` | `string[]` | Optional. Drop results from these domains. | The monitor-level `goal` and `judgeEnabled` apply just as in [Goals and judging](#goals-and-judging). Firecrawl expands the goal into queries when you don't supply your own, and the judge scores each new result against the goal. -### Depth modes +### Judging -`depth` controls how much work each check does per result, trading cost for thoroughness: - -| Depth | What it does | Cost profile | -|-------|--------------|--------------| -| `raw` | Search results only, deduped. No AI. | Cheapest, fully deterministic. | -| `standard` | AI judges each result's title and snippet. | Search plus a judge call per result. | -| `deep` | Scrapes each result page, then runs full AI judging on the content. | Most thorough. Each scraped result is billed like a scrape. | - - - With `judgeEnabled: false`, a search monitor returns deduped results with no AI judging — equivalent to `raw`, just the new SERP hits. - +How much work each check does per result is controlled by the monitor's `judgeEnabled` — the same flag described in [Goals and judging](#goals-and-judging). With judging on, Firecrawl scrapes each matching result and evaluates its content against the goal. With `judgeEnabled: false`, a search monitor returns the deduped search results with no AI judging — just the new SERP hits. ### Statuses and dedup -Search results are recorded as `new` or `already_seen`, rather than scrape and crawl's `same`, `changed`, and `removed`. A result alerts once when it first appears as `new`. If its title or snippet changes on re-index it fingerprints differently and can fire again. With `alertMode: "first_match"`, one real-world event spread across many articles alerts only once. +Search results are recorded as `new` or `already_seen`, rather than scrape and crawl's `same`, `changed`, and `removed`. A result alerts once when it first appears as `new`. If its title or snippet changes on re-index it fingerprints differently and can fire again. One real-world event spread across many articles alerts only once. ### Create a search monitor -A search monitor is created the same way as a scrape monitor — the only differences are the target (`type: "search"` with `queries`, `searchWindow`, `maxResults`, `depth`, and optional domain filters) and that there are no URLs: +A search monitor is created the same way as a scrape monitor — the only differences are the target (`type: "search"` with `queries`, `searchWindow`, `maxResults`, and optional domain filters) and that there are no URLs: ```python Python From 8455aa39a57375a852a6350c29053fb769dee2fb Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Thu, 18 Jun 2026 22:03:53 -0400 Subject: [PATCH 03/10] docs(monitoring): drop depth/alertMode from Search monitoring (not exposed in API) --- features/monitoring.mdx | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index cabfbbfe..79e487ba 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -340,7 +340,6 @@ monitor = firecrawl.create_monitor( "queries": ["open source AI coding assistant launch"], "searchWindow": "24h", "maxResults": 10, - "depth": "deep", } ], notification={ @@ -381,7 +380,6 @@ const monitor = await firecrawl.createMonitor({ queries: ["open source AI coding assistant launch"], searchWindow: "24h", maxResults: 10, - depth: "deep", }, ], }); @@ -412,8 +410,7 @@ curl -s -X POST "https://api.firecrawl.dev/v2/monitor" \ "type": "search", "queries": ["open source AI coding assistant launch"], "searchWindow": "24h", - "maxResults": 10, - "depth": "deep" + "maxResults": 10 } ] }' @@ -458,9 +455,9 @@ Monitors don't introduce a separate per-monitor fee. Each check pays for the und | Scrape monitor | 1 credit per URL per check | | Crawl monitor | 1 credit per discovered page per check | | Search monitor | ~2 credits per 10 results per check | -| Search monitor (`deep`) | Each scraped result is billed like a [scrape](/features/scrape), in addition to the search call | +| Search monitor (results scraped) | When AI judging is enabled, each scraped result is billed like a [scrape](/features/scrape), in addition to the search call | | Meaningful change enabled | 1 additional credit per changed page that the judge validates | -| Meaningful-change judging (search `standard` / `deep`) | LLM billed at cost for each result judged | +| Search monitor judging | When AI judging is enabled, the LLM is billed at cost for each result judged | | Format add-ons (JSON, PDF, question, enhanced mode, etc.) | Same as standalone [scrape](/features/scrape) | From 6323c2a75d1861a17a9de0c5f3f104066fe49fee Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Thu, 18 Jun 2026 22:15:05 -0400 Subject: [PATCH 04/10] =?UTF-8?q?docs(monitoring):=20flat=20search=20billi?= =?UTF-8?q?ng=20=E2=80=94=202=20per=2010=20results=20+=205=20per=20judged?= =?UTF-8?q?=20result?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- features/monitoring.mdx | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index 79e487ba..a99162af 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -311,7 +311,7 @@ The monitor-level `goal` and `judgeEnabled` apply just as in [Goals and judging] ### Judging -How much work each check does per result is controlled by the monitor's `judgeEnabled` — the same flag described in [Goals and judging](#goals-and-judging). With judging on, Firecrawl scrapes each matching result and evaluates its content against the goal. With `judgeEnabled: false`, a search monitor returns the deduped search results with no AI judging — just the new SERP hits. +How much work each check does per result is controlled by the monitor's `judgeEnabled` — the same flag described in [Goals and judging](#goals-and-judging). With judging on, Firecrawl scrapes each matching result and evaluates its content against the goal, billed at a flat 5 credits per judged result on top of the search call. With `judgeEnabled: false`, a search monitor returns the deduped search results with no AI judging — just the new SERP hits, and pays only the search-call credits (2 credits per 10 results). ### Statuses and dedup @@ -454,10 +454,9 @@ Monitors don't introduce a separate per-monitor fee. Each check pays for the und |-----------|---------| | Scrape monitor | 1 credit per URL per check | | Crawl monitor | 1 credit per discovered page per check | -| Search monitor | ~2 credits per 10 results per check | -| Search monitor (results scraped) | When AI judging is enabled, each scraped result is billed like a [scrape](/features/scrape), in addition to the search call | +| Search monitor | `ceil(results / 10) × 2` credits per check (≈ 2 credits per 10 results) | +| Search monitor judging | 5 credits per result judged, when AI judging is enabled (covers scraping and evaluating the result) | | Meaningful change enabled | 1 additional credit per changed page that the judge validates | -| Search monitor judging | When AI judging is enabled, the LLM is billed at cost for each result judged | | Format add-ons (JSON, PDF, question, enhanced mode, etc.) | Same as standalone [scrape](/features/scrape) | From 54b2e7ffa9bd16b47085cca809cabebc9e3df2cb Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Thu, 18 Jun 2026 22:22:29 -0400 Subject: [PATCH 05/10] docs(monitoring): plain-language search pricing (drop ceil formula) --- features/monitoring.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index a99162af..c23bd9cb 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -454,7 +454,7 @@ Monitors don't introduce a separate per-monitor fee. Each check pays for the und |-----------|---------| | Scrape monitor | 1 credit per URL per check | | Crawl monitor | 1 credit per discovered page per check | -| Search monitor | `ceil(results / 10) × 2` credits per check (≈ 2 credits per 10 results) | +| Search monitor | 2 credits per 10 results per check | | Search monitor judging | 5 credits per result judged, when AI judging is enabled (covers scraping and evaluating the result) | | Meaningful change enabled | 1 additional credit per changed page that the judge validates | | Format add-ons (JSON, PDF, question, enhanced mode, etc.) | Same as standalone [scrape](/features/scrape) | From 1d0a5a820074b735839b02e2bc8ef4dcfeff1898 Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Fri, 19 Jun 2026 11:33:41 -0400 Subject: [PATCH 06/10] docs(monitoring): clarify lazy goal re-eval + maxResults is a total cap --- features/monitoring.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index c23bd9cb..c23e2bf4 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -303,7 +303,7 @@ A search target uses `type: "search"` and replaces `urls` with the queries to ru | `type` | `"search"` | Selects the search target. | | `queries` | `string[]` | Search queries to run on each check. | | `searchWindow` | `"5m" \| "15m" \| "1h" \| "6h" \| "24h" \| "7d"` | Recency filter — only consider results published within this window. | -| `maxResults` | `number` | Results to evaluate per check, `1`–`50`. Defaults to `10`. | +| `maxResults` | `number` | Total results to evaluate per check, `1`–`50`. Defaults to `10`. This is a combined cap across all `queries` (results are merged and deduped first), not a per-query limit — so an individual query may contribute fewer results, or none, if other queries fill the cap first. | | `includeDomains` | `string[]` | Optional. Restrict results to these domains. | | `excludeDomains` | `string[]` | Optional. Drop results from these domains. | @@ -317,6 +317,8 @@ How much work each check does per result is controlled by the monitor's `judgeEn Search results are recorded as `new` or `already_seen`, rather than scrape and crawl's `same`, `changed`, and `removed`. A result alerts once when it first appears as `new`. If its title or snippet changes on re-index it fingerprints differently and can fire again. One real-world event spread across many articles alerts only once. +Editing the monitor's `goal` or `queries` bumps its `goalVersion`, which invalidates prior judge verdicts. Re-evaluation is lazy rather than a bulk re-judge: existing results are not all re-judged at once. Instead, each result is re-judged the next time it resurfaces in a check, picking up the new `goalVersion` then. Results that don't resurface keep their old verdict and `goalVersion` until they reappear. + ### Create a search monitor A search monitor is created the same way as a scrape monitor — the only differences are the target (`type: "search"` with `queries`, `searchWindow`, `maxResults`, and optional domain filters) and that there are no URLs: From b117646f8283c3301397f67c99af871a46583ec9 Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Wed, 24 Jun 2026 15:16:21 -0400 Subject: [PATCH 07/10] docs(monitoring): document search target in OpenAPI + fix status contract - Add the Search target as a third MonitorTarget oneOf member in the v2 OpenAPI spec (queries 1-12, searchWindow enum, maxResults 1-50, include/excludeDomains), and note goal is required when a search target is present. - Update the Targets section to list all three target types (was 'two'). - Fix the Statuses and dedup section: search pages use the standard same|new|changed|removed|error enum (alert->new, skipped->error, else->same); the granular searchStatus (alert|already_seen|watching|ignored|skipped) lives on page metadata. Document this on MonitorCheckPage.metadata too. - Mention web search in the intro. --- api-reference/v2-openapi.json | 76 +++++++++++++++++++++++++++++++++-- features/monitoring.mdx | 23 +++++++++-- 2 files changed, 93 insertions(+), 6 deletions(-) diff --git a/api-reference/v2-openapi.json b/api-reference/v2-openapi.json index 0d4b7263..51352152 100644 --- a/api-reference/v2-openapi.json +++ b/api-reference/v2-openapi.json @@ -5379,6 +5379,75 @@ "type", "url" ] + }, + { + "type": "object", + "title": "Search target", + "description": "Runs web search queries on each check and alerts on new results that match the monitor's goal. Requires a non-empty top-level `goal` on the monitor unless `judgeEnabled` is `false`.", + "properties": { + "id": { + "type": "string", + "format": "uuid", + "description": "Optional stable ID for this target. Generated if omitted." + }, + "type": { + "type": "string", + "enum": [ + "search" + ] + }, + "queries": { + "type": "array", + "minItems": 1, + "maxItems": 12, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 256 + }, + "description": "Search queries to run on each check (1-12)." + }, + "searchWindow": { + "type": "string", + "enum": [ + "5m", + "15m", + "1h", + "6h", + "24h", + "7d" + ], + "default": "24h", + "description": "Recency filter — only consider results published within this window." + }, + "maxResults": { + "type": "integer", + "minimum": 1, + "maximum": 50, + "default": 10, + "description": "Total results to evaluate per check, merged and deduped across all queries (a combined cap, not per-query)." + }, + "includeDomains": { + "type": "array", + "maxItems": 50, + "items": { + "type": "string" + }, + "description": "Optional. Restrict results to these domains." + }, + "excludeDomains": { + "type": "array", + "maxItems": 50, + "items": { + "type": "string" + }, + "description": "Optional. Drop results from these domains." + } + }, + "required": [ + "type", + "queries" + ] } ] }, @@ -5416,7 +5485,7 @@ "type": "string", "maxLength": 2000, "nullable": true, - "description": "Plain-language goal used to judge whether changed pages are meaningful. If provided and `judgeEnabled` is omitted, judging is enabled automatically." + "description": "Plain-language goal used to judge whether changed pages are meaningful. If provided and `judgeEnabled` is omitted, judging is enabled automatically. Required (non-empty) when any target is a `search` target, unless `judgeEnabled` is `false`." }, "judgeEnabled": { "type": "boolean", @@ -5463,7 +5532,7 @@ "type": "string", "maxLength": 2000, "nullable": true, - "description": "Plain-language goal used to judge whether changed pages are meaningful. If provided and `judgeEnabled` is omitted, judging is enabled automatically." + "description": "Plain-language goal used to judge whether changed pages are meaningful. If provided and `judgeEnabled` is omitted, judging is enabled automatically. Required (non-empty) when any target is a `search` target, unless `judgeEnabled` is `false`." }, "judgeEnabled": { "type": "boolean", @@ -5715,7 +5784,8 @@ }, "metadata": { "type": "object", - "nullable": true + "nullable": true, + "description": "Extra per-page metadata. For search monitors this includes `searchStatus`, the finer-grained search disposition behind the top-level `status`: `alert` (maps to `new`), `already_seen`, `watching`, `ignored` (all map to `same`), or `skipped` (maps to `error`)." }, "judgment": { "$ref": "#/components/schemas/MonitorPageJudgment" diff --git a/features/monitoring.mdx b/features/monitoring.mdx index c23e2bf4..c5ddabb2 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -35,7 +35,7 @@ import DiffMarkdown from "/snippets/v2/monitor/diff/markdown.mdx"; import DiffJson from "/snippets/v2/monitor/diff/json.mdx"; import DiffMixed from "/snippets/v2/monitor/diff/mixed.mdx"; -Firecrawl monitoring detects when content on a website changes and get notified by webhook or email. It runs recurring scrapes or crawls and compare each result against the last retained snapshot. Use monitors to track product pages, docs, blogs, changelogs, competitor sites, or any page where changes matter. +Firecrawl monitoring detects when content on a website changes and get notified by webhook or email. It runs recurring scrapes, crawls, or web searches and compares each result against the last retained snapshot. Use monitors to track product pages, docs, blogs, changelogs, competitor sites, or any page where changes matter — or to watch the whole web for new results that match a goal (see [Search monitoring](#search-monitoring)). Each check records page-level results as `same`, `new`, `changed`, `removed`, or `error`. You can receive a webhook as each monitored page finishes, a webhook for every completed check, email summaries when changes or errors happen, or any combination of those notifications. @@ -154,10 +154,11 @@ The minimum interval is 15 minutes. API responses always return the normalized c ## Targets -Monitors support two target types: +Monitors support three target types: - `scrape`: Runs one scrape per URL in `urls`. - `crawl`: Runs a full crawl for `url` on each check, then diffs all discovered pages. +- `search`: Runs web search `queries` on each check and alerts on new matching results. See [Search monitoring](#search-monitoring) for its fields and behavior. Each monitor accepts 1-50 targets. `retentionDays` defaults to `30` and can be set up to `365`. @@ -315,7 +316,23 @@ How much work each check does per result is controlled by the monitor's `judgeEn ### Statuses and dedup -Search results are recorded as `new` or `already_seen`, rather than scrape and crawl's `same`, `changed`, and `removed`. A result alerts once when it first appears as `new`. If its title or snippet changes on re-index it fingerprints differently and can fire again. One real-world event spread across many articles alerts only once. +Search results use the **same page-level `status` enum** as scrape and crawl monitors, so existing webhook and check-result consumers work unchanged. A search result maps to: + +- `new` — a result that matched the goal for the first time. This is what alerts. +- `same` — a result already seen on a previous check (no new alert). +- `error` — a result that could not be evaluated (for example, the scrape for judging was skipped). + +The finer-grained search disposition is exposed on each page's `metadata.searchStatus`, one of: + +| `searchStatus` | Page `status` | Meaning | +|----------------|---------------|---------| +| `alert` | `new` | New result the judge considers meaningful — fires a notification. | +| `already_seen` | `same` | Fingerprint matched a result from an earlier check. | +| `watching` | `same` | New result the judge isn't confident about yet; tracked but not alerted. | +| `ignored` | `same` | New result the judge scored as not meaningful to the goal. | +| `skipped` | `error` | Result could not be judged this check (for example, scrape failure or degraded judging). | + +A result alerts once when it first appears as `new`. If its title or snippet changes on re-index it fingerprints differently (canonical URL plus title plus snippet) and can fire again. One real-world event spread across many articles alerts only once. Editing the monitor's `goal` or `queries` bumps its `goalVersion`, which invalidates prior judge verdicts. Re-evaluation is lazy rather than a bulk re-judge: existing results are not all re-judged at once. Instead, each result is re-judged the next time it resurfaces in a check, picking up the new `goalVersion` then. Results that don't resurface keep their old verdict and `goalVersion` until they reappear. From 465740eb515ecb8d90ad6acb84789c81ca80a0ca Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Thu, 25 Jun 2026 16:17:49 -0400 Subject: [PATCH 08/10] Update monitoring.mdx --- features/monitoring.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index c5ddabb2..e05a2e7b 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -312,7 +312,7 @@ The monitor-level `goal` and `judgeEnabled` apply just as in [Goals and judging] ### Judging -How much work each check does per result is controlled by the monitor's `judgeEnabled` — the same flag described in [Goals and judging](#goals-and-judging). With judging on, Firecrawl scrapes each matching result and evaluates its content against the goal, billed at a flat 5 credits per judged result on top of the search call. With `judgeEnabled: false`, a search monitor returns the deduped search results with no AI judging — just the new SERP hits, and pays only the search-call credits (2 credits per 10 results). +How much work each check does per result is controlled by the monitor's `judgeEnabled` — the same flag described in [Goals and judging](#goals-and-judging). With judging on, Firecrawl scrapes each matching result and evaluates its content against the goal, billed at 1 credit per judged result on top of the search call. With `judgeEnabled: false`, a search monitor returns the deduped search results with no AI judging — just the new SERP hits, and pays only the search-call credits (2 credits per 10 results). ### Statuses and dedup @@ -474,7 +474,7 @@ Monitors don't introduce a separate per-monitor fee. Each check pays for the und | Scrape monitor | 1 credit per URL per check | | Crawl monitor | 1 credit per discovered page per check | | Search monitor | 2 credits per 10 results per check | -| Search monitor judging | 5 credits per result judged, when AI judging is enabled (covers scraping and evaluating the result) | +| Search monitor judging | 1 credit per result judged, when AI judging is enabled (covers scraping and evaluating the result) | | Meaningful change enabled | 1 additional credit per changed page that the judge validates | | Format add-ons (JSON, PDF, question, enhanced mode, etc.) | Same as standalone [scrape](/features/scrape) | From 5ae72d0c06fb92522eca1161373f6dada056d91d Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Thu, 25 Jun 2026 18:54:58 -0400 Subject: [PATCH 09/10] Update monitoring.mdx --- features/monitoring.mdx | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index e05a2e7b..9112d4f5 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -150,7 +150,7 @@ Supported natural language examples: - `daily at 5:30 PM` - `weekly` -The minimum interval is 15 minutes. API responses always return the normalized cron expression. For text schedules, `timezone` controls when phrases like `daily at 9am` run. Text schedules are spread by monitor ID before they are converted to cron so many monitors do not all run at the same instant. +The minimum interval is 5 minutes. API responses always return the normalized cron expression. For text schedules, `timezone` controls when phrases like `daily at 9am` run. Text schedules are spread by monitor ID before they are converted to cron so many monitors do not all run at the same instant. ## Targets @@ -294,8 +294,7 @@ A search target uses `type: "search"` and replaces `urls` with the queries to ru "queries": ["open source AI coding assistant launch"], "searchWindow": "24h", "maxResults": 10, - "includeDomains": ["news.ycombinator.com"], - "excludeDomains": ["example-spam.com"] + "includeDomains": ["news.ycombinator.com"] } ``` @@ -305,8 +304,8 @@ A search target uses `type: "search"` and replaces `urls` with the queries to ru | `queries` | `string[]` | Search queries to run on each check. | | `searchWindow` | `"5m" \| "15m" \| "1h" \| "6h" \| "24h" \| "7d"` | Recency filter — only consider results published within this window. | | `maxResults` | `number` | Total results to evaluate per check, `1`–`50`. Defaults to `10`. This is a combined cap across all `queries` (results are merged and deduped first), not a per-query limit — so an individual query may contribute fewer results, or none, if other queries fill the cap first. | -| `includeDomains` | `string[]` | Optional. Restrict results to these domains. | -| `excludeDomains` | `string[]` | Optional. Drop results from these domains. | +| `includeDomains` | `string[]` | Optional. Restrict results to these domains. Mutually exclusive with `excludeDomains`. | +| `excludeDomains` | `string[]` | Optional. Drop results from these domains. Mutually exclusive with `includeDomains`. | The monitor-level `goal` and `judgeEnabled` apply just as in [Goals and judging](#goals-and-judging). Firecrawl expands the goal into queries when you don't supply your own, and the judge scores each new result against the goal. From d2b6336bdd71218747570c72358c5bec0ac2b18b Mon Sep 17 00:00:00 2001 From: Developers Digest <124798203+developersdigest@users.noreply.github.com> Date: Fri, 26 Jun 2026 15:57:52 -0400 Subject: [PATCH 10/10] Update monitoring.mdx --- features/monitoring.mdx | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/features/monitoring.mdx b/features/monitoring.mdx index 9112d4f5..63965485 100644 --- a/features/monitoring.mdx +++ b/features/monitoring.mdx @@ -35,7 +35,7 @@ import DiffMarkdown from "/snippets/v2/monitor/diff/markdown.mdx"; import DiffJson from "/snippets/v2/monitor/diff/json.mdx"; import DiffMixed from "/snippets/v2/monitor/diff/mixed.mdx"; -Firecrawl monitoring detects when content on a website changes and get notified by webhook or email. It runs recurring scrapes, crawls, or web searches and compares each result against the last retained snapshot. Use monitors to track product pages, docs, blogs, changelogs, competitor sites, or any page where changes matter — or to watch the whole web for new results that match a goal (see [Search monitoring](#search-monitoring)). +Firecrawl monitoring detects when content on a website changes and get notified by webhook or email. It runs recurring scrapes, crawls, or web searches and compares each result against the last retained snapshot. Use monitors to track product pages, docs, blogs, changelogs, competitor sites, or any page where changes matter — or to watch the whole web for new results that match a goal (see [Web monitoring](#web-monitoring)). Each check records page-level results as `same`, `new`, `changed`, `removed`, or `error`. You can receive a webhook as each monitored page finishes, a webhook for every completed check, email summaries when changes or errors happen, or any combination of those notifications. @@ -158,7 +158,7 @@ Monitors support three target types: - `scrape`: Runs one scrape per URL in `urls`. - `crawl`: Runs a full crawl for `url` on each check, then diffs all discovered pages. -- `search`: Runs web search `queries` on each check and alerts on new matching results. See [Search monitoring](#search-monitoring) for its fields and behavior. +- `search`: Runs web search `queries` on each check and alerts on new matching results. See [Web monitoring](#web-monitoring) for its fields and behavior. Each monitor accepts 1-50 targets. `retentionDays` defaults to `30` and can be set up to `365`. @@ -274,14 +274,14 @@ The check detail response includes `estimatedCredits`, `actualCredits`, summary -## Search monitoring +## Web monitoring -Scrape and crawl monitors watch URLs you name. A **search** monitor watches the whole web. Instead of diffing pages you already know about, you give it a `goal`, and Firecrawl turns that goal into search queries, runs them on every check, and alerts on **new** results that match. It's discovery rather than diffing. +Scrape and crawl monitors watch URLs you name. A **web** monitor watches the whole web. Instead of diffing pages you already know about, you give it a `goal`, and Firecrawl turns that goal into search queries, runs them on every check, and alerts on **new** results that match. It's discovery rather than diffing. Each check runs the same cycle: take the goal and its `queries`, apply a recency window, run the search, dedupe the results by fingerprint (canonical URL plus title plus snippet), let the optional AI judge decide which new results are meaningful to the goal, and alert through the same webhook and email channels as scrape and crawl monitors. Scheduling, goals, judging, and notifications all work exactly as described above. - Page and crawl monitors diff content on URLs you name; search monitors discover new results across the web. Same scheduling, judge, and notifications underneath. + Page and crawl monitors diff content on URLs you name; web monitors discover new results across the web. Same scheduling, judge, and notifications underneath. ### Search target @@ -311,7 +311,7 @@ The monitor-level `goal` and `judgeEnabled` apply just as in [Goals and judging] ### Judging -How much work each check does per result is controlled by the monitor's `judgeEnabled` — the same flag described in [Goals and judging](#goals-and-judging). With judging on, Firecrawl scrapes each matching result and evaluates its content against the goal, billed at 1 credit per judged result on top of the search call. With `judgeEnabled: false`, a search monitor returns the deduped search results with no AI judging — just the new SERP hits, and pays only the search-call credits (2 credits per 10 results). +How much work each check does per result is controlled by the monitor's `judgeEnabled` — the same flag described in [Goals and judging](#goals-and-judging). With judging on, Firecrawl scrapes each matching result and evaluates its content against the goal, billed at 1 credit per judged result on top of the search call. With `judgeEnabled: false`, a web monitor returns the deduped search results with no AI judging — just the new SERP hits, and pays only the search-call credits (2 credits per 10 results). ### Statuses and dedup @@ -335,9 +335,9 @@ A result alerts once when it first appears as `new`. If its title or snippet cha Editing the monitor's `goal` or `queries` bumps its `goalVersion`, which invalidates prior judge verdicts. Re-evaluation is lazy rather than a bulk re-judge: existing results are not all re-judged at once. Instead, each result is re-judged the next time it resurfaces in a check, picking up the new `goalVersion` then. Results that don't resurface keep their old verdict and `goalVersion` until they reappear. -### Create a search monitor +### Create a web monitor -A search monitor is created the same way as a scrape monitor — the only differences are the target (`type: "search"` with `queries`, `searchWindow`, `maxResults`, and optional domain filters) and that there are no URLs: +A web monitor is created the same way as a scrape monitor — the only differences are the target (`type: "search"` with `queries`, `searchWindow`, `maxResults`, and optional domain filters) and that there are no URLs: ```python Python @@ -472,8 +472,8 @@ Monitors don't introduce a separate per-monitor fee. Each check pays for the und |-----------|---------| | Scrape monitor | 1 credit per URL per check | | Crawl monitor | 1 credit per discovered page per check | -| Search monitor | 2 credits per 10 results per check | -| Search monitor judging | 1 credit per result judged, when AI judging is enabled (covers scraping and evaluating the result) | +| Web monitor | 2 credits per 10 results per check | +| Web monitor judging | 1 credit per result judged, when AI judging is enabled (covers scraping and evaluating the result) | | Meaningful change enabled | 1 additional credit per changed page that the judge validates | | Format add-ons (JSON, PDF, question, enhanced mode, etc.) | Same as standalone [scrape](/features/scrape) |