From 25d88b6a0fe1dd570a7d37226badae274f07df19 Mon Sep 17 00:00:00 2001 From: Daniel Vataj <153308333+sent-dm@users.noreply.github.com> Date: Sun, 9 Aug 2026 17:26:13 -0400 Subject: [PATCH] fix(analyzer): report channel-aware MDR outcomes --- .../messaging-performance-analyzer/SKILL.md | 15 +- .../references/mdr-status-codes.md | 11 +- .../performance-diagnosis-playbook.md | 14 +- .../scripts/analyze_mdr_funnel.py | 586 ++++++++++++----- .../scripts/fixtures/bad.json | 612 ++++-------------- .../scripts/fixtures/good.json | 538 +++------------ evals/messaging-performance-analyzer.yaml | 6 + .../messaging-performance-analyzer/SKILL.md | 15 +- .../references/mdr-status-codes.md | 11 +- .../performance-diagnosis-playbook.md | 14 +- .../scripts/analyze_mdr_funnel.py | 586 ++++++++++++----- .../scripts/fixtures/bad.json | 612 ++++-------------- .../scripts/fixtures/good.json | 538 +++------------ .../messaging-performance-analyzer/SKILL.md | 15 +- .../references/mdr-status-codes.md | 11 +- .../performance-diagnosis-playbook.md | 14 +- .../scripts/analyze_mdr_funnel.py | 586 ++++++++++++----- .../scripts/fixtures/bad.json | 612 ++++-------------- .../scripts/fixtures/good.json | 538 +++------------ scripts/test_fixtures.py | 165 ++++- .../messaging-performance-analyzer/SKILL.md | 15 +- .../references/mdr-status-codes.md | 11 +- .../performance-diagnosis-playbook.md | 14 +- .../scripts/analyze_mdr_funnel.py | 586 ++++++++++++----- .../scripts/fixtures/bad.json | 612 ++++-------------- .../scripts/fixtures/good.json | 538 +++------------ 26 files changed, 2666 insertions(+), 4609 deletions(-) diff --git a/claude-plugins/sent/skills/messaging-performance-analyzer/SKILL.md b/claude-plugins/sent/skills/messaging-performance-analyzer/SKILL.md index c7b0d4d..3898887 100644 --- a/claude-plugins/sent/skills/messaging-performance-analyzer/SKILL.md +++ b/claude-plugins/sent/skills/messaging-performance-analyzer/SKILL.md @@ -48,13 +48,15 @@ Capture these dimensions before calculating anything: profile or sender identity ### 2. Build cohorts from Sent message IDs -Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once at its latest status, then add recipient-level or campaign-level rollups only after deduplication. +Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once in a terminal-outcome rollup, then add recipient-level or campaign-level rollups only after deduplication. + +Distinguish an activity history from a latest-status snapshot. A history can prove the transitions it contains. A snapshot such as `status=DELIVERED` proves only the observed current outcome; it does not prove that the export also observed `QUEUED`, `ROUTED`, or `SENT`. Report unavailable transition denominators as `N/A`, not zero, and never synthesize missing transitions. Do not use provider IDs such as WhatsApp `wamid`, SMS carrier IDs, or RCS message IDs as the primary join key unless the exported evidence lacks Sent IDs. Provider IDs are useful for escalation, but the Sent API and dashboard track status by Sent message ID. ### 3. Normalize lifecycle stages to Sent’s documented statuses -Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, `DELIVERED`, and `READ` for WhatsApp and RCS. Keep failed and error states in a separate terminal bucket using the exact status/error fields present in the evidence. +Use Sent’s documented delivery lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, and `DELIVERED`. Treat `READ` as a separate engagement measure for WhatsApp and RCS, never as an SMS delivery requirement. Keep terminal failures, deferred/in-flight messages, inbound `RECEIVED` messages, and malformed/unknown records in separate buckets using only fields present in the evidence. | Stage | Interpretation | Common diagnostic question | |---|---|---| @@ -62,7 +64,7 @@ Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, | `ROUTED` | Sent selected a channel/provider path. | Did routing choose the expected channel or fallback path? | | `SENT` | The message left Sent/provider processing toward the destination network. | Are provider accepts high but downstream delivery low? | | `DELIVERED` | Delivery was confirmed where supported. | Did the destination network confirm receipt? | -| `READ` | WhatsApp/RCS read receipt was observed where available. | Did users open the message after delivery? | +| `READ` | WhatsApp/RCS engagement receipt was observed where available. | Did users open the message after delivery? | | Error/failure | A terminal or recoverable error occurred. | Is the root cause compliance, payload, throughput, opt-out, or provider outage? | ### 4. Check webhook health before diagnosing delivery @@ -83,7 +85,7 @@ SMS, WhatsApp, and RCS fail differently. Do not average them together unless the ### 6. Quantify impact before recommending fixes -Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. +Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Reconcile the global totals with every channel × direction group, retaining explicit `unknown` groups instead of silently dropping incomplete dimensions. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. A practical analysis table should include: sent count, latest status distribution, failure count, failure-rate delta versus baseline, top exact error strings/codes, first observed timestamp, affected templates, affected countries, and affected profiles. @@ -108,7 +110,10 @@ Do not mistake broadcast for fallback. Omitted `channel` or `["sent"]` enables a - [ ] The analysis uses Sent `message_id` values as the primary unit. - [ ] The cohort is pinned by time window, profile/sender identity, template, channel, and recipient segment. - [ ] Status math uses the latest known status per Sent message ID. +- [ ] Transition math uses observed activity histories and never backfills stages from a latest-only status. - [ ] Pending or in-flight messages are either excluded or reported separately. +- [ ] SMS delivery analysis stops at `DELIVERED`; WhatsApp/RCS `READ` is labeled engagement. +- [ ] Global and channel × direction totals reconcile, including malformed and explicit `unknown` buckets. - [ ] Webhook configuration, event history, and endpoint test results are checked when the symptom is missing callbacks. - [ ] Channel-specific failures are split before aggregate rates are reported. - [ ] Provider or carrier codes are quoted exactly as observed and not invented from a lookup table. @@ -132,7 +137,7 @@ Use the `sent` skill for shared Sent terminology and routing. |---|---|---| | `references/mdr-status-codes.md` | Lookup table | Normalize observed SMS, WhatsApp, and RCS provider errors without putting long code dictionaries in the skill body. | | `references/performance-diagnosis-playbook.md` | Worked examples | Decision tree for which signal to investigate first, channel-specific diagnostic patterns, cross-skill handoff matrix, and escalation criteria. | -| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), prints per-stage counts and drop-off percentages, exits non-zero on anomalies. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N` to tune, default 20; pass `--show-errors` to also tally `ERR_*` codes parsed from FAILED message `description` fields). | +| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), groups channel × direction outcomes, separates delivery transitions from engagement, and retains malformed/unknown rows. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N`, `--show-errors`, or `--format json`). Exit `0` means no observed transition breach, `2` means bad input/no usable cohort, and `3` means an observed breach. JSON uses `null` where a denominator is unavailable; text uses `N/A`. | | `scripts/fixtures/good.json` | Fixture | Synthetic healthy-funnel MDR export. | | `scripts/fixtures/bad.json` | Fixture | Synthetic MDR export with deliberate >50% SENT→DELIVERED drop. | diff --git a/claude-plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md b/claude-plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md index 36ead53..44d8175 100644 --- a/claude-plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md +++ b/claude-plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md @@ -9,7 +9,7 @@ Authoritative upstream sources (for the downstream provider codes that may appea ## Message status lifecycle -Sent normalizes all channels into a single state machine. Only one terminal at a time; only the **latest** status per `message_id` is meaningful when computing funnel counts. +Sent normalizes channels into a shared delivery state machine. The latest status per `message_id` determines its current outcome, while an activity history supplies the evidence for transition counts. Do not backfill earlier stages from a latest-only record. ``` QUEUED -> ROUTED -> SENT -> DELIVERED -> READ (WhatsApp & RCS only) @@ -27,7 +27,7 @@ FAILED FAILED FAILED FAILED | `FAILED` | Terminal failure; the per-message reason is in the `description` field. | | `RECEIVED` | Inbound message from end user. | -A message can transition `SENT -> DELIVERED -> FAILED` (e.g. expired WhatsApp window, capability lost on RCS); count the latest status, not the journey. +A message can transition `SENT -> DELIVERED -> FAILED`; classify its terminal outcome from the final observed event while retaining the actual journey for transition analysis. `READ` is WhatsApp/RCS engagement, not an SMS delivery stage. ## Synchronous errors (HTTP response body) @@ -207,7 +207,10 @@ Carriers don't share an enum; the categories you actually need to triage on: ## Counting rules - **Use Sent `message_id`** as the primary unit. Provider IDs (carrier message IDs, `wamid`, RBM `messageId`) are useful for escalation but are **not** in the v3 docs as join keys. -- **Use the latest status** (`max(timestamp)`) — a `FAILED` after `DELIVERED` means `FAILED`; a `READ` after `DELIVERED` means `READ`. -- **Exclude pending** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from rate denominators — they're indeterminate. +- **Use the latest observed event for the outcome** — a `FAILED` after `DELIVERED` is a terminal failure; a `READ` after `DELIVERED` is a delivered message with observed engagement. +- **Use only explicit history for transitions.** A latest-only `DELIVERED` record does not prove the export observed `QUEUED`, `ROUTED`, or `SENT`; render those transition denominators as unavailable. +- **Separate pending/deferred records** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from terminal rate denominators. +- **Group by channel and direction.** Keep missing or unsupported dimensions in `unknown` buckets so totals reconcile instead of silently excluding them. +- **Stop SMS at `DELIVERED`.** Calculate `READ` engagement only for WhatsApp and RCS. - **Separate channel fan-out.** `POST /v3/messages` with `"channel": ["sms","whatsapp","rcs"]` creates one message per channel; each has its own `message_id` and its own lifecycle. Don't double-count at the recipient level unless the user explicitly asks for recipient-level rollup. - **Honor a minimum cohort size** before drawing conclusions about small rate shifts. A working heuristic is ≥1,000 messages per cohort; below that, noise dominates. This is an analyst rule of thumb, not a Sent API rule. diff --git a/claude-plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md b/claude-plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md index a24bd47..a8b5787 100644 --- a/claude-plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md +++ b/claude-plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md @@ -129,9 +129,11 @@ When escalating, include: account / profile ID, channel, cohort definition (temp Repeat until the symptom is explained or scoped: 1. Pin the cohort (channel × template × country × profile × window). -2. Compute the funnel; identify the broken lifecycle stage (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`/`READ`). -3. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. -4. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. -5. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. -6. Hand off via the matrix above, or escalate to Sent support with the required evidence. -7. Quantify the diagnosis — never "looks better now" without a recomputed funnel. +2. Split channel × direction groups, reconcile them to the input total, and separate progression, terminal failure, deferred, inbound, and malformed/unknown outcomes. +3. Compute delivery transitions only from explicit activity histories (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`). If the export contains latest-only rows, report their outcomes without inventing prior transitions. +4. For WhatsApp/RCS, report `READ` separately as engagement. Stop SMS delivery analysis at `DELIVERED`. +5. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. +6. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. +7. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. +8. Hand off via the matrix above, or escalate to Sent support with the required evidence. +9. Quantify the diagnosis — never "looks better now" without a recomputed funnel. diff --git a/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py b/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py index 2df3b5e..50df69f 100644 --- a/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py +++ b/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py @@ -1,21 +1,15 @@ #!/usr/bin/env python3 -"""Analyze a Sent MDR export and report funnel drop-off by lifecycle stage. +"""Analyze Sent MDR records by channel, direction, lifecycle, and outcome. -Lifecycle stages (in order): QUEUED, ROUTED, SENT, DELIVERED, READ. - -The script counts distinct messages that reached at least each stage, -computes the drop-off percentage between adjacent stages, and flags any -stage where drop-off exceeds the configured threshold. +The analyzer uses only statuses that are present in each record. A record with +only ``status=DELIVERED`` contributes one observed DELIVERED status; it does not +implicitly contribute QUEUED, ROUTED, or SENT transitions. Full ``statuses`` +histories can contribute explicit transition rates. Exit codes: - 0 - healthy funnel (no stage drops more than --threshold) - 2 - bad arguments / malformed input - 3 - unhealthy funnel (one or more stages exceed the threshold) - -Examples: - python analyze_mdr_funnel.py path/to/mdr.json - python analyze_mdr_funnel.py path/to/mdr.csv --threshold 15 - python analyze_mdr_funnel.py path/to/mdr.json --show-errors + 0 - usable cohort with no threshold breach, or usable outcome-only data + 2 - bad arguments, malformed input, or no usable analysis cohort + 3 - one or more explicit delivery/engagement transitions breach threshold """ from __future__ import annotations @@ -25,217 +19,463 @@ import json import re import sys -from collections import Counter +from collections import Counter, defaultdict +from dataclasses import dataclass from pathlib import Path -from typing import Iterable +from typing import Any, Iterable + + +DELIVERY_STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED") +ENGAGEMENT_STAGE = "READ" +KNOWN_STATUSES = set(DELIVERY_STAGES) | {ENGAGEMENT_STAGE, "FAILED", "DEFERRED", "RECEIVED"} +OUTCOMES: tuple[str, ...] = ( + "progression", + "terminal_failure", + "deferred", + "inbound", + "malformed", + "unknown", +) +SUPPORTED_CHANNELS = {"sms", "whatsapp", "rcs"} +READ_CHANNELS = {"whatsapp", "rcs"} +OUTBOUND_DIRECTIONS = {"outbound", "outgoing", "mt"} +INBOUND_DIRECTIONS = {"inbound", "incoming", "mo"} +_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") +_STATUS_KEYS = ("status", "stage", "latest_status") -# Ordered lifecycle stages. A message that reached stage N is counted as -# having reached every earlier stage as well. -STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED", "READ") -STAGE_RANK: dict[str, int] = {s: i for i, s in enumerate(STAGES)} + +class InputError(Exception): + """Raised for missing or structurally invalid input. Maps to exit code 2.""" + + +@dataclass(frozen=True) +class NormalizedRecord: + """One input row normalized without inventing lifecycle evidence.""" + + index: int + channel: str + direction: str + observed_statuses: tuple[str, ...] + has_history: bool + outcome: str + diagnostics: tuple[str, ...] + error_codes: tuple[str, ...] def _parse_args(argv: list[str]) -> argparse.Namespace: - parser = argparse.ArgumentParser( - prog="analyze_mdr_funnel.py", - description=( - "Analyze a Sent MDR export and report funnel drop-off by " - "lifecycle stage (QUEUED -> ROUTED -> SENT -> DELIVERED -> READ)." - ), - ) - parser.add_argument( - "path", - help="Path to the MDR export (CSV or JSON; format auto-detected by extension).", - ) + parser = argparse.ArgumentParser(prog="analyze_mdr_funnel.py", description=__doc__) + parser.add_argument("path", help="MDR export path (.json or .csv)") parser.add_argument( "--threshold", type=float, default=20.0, - help=( - "Drop-off percentage threshold per stage. Any stage with a " - "drop-off greater than this value triggers a non-zero exit. " - "Default: 20." - ), + help="Explicit transition drop-off percentage that triggers exit 3 (default: 20)", ) parser.add_argument( "--show-errors", action="store_true", - help=( - "When set, also summarise the count of each Sent send-time error " - "code (ERR_*) parsed from the 'description' field of FAILED " - "messages. Non-FAILED rows and rows without a description are " - "ignored. Does not change exit codes." - ), + help="Include ERR_* codes from FAILED record descriptions", + ) + parser.add_argument( + "--format", + choices=("text", "json"), + default="text", + help="Output format (default: text)", ) return parser.parse_args(argv) -class InputError(Exception): - """Raised for malformed or missing input. Mapped to exit code 2.""" - - -def _load_messages(path: Path) -> list[dict]: +def _load_messages(path: Path) -> list[object]: if not path.is_file(): raise InputError(f"file not found: {path}") - ext = path.suffix.lower() - if ext == ".json": + extension = path.suffix.lower() + if extension == ".json": try: - with path.open("r", encoding="utf-8") as fh: - data = json.load(fh) + data = json.loads(path.read_text(encoding="utf-8")) + except OSError as exc: + raise InputError(f"could not read {path}: {exc}") from exc except json.JSONDecodeError as exc: raise InputError(f"invalid JSON in {path}: {exc}") from exc if isinstance(data, dict) and "messages" in data: data = data["messages"] if not isinstance(data, list): - raise InputError("JSON must be a list of message records or {\"messages\": [...]}") - messages = data - if ext == ".csv": + raise InputError('JSON must be a list of message records or {"messages": [...]}') + return data + if extension == ".csv": try: - with path.open("r", encoding="utf-8", newline="") as fh: - messages = list(csv.DictReader(fh)) - except (csv.Error, UnicodeError) as exc: + with path.open("r", encoding="utf-8", newline="") as handle: + return list(csv.DictReader(handle)) + except (OSError, csv.Error, UnicodeError) as exc: raise InputError(f"invalid CSV in {path}: {exc}") from exc - elif ext != ".json": - raise InputError(f"unsupported file extension '{ext}'; expected .json or .csv") - for index, record in enumerate(messages, 1): - if not isinstance(record, dict): - raise InputError(f"record {index} must be an object") - return messages - - -def _latest_stage(record: dict) -> str | None: - """Return the furthest stage a message reached, or None if unknown. - - Accepts either a single `status`/`stage` field, or a `statuses` list of - {stage, ...} entries (treated as the lifecycle history; the max-rank stage - is the latest in funnel-progression sense). - """ - if "statuses" in record and isinstance(record["statuses"], list): - ranks = [ - STAGE_RANK[s["stage"].upper()] - for s in record["statuses"] - if isinstance(s, dict) and isinstance(s.get("stage"), str) and s["stage"].upper() in STAGE_RANK - ] - if not ranks: - return None - return STAGES[max(ranks)] - for key in ("status", "stage", "latest_status"): - val = record.get(key) - if isinstance(val, str) and val.upper() in STAGE_RANK: - return val.upper() - return None - - -def compute_funnel(messages: Iterable[dict]) -> dict[str, int]: - """Return {stage: count_of_messages_that_reached_at_least_this_stage}.""" - counts = {s: 0 for s in STAGES} - for record in messages: - latest = _latest_stage(record) - if latest is None: - continue - for stage in STAGES[: STAGE_RANK[latest] + 1]: - counts[stage] += 1 - return counts - - -def stage_dropoffs(counts: dict[str, int]) -> list[tuple[str, str, float]]: - """Return [(from_stage, to_stage, dropoff_pct)] for adjacent stages.""" - out: list[tuple[str, str, float]] = [] - for i in range(len(STAGES) - 1): - a, b = STAGES[i], STAGES[i + 1] - if counts[a] == 0: - pct = 0.0 - else: - pct = (counts[a] - counts[b]) / counts[a] * 100.0 - out.append((a, b, pct)) - return out + raise InputError(f"unsupported file extension '{extension}'; expected .json or .csv") + + +def _field(record: dict[str, Any], name: str) -> Any: + if name in record: + return record[name] + payload = record.get("payload") + return payload.get(name) if isinstance(payload, dict) else None + + +def _normalize_channel(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("channel missing; grouped as unknown") + return "unknown" + channel = value.strip().lower() + if channel not in SUPPORTED_CHANNELS: + diagnostics.append(f"unsupported channel {value!r}; grouped as unknown") + return "unknown" + return channel + + +def _normalize_direction(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("direction missing; grouped as unknown") + return "unknown" + direction = value.strip().lower() + if direction in OUTBOUND_DIRECTIONS: + return "outbound" + if direction in INBOUND_DIRECTIONS: + return "inbound" + diagnostics.append(f"unsupported direction {value!r}; grouped as unknown") + return "unknown" + + +def _normalize_statuses(record: dict[str, Any], diagnostics: list[str]) -> tuple[tuple[str, ...], bool]: + if "statuses" in record: + history = record["statuses"] + if not isinstance(history, list): + diagnostics.append("statuses must be an array") + return (), False + statuses: list[str] = [] + for position, entry in enumerate(history, 1): + if not isinstance(entry, dict): + diagnostics.append(f"statuses[{position}] must be an object") + continue + value = next((entry.get(key) for key in _STATUS_KEYS if isinstance(entry.get(key), str)), None) + if value is None or not value.strip(): + diagnostics.append(f"statuses[{position}] has no status/stage value") + continue + statuses.append(value.strip().upper()) + return tuple(statuses), True + for key in _STATUS_KEYS: + value = _field(record, key) + if isinstance(value, str) and value.strip(): + return (value.strip().upper(),), False + diagnostics.append("record has no observed status") + return (), False + + +def normalize_record(record: object, index: int) -> NormalizedRecord: + if not isinstance(record, dict): + return NormalizedRecord( + index=index, + channel="unknown", + direction="unknown", + observed_statuses=(), + has_history=False, + outcome="malformed", + diagnostics=("record must be an object",), + error_codes=(), + ) -def _print_report(counts: dict[str, int], drops: list[tuple[str, str, float]]) -> None: - print("Funnel:") - for stage in STAGES: - print(f" {stage:<10} {counts[stage]}") - print("\nStage drop-off:") - for a, b, pct in drops: - print(f" {a} -> {b}: {pct:.1f}%") + diagnostics: list[str] = [] + channel = _normalize_channel(_field(record, "channel"), diagnostics) + direction = _normalize_direction(_field(record, "direction"), diagnostics) + statuses, has_history = _normalize_statuses(record, diagnostics) + unknown_statuses = sorted({status for status in statuses if status not in KNOWN_STATUSES}) + if unknown_statuses: + diagnostics.append("unknown status values: " + ", ".join(unknown_statuses)) + + known = tuple(status for status in statuses if status in KNOWN_STATUSES) + latest = known[-1] if known else None + if direction == "inbound" or latest == "RECEIVED": + direction = "inbound" + outcome = "inbound" + elif latest == "FAILED": + outcome = "terminal_failure" + elif latest == "DEFERRED" or latest in {"QUEUED", "ROUTED", "SENT"}: + outcome = "deferred" + elif latest in {"DELIVERED", "READ"}: + outcome = "progression" + else: + outcome = "unknown" + + if channel == "sms" and "READ" in known: + diagnostics.append("READ is not used for SMS delivery or engagement analysis") + description = _field(record, "description") + error_codes = tuple(sorted(set(_ERR_CODE_RE.findall(description)))) if isinstance(description, str) else () + return NormalizedRecord( + index=index, + channel=channel, + direction=direction, + observed_statuses=statuses, + has_history=has_history, + outcome=outcome, + diagnostics=tuple(diagnostics), + error_codes=error_codes, + ) -# Sent's documented send-time per-message error codes appear in the message -# `description` field on FAILED rows. See references/mdr-status-codes.md. -_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") -_STATUS_KEYS = ("status", "stage", "latest_status") +def _completed_transition(statuses: tuple[str, ...], source: str, target: str) -> bool: + try: + source_index = statuses.index(source) + except ValueError: + return False + return target in statuses[source_index + 1 :] + + +def _transition( + records: list[NormalizedRecord], source: str, target: str, threshold: float +) -> dict[str, object]: + terminal = {"progression", "terminal_failure"} + eligible_records = [ + record + for record in records + if record.has_history and record.outcome in terminal and source in record.observed_statuses + ] + completed = sum( + _completed_transition(record.observed_statuses, source, target) for record in eligible_records + ) + eligible = len(eligible_records) + dropoff = round((eligible - completed) / eligible * 100.0, 4) if eligible else None + return { + "from": source, + "to": target, + "eligible": eligible, + "completed": completed, + "dropoff_pct": dropoff, + "exceeds_threshold": dropoff is not None and dropoff > threshold, + } + + +def _totals(records: Iterable[NormalizedRecord]) -> dict[str, int]: + records = list(records) + counts = Counter(record.outcome for record in records) + return { + "input": len(records), + "usable": sum(counts[name] for name in ("progression", "terminal_failure", "deferred")), + **{name: counts[name] for name in OUTCOMES}, + } + + +def _group_report( + channel: str, + direction: str, + records: list[NormalizedRecord], + threshold: float, +) -> dict[str, object]: + delivery_statuses = { + stage: sum(stage in record.observed_statuses for record in records) for stage in DELIVERY_STAGES + } + transitions = [ + _transition(records, source, target, threshold) + for source, target in zip(DELIVERY_STAGES, DELIVERY_STAGES[1:]) + ] + engagement: dict[str, object] | None = None + if channel in READ_CHANNELS: + read_transition = _transition(records, "DELIVERED", "READ", threshold) + engagement = { + "delivered": read_transition["eligible"], + "read": read_transition["completed"], + "read_rate_pct": ( + round(100.0 - float(read_transition["dropoff_pct"]), 4) + if read_transition["dropoff_pct"] is not None + else None + ), + "dropoff_pct": read_transition["dropoff_pct"], + "exceeds_threshold": read_transition["exceeds_threshold"], + } + return { + "channel": channel, + "direction": direction, + "totals": _totals(records), + "progression": {"observed": delivery_statuses, "transitions": transitions}, + "engagement": engagement, + } + + +def build_report( + records: list[NormalizedRecord], + threshold: float, + include_errors: bool, +) -> dict[str, object]: + grouped: dict[tuple[str, str], list[NormalizedRecord]] = defaultdict(list) + for record in records: + grouped[(record.channel, record.direction)].append(record) + groups = [ + _group_report(channel, direction, grouped[(channel, direction)], threshold) + for channel, direction in sorted(grouped) + ] + anomalies: list[dict[str, object]] = [] + evaluated = 0 + for group in groups: + for transition in group["progression"]["transitions"]: + if transition["dropoff_pct"] is not None: + evaluated += 1 + if transition["exceeds_threshold"]: + anomalies.append( + { + "kind": "delivery", + "channel": group["channel"], + "direction": group["direction"], + **transition, + } + ) + engagement = group["engagement"] + if engagement is not None: + if engagement["dropoff_pct"] is not None: + evaluated += 1 + if engagement["exceeds_threshold"]: + anomalies.append( + { + "kind": "engagement", + "channel": group["channel"], + "direction": group["direction"], + "from": "DELIVERED", + "to": "READ", + "eligible": engagement["delivered"], + "completed": engagement["read"], + "dropoff_pct": engagement["dropoff_pct"], + "exceeds_threshold": True, + } + ) + totals = _totals(records) + healthy: bool | None + if not totals["usable"] or not evaluated: + healthy = None + else: + healthy = not anomalies + diagnostics = [ + {"record": record.index, "messages": list(record.diagnostics)} + for record in records + if record.diagnostics + ] + error_codes = Counter( + code + for record in records + if record.outcome == "terminal_failure" + for code in record.error_codes + ) + return { + "schema_version": 1, + "threshold_pct": threshold, + "healthy": healthy, + "totals": totals, + "groups": groups, + "anomalies": anomalies, + "diagnostics": diagnostics, + "error_codes": dict(sorted(error_codes.items())) if include_errors else None, + } -def _is_failed(record: dict) -> bool: - # FAILED isn't in STAGES, so _latest_stage() returns None for it. Check - # the explicit status keys and the statuses history directly. - for key in _STATUS_KEYS: - val = record.get(key) - if isinstance(val, str) and val.upper() == "FAILED": - return True - history = record.get("statuses") - if isinstance(history, list): - for entry in history: - if isinstance(entry, dict) and isinstance(entry.get("stage"), str) \ - and entry["stage"].upper() == "FAILED": - return True - return False +def analyze(records: Iterable[object], threshold: float, include_errors: bool) -> dict[str, object]: + normalized = [normalize_record(record, index) for index, record in enumerate(records, 1)] + return build_report(normalized, threshold, include_errors) def summarise_errors(messages: Iterable[dict]) -> Counter: - """Return a Counter of ERR_* codes parsed from FAILED messages' descriptions.""" + """Return ERR_* code counts from records whose observed outcome is FAILED.""" counter: Counter = Counter() - for record in messages: - if not isinstance(record, dict) or not _is_failed(record): - continue - description = record.get("description") - if not isinstance(description, str): - continue - # Dedupe within a row so one message with the same code mentioned - # twice doesn't double-count. - counter.update(set(_ERR_CODE_RE.findall(description))) + for index, record in enumerate(messages, 1): + normalized = normalize_record(record, index) + if normalized.outcome == "terminal_failure": + counter.update(normalized.error_codes) return counter -def _print_error_summary(counter: Counter) -> None: - print("\nERR_* codes (from FAILED descriptions):") - if not counter: - print(" (none found)") - return - for code, count in counter.most_common(): - print(f" {code:<32} {count}") +def _percent(value: object) -> str: + return "N/A" if value is None else f"{float(value):.1f}%" + + +def render_text(report: dict[str, object]) -> str: + totals = report["totals"] + lines = [ + f"Records: {totals['input']} input, {totals['usable']} usable", + "Outcomes: " + ", ".join(f"{name}={totals[name]}" for name in OUTCOMES), + ] + for group in report["groups"]: + lines.extend( + ( + "", + f"Group: channel={group['channel']} direction={group['direction']}", + " Outcomes: " + ", ".join(f"{name}={group['totals'][name]}" for name in OUTCOMES), + " Observed: " + + ", ".join( + f"{stage}={group['progression']['observed'][stage]}" for stage in DELIVERY_STAGES + ), + " Delivery transitions:", + ) + ) + for transition in group["progression"]["transitions"]: + lines.append( + f" {transition['from']} -> {transition['to']}: " + f"{_percent(transition['dropoff_pct'])} " + f"({transition['completed']}/{transition['eligible']} completed)" + ) + if group["engagement"] is not None: + engagement = group["engagement"] + lines.append(" Engagement:") + lines.append( + " DELIVERED -> READ: " + f"{_percent(engagement['dropoff_pct'])} " + f"({engagement['read']}/{engagement['delivered']} read)" + ) + if report["error_codes"] is not None: + lines.append("") + lines.append("ERR_* codes:") + if report["error_codes"]: + lines.extend(f" {code}: {count}" for code, count in report["error_codes"].items()) + else: + lines.append(" (none found)") + if report["diagnostics"]: + lines.append("") + lines.append(f"Validation diagnostics: {len(report['diagnostics'])} record(s)") + for diagnostic in report["diagnostics"]: + lines.append(f" record {diagnostic['record']}: " + "; ".join(diagnostic["messages"])) + lines.append("") + if not totals["usable"]: + lines.append("INSUFFICIENT: no usable analysis cohort exists.") + elif report["healthy"] is None: + lines.append("INDETERMINATE: outcomes are usable, but no explicit transition denominator exists.") + elif report["healthy"]: + lines.append(f"OK: no explicit transition exceeded {report['threshold_pct']}% drop-off.") + else: + lines.append( + f"FAIL: {len(report['anomalies'])} explicit transition(s) exceeded " + f"{report['threshold_pct']}% drop-off." + ) + return "\n".join(lines) def main(argv: list[str] | None = None) -> int: args = _parse_args(sys.argv[1:] if argv is None else argv) + if not 0 <= args.threshold <= 100: + print("error: --threshold must be between 0 and 100", file=sys.stderr) + return 2 try: - messages = _load_messages(Path(args.path)) + records = _load_messages(Path(args.path)) except InputError as exc: print(f"error: {exc}", file=sys.stderr) return 2 - if not messages: - print("error: no messages found in input", file=sys.stderr) - return 2 - - counts = compute_funnel(messages) - drops = stage_dropoffs(counts) - _print_report(counts, drops) - if args.show_errors: - _print_error_summary(summarise_errors(messages)) + report = analyze(records, args.threshold, args.show_errors) + if args.format == "json": + print(json.dumps(report, indent=2, sort_keys=True)) + else: + print(render_text(report)) - anomalies = [(a, b, pct) for a, b, pct in drops if pct > args.threshold] - if anomalies: - print( - f"\nFAIL: {len(anomalies)} stage(s) exceeded the {args.threshold}% drop-off threshold:", - file=sys.stderr, - ) - for a, b, pct in anomalies: - print(f" {a} -> {b}: {pct:.1f}%", file=sys.stderr) + if report["totals"]["usable"] == 0: + if args.format == "text": + print("error: no usable analysis cohort exists", file=sys.stderr) + return 2 + if report["healthy"] is False: + if args.format == "text": + print(f"FAIL: {len(report['anomalies'])} transition threshold breach(es)", file=sys.stderr) return 3 - print(f"\nOK: no stage exceeded the {args.threshold}% drop-off threshold.") return 0 if __name__ == "__main__": - sys.exit(main()) + raise SystemExit(main()) diff --git a/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json b/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json index 038028d..e8bbd98 100644 --- a/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json +++ b/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json @@ -1,502 +1,118 @@ [ { - "message_id": "msg_test_001", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_002", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_003", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_004", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_005", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_010", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "message_id": "msg_test_101", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_102", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_103", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_104", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_105", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_TEMPLATE_PARAMS_INVALID" + }, + { + "message_id": "msg_test_106", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_107", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_108", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_109", + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"} + ] + }, + { + "message_id": "msg_test_110", + "channel": "whatsapp", + "direction": "inbound", + "status": "RECEIVED" + }, + "malformed synthetic record", + { + "message_id": "msg_test_112", + "channel": "sms", + "direction": "outbound", + "status": "MYSTERY" } ] diff --git a/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json b/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json index 56d3784..0b66c89 100644 --- a/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json +++ b/claude-plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json @@ -2,501 +2,111 @@ { "message_id": "msg_test_001", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_002", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_003", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_004", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_005", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "satellite", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_010", "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "direction": "inbound", + "status": "RECEIVED" } ] diff --git a/evals/messaging-performance-analyzer.yaml b/evals/messaging-performance-analyzer.yaml index c3ab30b..f2db5bc 100644 --- a/evals/messaging-performance-analyzer.yaml +++ b/evals/messaging-performance-analyzer.yaml @@ -9,6 +9,12 @@ cases: - query: "Help me read this MDR — I'm seeing a spike in Meta error 131047" expect: trigger rationale: Explicit MDR keyword + WhatsApp-specific status-code spike. Exactly the per-channel status-code triage path. + - query: "This export only has each message's latest status. Can you compare SMS delivery with WhatsApp reads without inventing stages?" + expect: trigger + rationale: Explicitly requests channel-aware outcome analysis and the latest-only evidence rule handled by the MDR analyzer. + - query: "My MDR has inbound, outbound, unknown-channel, and malformed rows. Reconcile every row and show me the usable cohorts." + expect: trigger + rationale: Requests the analyzer's direction grouping, unknown buckets, diagnostics, and reconciliation behavior. - query: "Compare AWS SNS vs Twilio pricing for transactional SMS" expect: no_trigger rationale: Vendor/pricing comparison, not delivery-funnel analysis on Sent. No MDR, no funnel, no Sent-tenant context. diff --git a/packages/sent/skills/messaging-performance-analyzer/SKILL.md b/packages/sent/skills/messaging-performance-analyzer/SKILL.md index c7b0d4d..3898887 100644 --- a/packages/sent/skills/messaging-performance-analyzer/SKILL.md +++ b/packages/sent/skills/messaging-performance-analyzer/SKILL.md @@ -48,13 +48,15 @@ Capture these dimensions before calculating anything: profile or sender identity ### 2. Build cohorts from Sent message IDs -Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once at its latest status, then add recipient-level or campaign-level rollups only after deduplication. +Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once in a terminal-outcome rollup, then add recipient-level or campaign-level rollups only after deduplication. + +Distinguish an activity history from a latest-status snapshot. A history can prove the transitions it contains. A snapshot such as `status=DELIVERED` proves only the observed current outcome; it does not prove that the export also observed `QUEUED`, `ROUTED`, or `SENT`. Report unavailable transition denominators as `N/A`, not zero, and never synthesize missing transitions. Do not use provider IDs such as WhatsApp `wamid`, SMS carrier IDs, or RCS message IDs as the primary join key unless the exported evidence lacks Sent IDs. Provider IDs are useful for escalation, but the Sent API and dashboard track status by Sent message ID. ### 3. Normalize lifecycle stages to Sent’s documented statuses -Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, `DELIVERED`, and `READ` for WhatsApp and RCS. Keep failed and error states in a separate terminal bucket using the exact status/error fields present in the evidence. +Use Sent’s documented delivery lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, and `DELIVERED`. Treat `READ` as a separate engagement measure for WhatsApp and RCS, never as an SMS delivery requirement. Keep terminal failures, deferred/in-flight messages, inbound `RECEIVED` messages, and malformed/unknown records in separate buckets using only fields present in the evidence. | Stage | Interpretation | Common diagnostic question | |---|---|---| @@ -62,7 +64,7 @@ Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, | `ROUTED` | Sent selected a channel/provider path. | Did routing choose the expected channel or fallback path? | | `SENT` | The message left Sent/provider processing toward the destination network. | Are provider accepts high but downstream delivery low? | | `DELIVERED` | Delivery was confirmed where supported. | Did the destination network confirm receipt? | -| `READ` | WhatsApp/RCS read receipt was observed where available. | Did users open the message after delivery? | +| `READ` | WhatsApp/RCS engagement receipt was observed where available. | Did users open the message after delivery? | | Error/failure | A terminal or recoverable error occurred. | Is the root cause compliance, payload, throughput, opt-out, or provider outage? | ### 4. Check webhook health before diagnosing delivery @@ -83,7 +85,7 @@ SMS, WhatsApp, and RCS fail differently. Do not average them together unless the ### 6. Quantify impact before recommending fixes -Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. +Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Reconcile the global totals with every channel × direction group, retaining explicit `unknown` groups instead of silently dropping incomplete dimensions. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. A practical analysis table should include: sent count, latest status distribution, failure count, failure-rate delta versus baseline, top exact error strings/codes, first observed timestamp, affected templates, affected countries, and affected profiles. @@ -108,7 +110,10 @@ Do not mistake broadcast for fallback. Omitted `channel` or `["sent"]` enables a - [ ] The analysis uses Sent `message_id` values as the primary unit. - [ ] The cohort is pinned by time window, profile/sender identity, template, channel, and recipient segment. - [ ] Status math uses the latest known status per Sent message ID. +- [ ] Transition math uses observed activity histories and never backfills stages from a latest-only status. - [ ] Pending or in-flight messages are either excluded or reported separately. +- [ ] SMS delivery analysis stops at `DELIVERED`; WhatsApp/RCS `READ` is labeled engagement. +- [ ] Global and channel × direction totals reconcile, including malformed and explicit `unknown` buckets. - [ ] Webhook configuration, event history, and endpoint test results are checked when the symptom is missing callbacks. - [ ] Channel-specific failures are split before aggregate rates are reported. - [ ] Provider or carrier codes are quoted exactly as observed and not invented from a lookup table. @@ -132,7 +137,7 @@ Use the `sent` skill for shared Sent terminology and routing. |---|---|---| | `references/mdr-status-codes.md` | Lookup table | Normalize observed SMS, WhatsApp, and RCS provider errors without putting long code dictionaries in the skill body. | | `references/performance-diagnosis-playbook.md` | Worked examples | Decision tree for which signal to investigate first, channel-specific diagnostic patterns, cross-skill handoff matrix, and escalation criteria. | -| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), prints per-stage counts and drop-off percentages, exits non-zero on anomalies. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N` to tune, default 20; pass `--show-errors` to also tally `ERR_*` codes parsed from FAILED message `description` fields). | +| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), groups channel × direction outcomes, separates delivery transitions from engagement, and retains malformed/unknown rows. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N`, `--show-errors`, or `--format json`). Exit `0` means no observed transition breach, `2` means bad input/no usable cohort, and `3` means an observed breach. JSON uses `null` where a denominator is unavailable; text uses `N/A`. | | `scripts/fixtures/good.json` | Fixture | Synthetic healthy-funnel MDR export. | | `scripts/fixtures/bad.json` | Fixture | Synthetic MDR export with deliberate >50% SENT→DELIVERED drop. | diff --git a/packages/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md b/packages/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md index 36ead53..44d8175 100644 --- a/packages/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md +++ b/packages/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md @@ -9,7 +9,7 @@ Authoritative upstream sources (for the downstream provider codes that may appea ## Message status lifecycle -Sent normalizes all channels into a single state machine. Only one terminal at a time; only the **latest** status per `message_id` is meaningful when computing funnel counts. +Sent normalizes channels into a shared delivery state machine. The latest status per `message_id` determines its current outcome, while an activity history supplies the evidence for transition counts. Do not backfill earlier stages from a latest-only record. ``` QUEUED -> ROUTED -> SENT -> DELIVERED -> READ (WhatsApp & RCS only) @@ -27,7 +27,7 @@ FAILED FAILED FAILED FAILED | `FAILED` | Terminal failure; the per-message reason is in the `description` field. | | `RECEIVED` | Inbound message from end user. | -A message can transition `SENT -> DELIVERED -> FAILED` (e.g. expired WhatsApp window, capability lost on RCS); count the latest status, not the journey. +A message can transition `SENT -> DELIVERED -> FAILED`; classify its terminal outcome from the final observed event while retaining the actual journey for transition analysis. `READ` is WhatsApp/RCS engagement, not an SMS delivery stage. ## Synchronous errors (HTTP response body) @@ -207,7 +207,10 @@ Carriers don't share an enum; the categories you actually need to triage on: ## Counting rules - **Use Sent `message_id`** as the primary unit. Provider IDs (carrier message IDs, `wamid`, RBM `messageId`) are useful for escalation but are **not** in the v3 docs as join keys. -- **Use the latest status** (`max(timestamp)`) — a `FAILED` after `DELIVERED` means `FAILED`; a `READ` after `DELIVERED` means `READ`. -- **Exclude pending** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from rate denominators — they're indeterminate. +- **Use the latest observed event for the outcome** — a `FAILED` after `DELIVERED` is a terminal failure; a `READ` after `DELIVERED` is a delivered message with observed engagement. +- **Use only explicit history for transitions.** A latest-only `DELIVERED` record does not prove the export observed `QUEUED`, `ROUTED`, or `SENT`; render those transition denominators as unavailable. +- **Separate pending/deferred records** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from terminal rate denominators. +- **Group by channel and direction.** Keep missing or unsupported dimensions in `unknown` buckets so totals reconcile instead of silently excluding them. +- **Stop SMS at `DELIVERED`.** Calculate `READ` engagement only for WhatsApp and RCS. - **Separate channel fan-out.** `POST /v3/messages` with `"channel": ["sms","whatsapp","rcs"]` creates one message per channel; each has its own `message_id` and its own lifecycle. Don't double-count at the recipient level unless the user explicitly asks for recipient-level rollup. - **Honor a minimum cohort size** before drawing conclusions about small rate shifts. A working heuristic is ≥1,000 messages per cohort; below that, noise dominates. This is an analyst rule of thumb, not a Sent API rule. diff --git a/packages/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md b/packages/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md index a24bd47..a8b5787 100644 --- a/packages/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md +++ b/packages/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md @@ -129,9 +129,11 @@ When escalating, include: account / profile ID, channel, cohort definition (temp Repeat until the symptom is explained or scoped: 1. Pin the cohort (channel × template × country × profile × window). -2. Compute the funnel; identify the broken lifecycle stage (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`/`READ`). -3. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. -4. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. -5. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. -6. Hand off via the matrix above, or escalate to Sent support with the required evidence. -7. Quantify the diagnosis — never "looks better now" without a recomputed funnel. +2. Split channel × direction groups, reconcile them to the input total, and separate progression, terminal failure, deferred, inbound, and malformed/unknown outcomes. +3. Compute delivery transitions only from explicit activity histories (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`). If the export contains latest-only rows, report their outcomes without inventing prior transitions. +4. For WhatsApp/RCS, report `READ` separately as engagement. Stop SMS delivery analysis at `DELIVERED`. +5. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. +6. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. +7. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. +8. Hand off via the matrix above, or escalate to Sent support with the required evidence. +9. Quantify the diagnosis — never "looks better now" without a recomputed funnel. diff --git a/packages/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py b/packages/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py index 2df3b5e..50df69f 100644 --- a/packages/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py +++ b/packages/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py @@ -1,21 +1,15 @@ #!/usr/bin/env python3 -"""Analyze a Sent MDR export and report funnel drop-off by lifecycle stage. +"""Analyze Sent MDR records by channel, direction, lifecycle, and outcome. -Lifecycle stages (in order): QUEUED, ROUTED, SENT, DELIVERED, READ. - -The script counts distinct messages that reached at least each stage, -computes the drop-off percentage between adjacent stages, and flags any -stage where drop-off exceeds the configured threshold. +The analyzer uses only statuses that are present in each record. A record with +only ``status=DELIVERED`` contributes one observed DELIVERED status; it does not +implicitly contribute QUEUED, ROUTED, or SENT transitions. Full ``statuses`` +histories can contribute explicit transition rates. Exit codes: - 0 - healthy funnel (no stage drops more than --threshold) - 2 - bad arguments / malformed input - 3 - unhealthy funnel (one or more stages exceed the threshold) - -Examples: - python analyze_mdr_funnel.py path/to/mdr.json - python analyze_mdr_funnel.py path/to/mdr.csv --threshold 15 - python analyze_mdr_funnel.py path/to/mdr.json --show-errors + 0 - usable cohort with no threshold breach, or usable outcome-only data + 2 - bad arguments, malformed input, or no usable analysis cohort + 3 - one or more explicit delivery/engagement transitions breach threshold """ from __future__ import annotations @@ -25,217 +19,463 @@ import json import re import sys -from collections import Counter +from collections import Counter, defaultdict +from dataclasses import dataclass from pathlib import Path -from typing import Iterable +from typing import Any, Iterable + + +DELIVERY_STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED") +ENGAGEMENT_STAGE = "READ" +KNOWN_STATUSES = set(DELIVERY_STAGES) | {ENGAGEMENT_STAGE, "FAILED", "DEFERRED", "RECEIVED"} +OUTCOMES: tuple[str, ...] = ( + "progression", + "terminal_failure", + "deferred", + "inbound", + "malformed", + "unknown", +) +SUPPORTED_CHANNELS = {"sms", "whatsapp", "rcs"} +READ_CHANNELS = {"whatsapp", "rcs"} +OUTBOUND_DIRECTIONS = {"outbound", "outgoing", "mt"} +INBOUND_DIRECTIONS = {"inbound", "incoming", "mo"} +_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") +_STATUS_KEYS = ("status", "stage", "latest_status") -# Ordered lifecycle stages. A message that reached stage N is counted as -# having reached every earlier stage as well. -STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED", "READ") -STAGE_RANK: dict[str, int] = {s: i for i, s in enumerate(STAGES)} + +class InputError(Exception): + """Raised for missing or structurally invalid input. Maps to exit code 2.""" + + +@dataclass(frozen=True) +class NormalizedRecord: + """One input row normalized without inventing lifecycle evidence.""" + + index: int + channel: str + direction: str + observed_statuses: tuple[str, ...] + has_history: bool + outcome: str + diagnostics: tuple[str, ...] + error_codes: tuple[str, ...] def _parse_args(argv: list[str]) -> argparse.Namespace: - parser = argparse.ArgumentParser( - prog="analyze_mdr_funnel.py", - description=( - "Analyze a Sent MDR export and report funnel drop-off by " - "lifecycle stage (QUEUED -> ROUTED -> SENT -> DELIVERED -> READ)." - ), - ) - parser.add_argument( - "path", - help="Path to the MDR export (CSV or JSON; format auto-detected by extension).", - ) + parser = argparse.ArgumentParser(prog="analyze_mdr_funnel.py", description=__doc__) + parser.add_argument("path", help="MDR export path (.json or .csv)") parser.add_argument( "--threshold", type=float, default=20.0, - help=( - "Drop-off percentage threshold per stage. Any stage with a " - "drop-off greater than this value triggers a non-zero exit. " - "Default: 20." - ), + help="Explicit transition drop-off percentage that triggers exit 3 (default: 20)", ) parser.add_argument( "--show-errors", action="store_true", - help=( - "When set, also summarise the count of each Sent send-time error " - "code (ERR_*) parsed from the 'description' field of FAILED " - "messages. Non-FAILED rows and rows without a description are " - "ignored. Does not change exit codes." - ), + help="Include ERR_* codes from FAILED record descriptions", + ) + parser.add_argument( + "--format", + choices=("text", "json"), + default="text", + help="Output format (default: text)", ) return parser.parse_args(argv) -class InputError(Exception): - """Raised for malformed or missing input. Mapped to exit code 2.""" - - -def _load_messages(path: Path) -> list[dict]: +def _load_messages(path: Path) -> list[object]: if not path.is_file(): raise InputError(f"file not found: {path}") - ext = path.suffix.lower() - if ext == ".json": + extension = path.suffix.lower() + if extension == ".json": try: - with path.open("r", encoding="utf-8") as fh: - data = json.load(fh) + data = json.loads(path.read_text(encoding="utf-8")) + except OSError as exc: + raise InputError(f"could not read {path}: {exc}") from exc except json.JSONDecodeError as exc: raise InputError(f"invalid JSON in {path}: {exc}") from exc if isinstance(data, dict) and "messages" in data: data = data["messages"] if not isinstance(data, list): - raise InputError("JSON must be a list of message records or {\"messages\": [...]}") - messages = data - if ext == ".csv": + raise InputError('JSON must be a list of message records or {"messages": [...]}') + return data + if extension == ".csv": try: - with path.open("r", encoding="utf-8", newline="") as fh: - messages = list(csv.DictReader(fh)) - except (csv.Error, UnicodeError) as exc: + with path.open("r", encoding="utf-8", newline="") as handle: + return list(csv.DictReader(handle)) + except (OSError, csv.Error, UnicodeError) as exc: raise InputError(f"invalid CSV in {path}: {exc}") from exc - elif ext != ".json": - raise InputError(f"unsupported file extension '{ext}'; expected .json or .csv") - for index, record in enumerate(messages, 1): - if not isinstance(record, dict): - raise InputError(f"record {index} must be an object") - return messages - - -def _latest_stage(record: dict) -> str | None: - """Return the furthest stage a message reached, or None if unknown. - - Accepts either a single `status`/`stage` field, or a `statuses` list of - {stage, ...} entries (treated as the lifecycle history; the max-rank stage - is the latest in funnel-progression sense). - """ - if "statuses" in record and isinstance(record["statuses"], list): - ranks = [ - STAGE_RANK[s["stage"].upper()] - for s in record["statuses"] - if isinstance(s, dict) and isinstance(s.get("stage"), str) and s["stage"].upper() in STAGE_RANK - ] - if not ranks: - return None - return STAGES[max(ranks)] - for key in ("status", "stage", "latest_status"): - val = record.get(key) - if isinstance(val, str) and val.upper() in STAGE_RANK: - return val.upper() - return None - - -def compute_funnel(messages: Iterable[dict]) -> dict[str, int]: - """Return {stage: count_of_messages_that_reached_at_least_this_stage}.""" - counts = {s: 0 for s in STAGES} - for record in messages: - latest = _latest_stage(record) - if latest is None: - continue - for stage in STAGES[: STAGE_RANK[latest] + 1]: - counts[stage] += 1 - return counts - - -def stage_dropoffs(counts: dict[str, int]) -> list[tuple[str, str, float]]: - """Return [(from_stage, to_stage, dropoff_pct)] for adjacent stages.""" - out: list[tuple[str, str, float]] = [] - for i in range(len(STAGES) - 1): - a, b = STAGES[i], STAGES[i + 1] - if counts[a] == 0: - pct = 0.0 - else: - pct = (counts[a] - counts[b]) / counts[a] * 100.0 - out.append((a, b, pct)) - return out + raise InputError(f"unsupported file extension '{extension}'; expected .json or .csv") + + +def _field(record: dict[str, Any], name: str) -> Any: + if name in record: + return record[name] + payload = record.get("payload") + return payload.get(name) if isinstance(payload, dict) else None + + +def _normalize_channel(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("channel missing; grouped as unknown") + return "unknown" + channel = value.strip().lower() + if channel not in SUPPORTED_CHANNELS: + diagnostics.append(f"unsupported channel {value!r}; grouped as unknown") + return "unknown" + return channel + + +def _normalize_direction(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("direction missing; grouped as unknown") + return "unknown" + direction = value.strip().lower() + if direction in OUTBOUND_DIRECTIONS: + return "outbound" + if direction in INBOUND_DIRECTIONS: + return "inbound" + diagnostics.append(f"unsupported direction {value!r}; grouped as unknown") + return "unknown" + + +def _normalize_statuses(record: dict[str, Any], diagnostics: list[str]) -> tuple[tuple[str, ...], bool]: + if "statuses" in record: + history = record["statuses"] + if not isinstance(history, list): + diagnostics.append("statuses must be an array") + return (), False + statuses: list[str] = [] + for position, entry in enumerate(history, 1): + if not isinstance(entry, dict): + diagnostics.append(f"statuses[{position}] must be an object") + continue + value = next((entry.get(key) for key in _STATUS_KEYS if isinstance(entry.get(key), str)), None) + if value is None or not value.strip(): + diagnostics.append(f"statuses[{position}] has no status/stage value") + continue + statuses.append(value.strip().upper()) + return tuple(statuses), True + for key in _STATUS_KEYS: + value = _field(record, key) + if isinstance(value, str) and value.strip(): + return (value.strip().upper(),), False + diagnostics.append("record has no observed status") + return (), False + + +def normalize_record(record: object, index: int) -> NormalizedRecord: + if not isinstance(record, dict): + return NormalizedRecord( + index=index, + channel="unknown", + direction="unknown", + observed_statuses=(), + has_history=False, + outcome="malformed", + diagnostics=("record must be an object",), + error_codes=(), + ) -def _print_report(counts: dict[str, int], drops: list[tuple[str, str, float]]) -> None: - print("Funnel:") - for stage in STAGES: - print(f" {stage:<10} {counts[stage]}") - print("\nStage drop-off:") - for a, b, pct in drops: - print(f" {a} -> {b}: {pct:.1f}%") + diagnostics: list[str] = [] + channel = _normalize_channel(_field(record, "channel"), diagnostics) + direction = _normalize_direction(_field(record, "direction"), diagnostics) + statuses, has_history = _normalize_statuses(record, diagnostics) + unknown_statuses = sorted({status for status in statuses if status not in KNOWN_STATUSES}) + if unknown_statuses: + diagnostics.append("unknown status values: " + ", ".join(unknown_statuses)) + + known = tuple(status for status in statuses if status in KNOWN_STATUSES) + latest = known[-1] if known else None + if direction == "inbound" or latest == "RECEIVED": + direction = "inbound" + outcome = "inbound" + elif latest == "FAILED": + outcome = "terminal_failure" + elif latest == "DEFERRED" or latest in {"QUEUED", "ROUTED", "SENT"}: + outcome = "deferred" + elif latest in {"DELIVERED", "READ"}: + outcome = "progression" + else: + outcome = "unknown" + + if channel == "sms" and "READ" in known: + diagnostics.append("READ is not used for SMS delivery or engagement analysis") + description = _field(record, "description") + error_codes = tuple(sorted(set(_ERR_CODE_RE.findall(description)))) if isinstance(description, str) else () + return NormalizedRecord( + index=index, + channel=channel, + direction=direction, + observed_statuses=statuses, + has_history=has_history, + outcome=outcome, + diagnostics=tuple(diagnostics), + error_codes=error_codes, + ) -# Sent's documented send-time per-message error codes appear in the message -# `description` field on FAILED rows. See references/mdr-status-codes.md. -_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") -_STATUS_KEYS = ("status", "stage", "latest_status") +def _completed_transition(statuses: tuple[str, ...], source: str, target: str) -> bool: + try: + source_index = statuses.index(source) + except ValueError: + return False + return target in statuses[source_index + 1 :] + + +def _transition( + records: list[NormalizedRecord], source: str, target: str, threshold: float +) -> dict[str, object]: + terminal = {"progression", "terminal_failure"} + eligible_records = [ + record + for record in records + if record.has_history and record.outcome in terminal and source in record.observed_statuses + ] + completed = sum( + _completed_transition(record.observed_statuses, source, target) for record in eligible_records + ) + eligible = len(eligible_records) + dropoff = round((eligible - completed) / eligible * 100.0, 4) if eligible else None + return { + "from": source, + "to": target, + "eligible": eligible, + "completed": completed, + "dropoff_pct": dropoff, + "exceeds_threshold": dropoff is not None and dropoff > threshold, + } + + +def _totals(records: Iterable[NormalizedRecord]) -> dict[str, int]: + records = list(records) + counts = Counter(record.outcome for record in records) + return { + "input": len(records), + "usable": sum(counts[name] for name in ("progression", "terminal_failure", "deferred")), + **{name: counts[name] for name in OUTCOMES}, + } + + +def _group_report( + channel: str, + direction: str, + records: list[NormalizedRecord], + threshold: float, +) -> dict[str, object]: + delivery_statuses = { + stage: sum(stage in record.observed_statuses for record in records) for stage in DELIVERY_STAGES + } + transitions = [ + _transition(records, source, target, threshold) + for source, target in zip(DELIVERY_STAGES, DELIVERY_STAGES[1:]) + ] + engagement: dict[str, object] | None = None + if channel in READ_CHANNELS: + read_transition = _transition(records, "DELIVERED", "READ", threshold) + engagement = { + "delivered": read_transition["eligible"], + "read": read_transition["completed"], + "read_rate_pct": ( + round(100.0 - float(read_transition["dropoff_pct"]), 4) + if read_transition["dropoff_pct"] is not None + else None + ), + "dropoff_pct": read_transition["dropoff_pct"], + "exceeds_threshold": read_transition["exceeds_threshold"], + } + return { + "channel": channel, + "direction": direction, + "totals": _totals(records), + "progression": {"observed": delivery_statuses, "transitions": transitions}, + "engagement": engagement, + } + + +def build_report( + records: list[NormalizedRecord], + threshold: float, + include_errors: bool, +) -> dict[str, object]: + grouped: dict[tuple[str, str], list[NormalizedRecord]] = defaultdict(list) + for record in records: + grouped[(record.channel, record.direction)].append(record) + groups = [ + _group_report(channel, direction, grouped[(channel, direction)], threshold) + for channel, direction in sorted(grouped) + ] + anomalies: list[dict[str, object]] = [] + evaluated = 0 + for group in groups: + for transition in group["progression"]["transitions"]: + if transition["dropoff_pct"] is not None: + evaluated += 1 + if transition["exceeds_threshold"]: + anomalies.append( + { + "kind": "delivery", + "channel": group["channel"], + "direction": group["direction"], + **transition, + } + ) + engagement = group["engagement"] + if engagement is not None: + if engagement["dropoff_pct"] is not None: + evaluated += 1 + if engagement["exceeds_threshold"]: + anomalies.append( + { + "kind": "engagement", + "channel": group["channel"], + "direction": group["direction"], + "from": "DELIVERED", + "to": "READ", + "eligible": engagement["delivered"], + "completed": engagement["read"], + "dropoff_pct": engagement["dropoff_pct"], + "exceeds_threshold": True, + } + ) + totals = _totals(records) + healthy: bool | None + if not totals["usable"] or not evaluated: + healthy = None + else: + healthy = not anomalies + diagnostics = [ + {"record": record.index, "messages": list(record.diagnostics)} + for record in records + if record.diagnostics + ] + error_codes = Counter( + code + for record in records + if record.outcome == "terminal_failure" + for code in record.error_codes + ) + return { + "schema_version": 1, + "threshold_pct": threshold, + "healthy": healthy, + "totals": totals, + "groups": groups, + "anomalies": anomalies, + "diagnostics": diagnostics, + "error_codes": dict(sorted(error_codes.items())) if include_errors else None, + } -def _is_failed(record: dict) -> bool: - # FAILED isn't in STAGES, so _latest_stage() returns None for it. Check - # the explicit status keys and the statuses history directly. - for key in _STATUS_KEYS: - val = record.get(key) - if isinstance(val, str) and val.upper() == "FAILED": - return True - history = record.get("statuses") - if isinstance(history, list): - for entry in history: - if isinstance(entry, dict) and isinstance(entry.get("stage"), str) \ - and entry["stage"].upper() == "FAILED": - return True - return False +def analyze(records: Iterable[object], threshold: float, include_errors: bool) -> dict[str, object]: + normalized = [normalize_record(record, index) for index, record in enumerate(records, 1)] + return build_report(normalized, threshold, include_errors) def summarise_errors(messages: Iterable[dict]) -> Counter: - """Return a Counter of ERR_* codes parsed from FAILED messages' descriptions.""" + """Return ERR_* code counts from records whose observed outcome is FAILED.""" counter: Counter = Counter() - for record in messages: - if not isinstance(record, dict) or not _is_failed(record): - continue - description = record.get("description") - if not isinstance(description, str): - continue - # Dedupe within a row so one message with the same code mentioned - # twice doesn't double-count. - counter.update(set(_ERR_CODE_RE.findall(description))) + for index, record in enumerate(messages, 1): + normalized = normalize_record(record, index) + if normalized.outcome == "terminal_failure": + counter.update(normalized.error_codes) return counter -def _print_error_summary(counter: Counter) -> None: - print("\nERR_* codes (from FAILED descriptions):") - if not counter: - print(" (none found)") - return - for code, count in counter.most_common(): - print(f" {code:<32} {count}") +def _percent(value: object) -> str: + return "N/A" if value is None else f"{float(value):.1f}%" + + +def render_text(report: dict[str, object]) -> str: + totals = report["totals"] + lines = [ + f"Records: {totals['input']} input, {totals['usable']} usable", + "Outcomes: " + ", ".join(f"{name}={totals[name]}" for name in OUTCOMES), + ] + for group in report["groups"]: + lines.extend( + ( + "", + f"Group: channel={group['channel']} direction={group['direction']}", + " Outcomes: " + ", ".join(f"{name}={group['totals'][name]}" for name in OUTCOMES), + " Observed: " + + ", ".join( + f"{stage}={group['progression']['observed'][stage]}" for stage in DELIVERY_STAGES + ), + " Delivery transitions:", + ) + ) + for transition in group["progression"]["transitions"]: + lines.append( + f" {transition['from']} -> {transition['to']}: " + f"{_percent(transition['dropoff_pct'])} " + f"({transition['completed']}/{transition['eligible']} completed)" + ) + if group["engagement"] is not None: + engagement = group["engagement"] + lines.append(" Engagement:") + lines.append( + " DELIVERED -> READ: " + f"{_percent(engagement['dropoff_pct'])} " + f"({engagement['read']}/{engagement['delivered']} read)" + ) + if report["error_codes"] is not None: + lines.append("") + lines.append("ERR_* codes:") + if report["error_codes"]: + lines.extend(f" {code}: {count}" for code, count in report["error_codes"].items()) + else: + lines.append(" (none found)") + if report["diagnostics"]: + lines.append("") + lines.append(f"Validation diagnostics: {len(report['diagnostics'])} record(s)") + for diagnostic in report["diagnostics"]: + lines.append(f" record {diagnostic['record']}: " + "; ".join(diagnostic["messages"])) + lines.append("") + if not totals["usable"]: + lines.append("INSUFFICIENT: no usable analysis cohort exists.") + elif report["healthy"] is None: + lines.append("INDETERMINATE: outcomes are usable, but no explicit transition denominator exists.") + elif report["healthy"]: + lines.append(f"OK: no explicit transition exceeded {report['threshold_pct']}% drop-off.") + else: + lines.append( + f"FAIL: {len(report['anomalies'])} explicit transition(s) exceeded " + f"{report['threshold_pct']}% drop-off." + ) + return "\n".join(lines) def main(argv: list[str] | None = None) -> int: args = _parse_args(sys.argv[1:] if argv is None else argv) + if not 0 <= args.threshold <= 100: + print("error: --threshold must be between 0 and 100", file=sys.stderr) + return 2 try: - messages = _load_messages(Path(args.path)) + records = _load_messages(Path(args.path)) except InputError as exc: print(f"error: {exc}", file=sys.stderr) return 2 - if not messages: - print("error: no messages found in input", file=sys.stderr) - return 2 - - counts = compute_funnel(messages) - drops = stage_dropoffs(counts) - _print_report(counts, drops) - if args.show_errors: - _print_error_summary(summarise_errors(messages)) + report = analyze(records, args.threshold, args.show_errors) + if args.format == "json": + print(json.dumps(report, indent=2, sort_keys=True)) + else: + print(render_text(report)) - anomalies = [(a, b, pct) for a, b, pct in drops if pct > args.threshold] - if anomalies: - print( - f"\nFAIL: {len(anomalies)} stage(s) exceeded the {args.threshold}% drop-off threshold:", - file=sys.stderr, - ) - for a, b, pct in anomalies: - print(f" {a} -> {b}: {pct:.1f}%", file=sys.stderr) + if report["totals"]["usable"] == 0: + if args.format == "text": + print("error: no usable analysis cohort exists", file=sys.stderr) + return 2 + if report["healthy"] is False: + if args.format == "text": + print(f"FAIL: {len(report['anomalies'])} transition threshold breach(es)", file=sys.stderr) return 3 - print(f"\nOK: no stage exceeded the {args.threshold}% drop-off threshold.") return 0 if __name__ == "__main__": - sys.exit(main()) + raise SystemExit(main()) diff --git a/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json b/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json index 038028d..e8bbd98 100644 --- a/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json +++ b/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json @@ -1,502 +1,118 @@ [ { - "message_id": "msg_test_001", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_002", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_003", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_004", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_005", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_010", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "message_id": "msg_test_101", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_102", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_103", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_104", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_105", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_TEMPLATE_PARAMS_INVALID" + }, + { + "message_id": "msg_test_106", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_107", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_108", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_109", + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"} + ] + }, + { + "message_id": "msg_test_110", + "channel": "whatsapp", + "direction": "inbound", + "status": "RECEIVED" + }, + "malformed synthetic record", + { + "message_id": "msg_test_112", + "channel": "sms", + "direction": "outbound", + "status": "MYSTERY" } ] diff --git a/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json b/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json index 56d3784..0b66c89 100644 --- a/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json +++ b/packages/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json @@ -2,501 +2,111 @@ { "message_id": "msg_test_001", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_002", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_003", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_004", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_005", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "satellite", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_010", "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "direction": "inbound", + "status": "RECEIVED" } ] diff --git a/plugins/sent/skills/messaging-performance-analyzer/SKILL.md b/plugins/sent/skills/messaging-performance-analyzer/SKILL.md index c7b0d4d..3898887 100644 --- a/plugins/sent/skills/messaging-performance-analyzer/SKILL.md +++ b/plugins/sent/skills/messaging-performance-analyzer/SKILL.md @@ -48,13 +48,15 @@ Capture these dimensions before calculating anything: profile or sender identity ### 2. Build cohorts from Sent message IDs -Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once at its latest status, then add recipient-level or campaign-level rollups only after deduplication. +Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once in a terminal-outcome rollup, then add recipient-level or campaign-level rollups only after deduplication. + +Distinguish an activity history from a latest-status snapshot. A history can prove the transitions it contains. A snapshot such as `status=DELIVERED` proves only the observed current outcome; it does not prove that the export also observed `QUEUED`, `ROUTED`, or `SENT`. Report unavailable transition denominators as `N/A`, not zero, and never synthesize missing transitions. Do not use provider IDs such as WhatsApp `wamid`, SMS carrier IDs, or RCS message IDs as the primary join key unless the exported evidence lacks Sent IDs. Provider IDs are useful for escalation, but the Sent API and dashboard track status by Sent message ID. ### 3. Normalize lifecycle stages to Sent’s documented statuses -Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, `DELIVERED`, and `READ` for WhatsApp and RCS. Keep failed and error states in a separate terminal bucket using the exact status/error fields present in the evidence. +Use Sent’s documented delivery lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, and `DELIVERED`. Treat `READ` as a separate engagement measure for WhatsApp and RCS, never as an SMS delivery requirement. Keep terminal failures, deferred/in-flight messages, inbound `RECEIVED` messages, and malformed/unknown records in separate buckets using only fields present in the evidence. | Stage | Interpretation | Common diagnostic question | |---|---|---| @@ -62,7 +64,7 @@ Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, | `ROUTED` | Sent selected a channel/provider path. | Did routing choose the expected channel or fallback path? | | `SENT` | The message left Sent/provider processing toward the destination network. | Are provider accepts high but downstream delivery low? | | `DELIVERED` | Delivery was confirmed where supported. | Did the destination network confirm receipt? | -| `READ` | WhatsApp/RCS read receipt was observed where available. | Did users open the message after delivery? | +| `READ` | WhatsApp/RCS engagement receipt was observed where available. | Did users open the message after delivery? | | Error/failure | A terminal or recoverable error occurred. | Is the root cause compliance, payload, throughput, opt-out, or provider outage? | ### 4. Check webhook health before diagnosing delivery @@ -83,7 +85,7 @@ SMS, WhatsApp, and RCS fail differently. Do not average them together unless the ### 6. Quantify impact before recommending fixes -Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. +Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Reconcile the global totals with every channel × direction group, retaining explicit `unknown` groups instead of silently dropping incomplete dimensions. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. A practical analysis table should include: sent count, latest status distribution, failure count, failure-rate delta versus baseline, top exact error strings/codes, first observed timestamp, affected templates, affected countries, and affected profiles. @@ -108,7 +110,10 @@ Do not mistake broadcast for fallback. Omitted `channel` or `["sent"]` enables a - [ ] The analysis uses Sent `message_id` values as the primary unit. - [ ] The cohort is pinned by time window, profile/sender identity, template, channel, and recipient segment. - [ ] Status math uses the latest known status per Sent message ID. +- [ ] Transition math uses observed activity histories and never backfills stages from a latest-only status. - [ ] Pending or in-flight messages are either excluded or reported separately. +- [ ] SMS delivery analysis stops at `DELIVERED`; WhatsApp/RCS `READ` is labeled engagement. +- [ ] Global and channel × direction totals reconcile, including malformed and explicit `unknown` buckets. - [ ] Webhook configuration, event history, and endpoint test results are checked when the symptom is missing callbacks. - [ ] Channel-specific failures are split before aggregate rates are reported. - [ ] Provider or carrier codes are quoted exactly as observed and not invented from a lookup table. @@ -132,7 +137,7 @@ Use the `sent` skill for shared Sent terminology and routing. |---|---|---| | `references/mdr-status-codes.md` | Lookup table | Normalize observed SMS, WhatsApp, and RCS provider errors without putting long code dictionaries in the skill body. | | `references/performance-diagnosis-playbook.md` | Worked examples | Decision tree for which signal to investigate first, channel-specific diagnostic patterns, cross-skill handoff matrix, and escalation criteria. | -| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), prints per-stage counts and drop-off percentages, exits non-zero on anomalies. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N` to tune, default 20; pass `--show-errors` to also tally `ERR_*` codes parsed from FAILED message `description` fields). | +| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), groups channel × direction outcomes, separates delivery transitions from engagement, and retains malformed/unknown rows. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N`, `--show-errors`, or `--format json`). Exit `0` means no observed transition breach, `2` means bad input/no usable cohort, and `3` means an observed breach. JSON uses `null` where a denominator is unavailable; text uses `N/A`. | | `scripts/fixtures/good.json` | Fixture | Synthetic healthy-funnel MDR export. | | `scripts/fixtures/bad.json` | Fixture | Synthetic MDR export with deliberate >50% SENT→DELIVERED drop. | diff --git a/plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md b/plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md index 36ead53..44d8175 100644 --- a/plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md +++ b/plugins/sent/skills/messaging-performance-analyzer/references/mdr-status-codes.md @@ -9,7 +9,7 @@ Authoritative upstream sources (for the downstream provider codes that may appea ## Message status lifecycle -Sent normalizes all channels into a single state machine. Only one terminal at a time; only the **latest** status per `message_id` is meaningful when computing funnel counts. +Sent normalizes channels into a shared delivery state machine. The latest status per `message_id` determines its current outcome, while an activity history supplies the evidence for transition counts. Do not backfill earlier stages from a latest-only record. ``` QUEUED -> ROUTED -> SENT -> DELIVERED -> READ (WhatsApp & RCS only) @@ -27,7 +27,7 @@ FAILED FAILED FAILED FAILED | `FAILED` | Terminal failure; the per-message reason is in the `description` field. | | `RECEIVED` | Inbound message from end user. | -A message can transition `SENT -> DELIVERED -> FAILED` (e.g. expired WhatsApp window, capability lost on RCS); count the latest status, not the journey. +A message can transition `SENT -> DELIVERED -> FAILED`; classify its terminal outcome from the final observed event while retaining the actual journey for transition analysis. `READ` is WhatsApp/RCS engagement, not an SMS delivery stage. ## Synchronous errors (HTTP response body) @@ -207,7 +207,10 @@ Carriers don't share an enum; the categories you actually need to triage on: ## Counting rules - **Use Sent `message_id`** as the primary unit. Provider IDs (carrier message IDs, `wamid`, RBM `messageId`) are useful for escalation but are **not** in the v3 docs as join keys. -- **Use the latest status** (`max(timestamp)`) — a `FAILED` after `DELIVERED` means `FAILED`; a `READ` after `DELIVERED` means `READ`. -- **Exclude pending** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from rate denominators — they're indeterminate. +- **Use the latest observed event for the outcome** — a `FAILED` after `DELIVERED` is a terminal failure; a `READ` after `DELIVERED` is a delivered message with observed engagement. +- **Use only explicit history for transitions.** A latest-only `DELIVERED` record does not prove the export observed `QUEUED`, `ROUTED`, or `SENT`; render those transition denominators as unavailable. +- **Separate pending/deferred records** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from terminal rate denominators. +- **Group by channel and direction.** Keep missing or unsupported dimensions in `unknown` buckets so totals reconcile instead of silently excluding them. +- **Stop SMS at `DELIVERED`.** Calculate `READ` engagement only for WhatsApp and RCS. - **Separate channel fan-out.** `POST /v3/messages` with `"channel": ["sms","whatsapp","rcs"]` creates one message per channel; each has its own `message_id` and its own lifecycle. Don't double-count at the recipient level unless the user explicitly asks for recipient-level rollup. - **Honor a minimum cohort size** before drawing conclusions about small rate shifts. A working heuristic is ≥1,000 messages per cohort; below that, noise dominates. This is an analyst rule of thumb, not a Sent API rule. diff --git a/plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md b/plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md index a24bd47..a8b5787 100644 --- a/plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md +++ b/plugins/sent/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md @@ -129,9 +129,11 @@ When escalating, include: account / profile ID, channel, cohort definition (temp Repeat until the symptom is explained or scoped: 1. Pin the cohort (channel × template × country × profile × window). -2. Compute the funnel; identify the broken lifecycle stage (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`/`READ`). -3. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. -4. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. -5. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. -6. Hand off via the matrix above, or escalate to Sent support with the required evidence. -7. Quantify the diagnosis — never "looks better now" without a recomputed funnel. +2. Split channel × direction groups, reconcile them to the input total, and separate progression, terminal failure, deferred, inbound, and malformed/unknown outcomes. +3. Compute delivery transitions only from explicit activity histories (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`). If the export contains latest-only rows, report their outcomes without inventing prior transitions. +4. For WhatsApp/RCS, report `READ` separately as engagement. Stop SMS delivery analysis at `DELIVERED`. +5. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. +6. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. +7. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. +8. Hand off via the matrix above, or escalate to Sent support with the required evidence. +9. Quantify the diagnosis — never "looks better now" without a recomputed funnel. diff --git a/plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py b/plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py index 2df3b5e..50df69f 100644 --- a/plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py +++ b/plugins/sent/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py @@ -1,21 +1,15 @@ #!/usr/bin/env python3 -"""Analyze a Sent MDR export and report funnel drop-off by lifecycle stage. +"""Analyze Sent MDR records by channel, direction, lifecycle, and outcome. -Lifecycle stages (in order): QUEUED, ROUTED, SENT, DELIVERED, READ. - -The script counts distinct messages that reached at least each stage, -computes the drop-off percentage between adjacent stages, and flags any -stage where drop-off exceeds the configured threshold. +The analyzer uses only statuses that are present in each record. A record with +only ``status=DELIVERED`` contributes one observed DELIVERED status; it does not +implicitly contribute QUEUED, ROUTED, or SENT transitions. Full ``statuses`` +histories can contribute explicit transition rates. Exit codes: - 0 - healthy funnel (no stage drops more than --threshold) - 2 - bad arguments / malformed input - 3 - unhealthy funnel (one or more stages exceed the threshold) - -Examples: - python analyze_mdr_funnel.py path/to/mdr.json - python analyze_mdr_funnel.py path/to/mdr.csv --threshold 15 - python analyze_mdr_funnel.py path/to/mdr.json --show-errors + 0 - usable cohort with no threshold breach, or usable outcome-only data + 2 - bad arguments, malformed input, or no usable analysis cohort + 3 - one or more explicit delivery/engagement transitions breach threshold """ from __future__ import annotations @@ -25,217 +19,463 @@ import json import re import sys -from collections import Counter +from collections import Counter, defaultdict +from dataclasses import dataclass from pathlib import Path -from typing import Iterable +from typing import Any, Iterable + + +DELIVERY_STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED") +ENGAGEMENT_STAGE = "READ" +KNOWN_STATUSES = set(DELIVERY_STAGES) | {ENGAGEMENT_STAGE, "FAILED", "DEFERRED", "RECEIVED"} +OUTCOMES: tuple[str, ...] = ( + "progression", + "terminal_failure", + "deferred", + "inbound", + "malformed", + "unknown", +) +SUPPORTED_CHANNELS = {"sms", "whatsapp", "rcs"} +READ_CHANNELS = {"whatsapp", "rcs"} +OUTBOUND_DIRECTIONS = {"outbound", "outgoing", "mt"} +INBOUND_DIRECTIONS = {"inbound", "incoming", "mo"} +_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") +_STATUS_KEYS = ("status", "stage", "latest_status") -# Ordered lifecycle stages. A message that reached stage N is counted as -# having reached every earlier stage as well. -STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED", "READ") -STAGE_RANK: dict[str, int] = {s: i for i, s in enumerate(STAGES)} + +class InputError(Exception): + """Raised for missing or structurally invalid input. Maps to exit code 2.""" + + +@dataclass(frozen=True) +class NormalizedRecord: + """One input row normalized without inventing lifecycle evidence.""" + + index: int + channel: str + direction: str + observed_statuses: tuple[str, ...] + has_history: bool + outcome: str + diagnostics: tuple[str, ...] + error_codes: tuple[str, ...] def _parse_args(argv: list[str]) -> argparse.Namespace: - parser = argparse.ArgumentParser( - prog="analyze_mdr_funnel.py", - description=( - "Analyze a Sent MDR export and report funnel drop-off by " - "lifecycle stage (QUEUED -> ROUTED -> SENT -> DELIVERED -> READ)." - ), - ) - parser.add_argument( - "path", - help="Path to the MDR export (CSV or JSON; format auto-detected by extension).", - ) + parser = argparse.ArgumentParser(prog="analyze_mdr_funnel.py", description=__doc__) + parser.add_argument("path", help="MDR export path (.json or .csv)") parser.add_argument( "--threshold", type=float, default=20.0, - help=( - "Drop-off percentage threshold per stage. Any stage with a " - "drop-off greater than this value triggers a non-zero exit. " - "Default: 20." - ), + help="Explicit transition drop-off percentage that triggers exit 3 (default: 20)", ) parser.add_argument( "--show-errors", action="store_true", - help=( - "When set, also summarise the count of each Sent send-time error " - "code (ERR_*) parsed from the 'description' field of FAILED " - "messages. Non-FAILED rows and rows without a description are " - "ignored. Does not change exit codes." - ), + help="Include ERR_* codes from FAILED record descriptions", + ) + parser.add_argument( + "--format", + choices=("text", "json"), + default="text", + help="Output format (default: text)", ) return parser.parse_args(argv) -class InputError(Exception): - """Raised for malformed or missing input. Mapped to exit code 2.""" - - -def _load_messages(path: Path) -> list[dict]: +def _load_messages(path: Path) -> list[object]: if not path.is_file(): raise InputError(f"file not found: {path}") - ext = path.suffix.lower() - if ext == ".json": + extension = path.suffix.lower() + if extension == ".json": try: - with path.open("r", encoding="utf-8") as fh: - data = json.load(fh) + data = json.loads(path.read_text(encoding="utf-8")) + except OSError as exc: + raise InputError(f"could not read {path}: {exc}") from exc except json.JSONDecodeError as exc: raise InputError(f"invalid JSON in {path}: {exc}") from exc if isinstance(data, dict) and "messages" in data: data = data["messages"] if not isinstance(data, list): - raise InputError("JSON must be a list of message records or {\"messages\": [...]}") - messages = data - if ext == ".csv": + raise InputError('JSON must be a list of message records or {"messages": [...]}') + return data + if extension == ".csv": try: - with path.open("r", encoding="utf-8", newline="") as fh: - messages = list(csv.DictReader(fh)) - except (csv.Error, UnicodeError) as exc: + with path.open("r", encoding="utf-8", newline="") as handle: + return list(csv.DictReader(handle)) + except (OSError, csv.Error, UnicodeError) as exc: raise InputError(f"invalid CSV in {path}: {exc}") from exc - elif ext != ".json": - raise InputError(f"unsupported file extension '{ext}'; expected .json or .csv") - for index, record in enumerate(messages, 1): - if not isinstance(record, dict): - raise InputError(f"record {index} must be an object") - return messages - - -def _latest_stage(record: dict) -> str | None: - """Return the furthest stage a message reached, or None if unknown. - - Accepts either a single `status`/`stage` field, or a `statuses` list of - {stage, ...} entries (treated as the lifecycle history; the max-rank stage - is the latest in funnel-progression sense). - """ - if "statuses" in record and isinstance(record["statuses"], list): - ranks = [ - STAGE_RANK[s["stage"].upper()] - for s in record["statuses"] - if isinstance(s, dict) and isinstance(s.get("stage"), str) and s["stage"].upper() in STAGE_RANK - ] - if not ranks: - return None - return STAGES[max(ranks)] - for key in ("status", "stage", "latest_status"): - val = record.get(key) - if isinstance(val, str) and val.upper() in STAGE_RANK: - return val.upper() - return None - - -def compute_funnel(messages: Iterable[dict]) -> dict[str, int]: - """Return {stage: count_of_messages_that_reached_at_least_this_stage}.""" - counts = {s: 0 for s in STAGES} - for record in messages: - latest = _latest_stage(record) - if latest is None: - continue - for stage in STAGES[: STAGE_RANK[latest] + 1]: - counts[stage] += 1 - return counts - - -def stage_dropoffs(counts: dict[str, int]) -> list[tuple[str, str, float]]: - """Return [(from_stage, to_stage, dropoff_pct)] for adjacent stages.""" - out: list[tuple[str, str, float]] = [] - for i in range(len(STAGES) - 1): - a, b = STAGES[i], STAGES[i + 1] - if counts[a] == 0: - pct = 0.0 - else: - pct = (counts[a] - counts[b]) / counts[a] * 100.0 - out.append((a, b, pct)) - return out + raise InputError(f"unsupported file extension '{extension}'; expected .json or .csv") + + +def _field(record: dict[str, Any], name: str) -> Any: + if name in record: + return record[name] + payload = record.get("payload") + return payload.get(name) if isinstance(payload, dict) else None + + +def _normalize_channel(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("channel missing; grouped as unknown") + return "unknown" + channel = value.strip().lower() + if channel not in SUPPORTED_CHANNELS: + diagnostics.append(f"unsupported channel {value!r}; grouped as unknown") + return "unknown" + return channel + + +def _normalize_direction(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("direction missing; grouped as unknown") + return "unknown" + direction = value.strip().lower() + if direction in OUTBOUND_DIRECTIONS: + return "outbound" + if direction in INBOUND_DIRECTIONS: + return "inbound" + diagnostics.append(f"unsupported direction {value!r}; grouped as unknown") + return "unknown" + + +def _normalize_statuses(record: dict[str, Any], diagnostics: list[str]) -> tuple[tuple[str, ...], bool]: + if "statuses" in record: + history = record["statuses"] + if not isinstance(history, list): + diagnostics.append("statuses must be an array") + return (), False + statuses: list[str] = [] + for position, entry in enumerate(history, 1): + if not isinstance(entry, dict): + diagnostics.append(f"statuses[{position}] must be an object") + continue + value = next((entry.get(key) for key in _STATUS_KEYS if isinstance(entry.get(key), str)), None) + if value is None or not value.strip(): + diagnostics.append(f"statuses[{position}] has no status/stage value") + continue + statuses.append(value.strip().upper()) + return tuple(statuses), True + for key in _STATUS_KEYS: + value = _field(record, key) + if isinstance(value, str) and value.strip(): + return (value.strip().upper(),), False + diagnostics.append("record has no observed status") + return (), False + + +def normalize_record(record: object, index: int) -> NormalizedRecord: + if not isinstance(record, dict): + return NormalizedRecord( + index=index, + channel="unknown", + direction="unknown", + observed_statuses=(), + has_history=False, + outcome="malformed", + diagnostics=("record must be an object",), + error_codes=(), + ) -def _print_report(counts: dict[str, int], drops: list[tuple[str, str, float]]) -> None: - print("Funnel:") - for stage in STAGES: - print(f" {stage:<10} {counts[stage]}") - print("\nStage drop-off:") - for a, b, pct in drops: - print(f" {a} -> {b}: {pct:.1f}%") + diagnostics: list[str] = [] + channel = _normalize_channel(_field(record, "channel"), diagnostics) + direction = _normalize_direction(_field(record, "direction"), diagnostics) + statuses, has_history = _normalize_statuses(record, diagnostics) + unknown_statuses = sorted({status for status in statuses if status not in KNOWN_STATUSES}) + if unknown_statuses: + diagnostics.append("unknown status values: " + ", ".join(unknown_statuses)) + + known = tuple(status for status in statuses if status in KNOWN_STATUSES) + latest = known[-1] if known else None + if direction == "inbound" or latest == "RECEIVED": + direction = "inbound" + outcome = "inbound" + elif latest == "FAILED": + outcome = "terminal_failure" + elif latest == "DEFERRED" or latest in {"QUEUED", "ROUTED", "SENT"}: + outcome = "deferred" + elif latest in {"DELIVERED", "READ"}: + outcome = "progression" + else: + outcome = "unknown" + + if channel == "sms" and "READ" in known: + diagnostics.append("READ is not used for SMS delivery or engagement analysis") + description = _field(record, "description") + error_codes = tuple(sorted(set(_ERR_CODE_RE.findall(description)))) if isinstance(description, str) else () + return NormalizedRecord( + index=index, + channel=channel, + direction=direction, + observed_statuses=statuses, + has_history=has_history, + outcome=outcome, + diagnostics=tuple(diagnostics), + error_codes=error_codes, + ) -# Sent's documented send-time per-message error codes appear in the message -# `description` field on FAILED rows. See references/mdr-status-codes.md. -_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") -_STATUS_KEYS = ("status", "stage", "latest_status") +def _completed_transition(statuses: tuple[str, ...], source: str, target: str) -> bool: + try: + source_index = statuses.index(source) + except ValueError: + return False + return target in statuses[source_index + 1 :] + + +def _transition( + records: list[NormalizedRecord], source: str, target: str, threshold: float +) -> dict[str, object]: + terminal = {"progression", "terminal_failure"} + eligible_records = [ + record + for record in records + if record.has_history and record.outcome in terminal and source in record.observed_statuses + ] + completed = sum( + _completed_transition(record.observed_statuses, source, target) for record in eligible_records + ) + eligible = len(eligible_records) + dropoff = round((eligible - completed) / eligible * 100.0, 4) if eligible else None + return { + "from": source, + "to": target, + "eligible": eligible, + "completed": completed, + "dropoff_pct": dropoff, + "exceeds_threshold": dropoff is not None and dropoff > threshold, + } + + +def _totals(records: Iterable[NormalizedRecord]) -> dict[str, int]: + records = list(records) + counts = Counter(record.outcome for record in records) + return { + "input": len(records), + "usable": sum(counts[name] for name in ("progression", "terminal_failure", "deferred")), + **{name: counts[name] for name in OUTCOMES}, + } + + +def _group_report( + channel: str, + direction: str, + records: list[NormalizedRecord], + threshold: float, +) -> dict[str, object]: + delivery_statuses = { + stage: sum(stage in record.observed_statuses for record in records) for stage in DELIVERY_STAGES + } + transitions = [ + _transition(records, source, target, threshold) + for source, target in zip(DELIVERY_STAGES, DELIVERY_STAGES[1:]) + ] + engagement: dict[str, object] | None = None + if channel in READ_CHANNELS: + read_transition = _transition(records, "DELIVERED", "READ", threshold) + engagement = { + "delivered": read_transition["eligible"], + "read": read_transition["completed"], + "read_rate_pct": ( + round(100.0 - float(read_transition["dropoff_pct"]), 4) + if read_transition["dropoff_pct"] is not None + else None + ), + "dropoff_pct": read_transition["dropoff_pct"], + "exceeds_threshold": read_transition["exceeds_threshold"], + } + return { + "channel": channel, + "direction": direction, + "totals": _totals(records), + "progression": {"observed": delivery_statuses, "transitions": transitions}, + "engagement": engagement, + } + + +def build_report( + records: list[NormalizedRecord], + threshold: float, + include_errors: bool, +) -> dict[str, object]: + grouped: dict[tuple[str, str], list[NormalizedRecord]] = defaultdict(list) + for record in records: + grouped[(record.channel, record.direction)].append(record) + groups = [ + _group_report(channel, direction, grouped[(channel, direction)], threshold) + for channel, direction in sorted(grouped) + ] + anomalies: list[dict[str, object]] = [] + evaluated = 0 + for group in groups: + for transition in group["progression"]["transitions"]: + if transition["dropoff_pct"] is not None: + evaluated += 1 + if transition["exceeds_threshold"]: + anomalies.append( + { + "kind": "delivery", + "channel": group["channel"], + "direction": group["direction"], + **transition, + } + ) + engagement = group["engagement"] + if engagement is not None: + if engagement["dropoff_pct"] is not None: + evaluated += 1 + if engagement["exceeds_threshold"]: + anomalies.append( + { + "kind": "engagement", + "channel": group["channel"], + "direction": group["direction"], + "from": "DELIVERED", + "to": "READ", + "eligible": engagement["delivered"], + "completed": engagement["read"], + "dropoff_pct": engagement["dropoff_pct"], + "exceeds_threshold": True, + } + ) + totals = _totals(records) + healthy: bool | None + if not totals["usable"] or not evaluated: + healthy = None + else: + healthy = not anomalies + diagnostics = [ + {"record": record.index, "messages": list(record.diagnostics)} + for record in records + if record.diagnostics + ] + error_codes = Counter( + code + for record in records + if record.outcome == "terminal_failure" + for code in record.error_codes + ) + return { + "schema_version": 1, + "threshold_pct": threshold, + "healthy": healthy, + "totals": totals, + "groups": groups, + "anomalies": anomalies, + "diagnostics": diagnostics, + "error_codes": dict(sorted(error_codes.items())) if include_errors else None, + } -def _is_failed(record: dict) -> bool: - # FAILED isn't in STAGES, so _latest_stage() returns None for it. Check - # the explicit status keys and the statuses history directly. - for key in _STATUS_KEYS: - val = record.get(key) - if isinstance(val, str) and val.upper() == "FAILED": - return True - history = record.get("statuses") - if isinstance(history, list): - for entry in history: - if isinstance(entry, dict) and isinstance(entry.get("stage"), str) \ - and entry["stage"].upper() == "FAILED": - return True - return False +def analyze(records: Iterable[object], threshold: float, include_errors: bool) -> dict[str, object]: + normalized = [normalize_record(record, index) for index, record in enumerate(records, 1)] + return build_report(normalized, threshold, include_errors) def summarise_errors(messages: Iterable[dict]) -> Counter: - """Return a Counter of ERR_* codes parsed from FAILED messages' descriptions.""" + """Return ERR_* code counts from records whose observed outcome is FAILED.""" counter: Counter = Counter() - for record in messages: - if not isinstance(record, dict) or not _is_failed(record): - continue - description = record.get("description") - if not isinstance(description, str): - continue - # Dedupe within a row so one message with the same code mentioned - # twice doesn't double-count. - counter.update(set(_ERR_CODE_RE.findall(description))) + for index, record in enumerate(messages, 1): + normalized = normalize_record(record, index) + if normalized.outcome == "terminal_failure": + counter.update(normalized.error_codes) return counter -def _print_error_summary(counter: Counter) -> None: - print("\nERR_* codes (from FAILED descriptions):") - if not counter: - print(" (none found)") - return - for code, count in counter.most_common(): - print(f" {code:<32} {count}") +def _percent(value: object) -> str: + return "N/A" if value is None else f"{float(value):.1f}%" + + +def render_text(report: dict[str, object]) -> str: + totals = report["totals"] + lines = [ + f"Records: {totals['input']} input, {totals['usable']} usable", + "Outcomes: " + ", ".join(f"{name}={totals[name]}" for name in OUTCOMES), + ] + for group in report["groups"]: + lines.extend( + ( + "", + f"Group: channel={group['channel']} direction={group['direction']}", + " Outcomes: " + ", ".join(f"{name}={group['totals'][name]}" for name in OUTCOMES), + " Observed: " + + ", ".join( + f"{stage}={group['progression']['observed'][stage]}" for stage in DELIVERY_STAGES + ), + " Delivery transitions:", + ) + ) + for transition in group["progression"]["transitions"]: + lines.append( + f" {transition['from']} -> {transition['to']}: " + f"{_percent(transition['dropoff_pct'])} " + f"({transition['completed']}/{transition['eligible']} completed)" + ) + if group["engagement"] is not None: + engagement = group["engagement"] + lines.append(" Engagement:") + lines.append( + " DELIVERED -> READ: " + f"{_percent(engagement['dropoff_pct'])} " + f"({engagement['read']}/{engagement['delivered']} read)" + ) + if report["error_codes"] is not None: + lines.append("") + lines.append("ERR_* codes:") + if report["error_codes"]: + lines.extend(f" {code}: {count}" for code, count in report["error_codes"].items()) + else: + lines.append(" (none found)") + if report["diagnostics"]: + lines.append("") + lines.append(f"Validation diagnostics: {len(report['diagnostics'])} record(s)") + for diagnostic in report["diagnostics"]: + lines.append(f" record {diagnostic['record']}: " + "; ".join(diagnostic["messages"])) + lines.append("") + if not totals["usable"]: + lines.append("INSUFFICIENT: no usable analysis cohort exists.") + elif report["healthy"] is None: + lines.append("INDETERMINATE: outcomes are usable, but no explicit transition denominator exists.") + elif report["healthy"]: + lines.append(f"OK: no explicit transition exceeded {report['threshold_pct']}% drop-off.") + else: + lines.append( + f"FAIL: {len(report['anomalies'])} explicit transition(s) exceeded " + f"{report['threshold_pct']}% drop-off." + ) + return "\n".join(lines) def main(argv: list[str] | None = None) -> int: args = _parse_args(sys.argv[1:] if argv is None else argv) + if not 0 <= args.threshold <= 100: + print("error: --threshold must be between 0 and 100", file=sys.stderr) + return 2 try: - messages = _load_messages(Path(args.path)) + records = _load_messages(Path(args.path)) except InputError as exc: print(f"error: {exc}", file=sys.stderr) return 2 - if not messages: - print("error: no messages found in input", file=sys.stderr) - return 2 - - counts = compute_funnel(messages) - drops = stage_dropoffs(counts) - _print_report(counts, drops) - if args.show_errors: - _print_error_summary(summarise_errors(messages)) + report = analyze(records, args.threshold, args.show_errors) + if args.format == "json": + print(json.dumps(report, indent=2, sort_keys=True)) + else: + print(render_text(report)) - anomalies = [(a, b, pct) for a, b, pct in drops if pct > args.threshold] - if anomalies: - print( - f"\nFAIL: {len(anomalies)} stage(s) exceeded the {args.threshold}% drop-off threshold:", - file=sys.stderr, - ) - for a, b, pct in anomalies: - print(f" {a} -> {b}: {pct:.1f}%", file=sys.stderr) + if report["totals"]["usable"] == 0: + if args.format == "text": + print("error: no usable analysis cohort exists", file=sys.stderr) + return 2 + if report["healthy"] is False: + if args.format == "text": + print(f"FAIL: {len(report['anomalies'])} transition threshold breach(es)", file=sys.stderr) return 3 - print(f"\nOK: no stage exceeded the {args.threshold}% drop-off threshold.") return 0 if __name__ == "__main__": - sys.exit(main()) + raise SystemExit(main()) diff --git a/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json b/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json index 038028d..e8bbd98 100644 --- a/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json +++ b/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/bad.json @@ -1,502 +1,118 @@ [ { - "message_id": "msg_test_001", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_002", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_003", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_004", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_005", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_010", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "message_id": "msg_test_101", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_102", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_103", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_104", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_105", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_TEMPLATE_PARAMS_INVALID" + }, + { + "message_id": "msg_test_106", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_107", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_108", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_109", + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"} + ] + }, + { + "message_id": "msg_test_110", + "channel": "whatsapp", + "direction": "inbound", + "status": "RECEIVED" + }, + "malformed synthetic record", + { + "message_id": "msg_test_112", + "channel": "sms", + "direction": "outbound", + "status": "MYSTERY" } ] diff --git a/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json b/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json index 56d3784..0b66c89 100644 --- a/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json +++ b/plugins/sent/skills/messaging-performance-analyzer/scripts/fixtures/good.json @@ -2,501 +2,111 @@ { "message_id": "msg_test_001", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_002", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_003", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_004", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_005", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "satellite", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_010", "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "direction": "inbound", + "status": "RECEIVED" } ] diff --git a/scripts/test_fixtures.py b/scripts/test_fixtures.py index e9e535a..5a24fc1 100644 --- a/scripts/test_fixtures.py +++ b/scripts/test_fixtures.py @@ -107,18 +107,99 @@ def test_loads_list_and_wrapped_json(self) -> None: path.write_text(json.dumps(value), encoding="utf-8") self.assertEqual(MDR._load_messages(path), records) - def test_rejects_malformed_json_wrong_root_and_invalid_records(self) -> None: + def test_rejects_malformed_json_and_wrong_root(self) -> None: with tempfile.TemporaryDirectory() as directory: path = Path(directory) / "messages.json" - for value in ("{", json.dumps({"unexpected": []}), json.dumps(["not-an-object"])): + for value in ("{", json.dumps({"unexpected": []})): path.write_text(value, encoding="utf-8") with self.subTest(value=value), self.assertRaises(MDR.InputError): MDR._load_messages(path) - def test_threshold_is_strictly_greater_than_boundary(self) -> None: - counts = {"QUEUED": 5, "ROUTED": 5, "SENT": 5, "DELIVERED": 5, "READ": 4} - drops = MDR.stage_dropoffs(counts) - self.assertEqual(drops[-1], ("DELIVERED", "READ", 20.0)) + def test_latest_only_record_never_infers_earlier_transitions(self) -> None: + normalized = MDR.normalize_record( + {"message_id": "msg_test_001", "channel": "whatsapp", "status": "DELIVERED"}, + 1, + ) + self.assertEqual(normalized.observed_statuses, ("DELIVERED",)) + self.assertFalse(normalized.has_history) + report = MDR.build_report([normalized], threshold=20.0, include_errors=False) + group = report["groups"][0] + self.assertEqual(group["progression"]["observed"], {"QUEUED": 0, "ROUTED": 0, "SENT": 0, "DELIVERED": 1}) + self.assertTrue(all(item["dropoff_pct"] is None for item in group["progression"]["transitions"])) + self.assertIsNone(report["healthy"]) + + def test_groups_by_channel_and_direction_with_unknown_buckets(self) -> None: + records = [ + {"channel": "sms", "direction": "outbound", "status": "DELIVERED"}, + {"channel": "whatsapp", "direction": "inbound", "status": "RECEIVED"}, + {"channel": "satellite", "status": "SENT"}, + ] + report = MDR.analyze(records, threshold=20.0, include_errors=False) + keys = {(group["channel"], group["direction"]) for group in report["groups"]} + self.assertEqual(keys, {("sms", "outbound"), ("whatsapp", "inbound"), ("unknown", "unknown")}) + self.assertEqual(report["totals"]["input"], 3) + self.assertEqual(sum(group["totals"]["input"] for group in report["groups"]), 3) + self.assertEqual(sum(report["totals"][name] for name in MDR.OUTCOMES), 3) + + def test_full_history_drives_transition_rates_and_terminal_outcomes(self) -> None: + records = [ + { + "channel": "whatsapp", + "direction": "outbound", + "statuses": [{"stage": stage} for stage in ("QUEUED", "ROUTED", "SENT", "DELIVERED", "READ")], + }, + { + "channel": "whatsapp", + "direction": "outbound", + "statuses": [{"stage": stage} for stage in ("QUEUED", "ROUTED", "SENT", "FAILED")], + }, + { + "channel": "whatsapp", + "direction": "outbound", + "statuses": [{"stage": stage} for stage in ("QUEUED", "ROUTED", "SENT")], + }, + ] + report = MDR.analyze(records, threshold=20.0, include_errors=False) + group = report["groups"][0] + self.assertEqual(group["totals"]["progression"], 1) + self.assertEqual(group["totals"]["terminal_failure"], 1) + self.assertEqual(group["totals"]["deferred"], 1) + sent_to_delivered = next( + item for item in group["progression"]["transitions"] if item["from"] == "SENT" + ) + self.assertEqual(sent_to_delivered["eligible"], 2) + self.assertEqual(sent_to_delivered["completed"], 1) + self.assertEqual(sent_to_delivered["dropoff_pct"], 50.0) + self.assertFalse(report["healthy"]) + + def test_sms_stops_at_delivery_and_read_is_channel_engagement(self) -> None: + records = [ + { + "channel": "sms", + "direction": "outbound", + "statuses": [{"stage": stage} for stage in ("QUEUED", "ROUTED", "SENT", "DELIVERED")], + }, + { + "channel": "whatsapp", + "direction": "outbound", + "statuses": [{"stage": stage} for stage in ("QUEUED", "ROUTED", "SENT", "DELIVERED", "READ")], + }, + ] + report = MDR.analyze(records, threshold=20.0, include_errors=False) + groups = {(group["channel"], group["direction"]): group for group in report["groups"]} + self.assertIsNone(groups[("sms", "outbound")]["engagement"]) + self.assertEqual(groups[("sms", "outbound")]["progression"]["observed"]["DELIVERED"], 1) + self.assertEqual(groups[("whatsapp", "outbound")]["engagement"]["read_rate_pct"], 100.0) + + def test_zero_denominators_render_as_na_and_json_null(self) -> None: + report = MDR.analyze( + [{"channel": "rcs", "direction": "outbound", "status": "DELIVERED"}], + threshold=20.0, + include_errors=False, + ) + rendered = MDR.render_text(report) + self.assertIn("N/A", rendered) + self.assertIn('"dropoff_pct": null', json.dumps(report)) def test_error_summary_only_counts_failed_records(self) -> None: records = [ @@ -131,6 +212,18 @@ def test_error_summary_only_counts_failed_records(self) -> None: {"ERR_ROUTE_DENIED": 1, "ERR_CONSENT_BLOCKED": 1}, ) + def test_malformed_and_unknown_records_are_retained_in_totals(self) -> None: + report = MDR.analyze( + ["not-an-object", {"channel": "sms", "direction": "outbound", "status": "MYSTERY"}], + threshold=20.0, + include_errors=True, + ) + self.assertEqual(report["totals"]["malformed"], 1) + self.assertEqual(report["totals"]["unknown"], 1) + self.assertEqual(report["totals"]["usable"], 0) + self.assertIsNone(report["healthy"]) + self.assertEqual(len(report["diagnostics"]), 2) + class DirectEvidencePacketTests(unittest.TestCase): def setUp(self) -> None: @@ -326,13 +419,37 @@ def test_mdr_rejects_unsupported_extension_and_invalid_records(self) -> None: invalid.write_text('["not-an-object"]', encoding="utf-8") result = self.run_cli(MDR_ROOT / "analyze_mdr_funnel.py", invalid) self.assertEqual(result.returncode, 2) - self.assertIn("record 1 must be an object", result.stderr) + self.assertIn("no usable analysis cohort", result.stderr) self.assertNotIn("Traceback", result.stderr) def test_mdr_exact_threshold_is_healthy(self) -> None: records = [ - *({"message_id": f"msg_test_{index}", "statuses": [{"stage": "READ"}]} for index in range(4)), - {"message_id": "msg_test_5", "statuses": [{"stage": "DELIVERED"}]}, + *( + { + "message_id": f"msg_test_{index}", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"}, + ], + } + for index in range(4) + ), + { + "message_id": "msg_test_5", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + ], + }, ] with tempfile.TemporaryDirectory() as directory: path = Path(directory) / "boundary.json" @@ -341,6 +458,36 @@ def test_mdr_exact_threshold_is_healthy(self) -> None: self.assertEqual(result.returncode, 0, (result.stdout, result.stderr)) self.assertIn("DELIVERED -> READ: 20.0%", result.stdout) + def test_mdr_json_output_is_machine_readable_and_all_unknown_is_not_healthy(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "unknown.json" + path.write_text(json.dumps([{"channel": "unknown", "status": "MYSTERY"}]), encoding="utf-8") + result = self.run_cli(MDR_ROOT / "analyze_mdr_funnel.py", path, "--format", "json") + self.assertEqual(result.returncode, 2, (result.stdout, result.stderr)) + report = json.loads(result.stdout) + self.assertIsNone(report["healthy"]) + self.assertEqual(report["totals"]["usable"], 0) + self.assertEqual(result.stderr.strip(), "") + + def test_mdr_json_show_errors_has_stable_error_counts(self) -> None: + records = [ + { + "channel": "sms", + "direction": "outbound", + "statuses": [{"stage": "QUEUED"}, {"stage": "FAILED"}], + "description": "ERR_ROUTE_DENIED", + } + ] + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "failed.json" + path.write_text(json.dumps(records), encoding="utf-8") + result = self.run_cli( + MDR_ROOT / "analyze_mdr_funnel.py", path, "--format", "json", "--show-errors" + ) + self.assertEqual(result.returncode, 3, (result.stdout, result.stderr)) + report = json.loads(result.stdout) + self.assertEqual(report["error_codes"], {"ERR_ROUTE_DENIED": 1}) + class FixturePrivacyTests(unittest.TestCase): def test_checked_in_fixtures_are_synthetic_and_privacy_safe(self) -> None: diff --git a/skills/messaging-performance-analyzer/SKILL.md b/skills/messaging-performance-analyzer/SKILL.md index c7b0d4d..3898887 100644 --- a/skills/messaging-performance-analyzer/SKILL.md +++ b/skills/messaging-performance-analyzer/SKILL.md @@ -48,13 +48,15 @@ Capture these dimensions before calculating anything: profile or sender identity ### 2. Build cohorts from Sent message IDs -Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once at its latest status, then add recipient-level or campaign-level rollups only after deduplication. +Use Sent `message_id` as the primary unit. A v3 send can create separate messages for each recipient and channel pair when multiple channels are specified. Count each Sent message once in a terminal-outcome rollup, then add recipient-level or campaign-level rollups only after deduplication. + +Distinguish an activity history from a latest-status snapshot. A history can prove the transitions it contains. A snapshot such as `status=DELIVERED` proves only the observed current outcome; it does not prove that the export also observed `QUEUED`, `ROUTED`, or `SENT`. Report unavailable transition denominators as `N/A`, not zero, and never synthesize missing transitions. Do not use provider IDs such as WhatsApp `wamid`, SMS carrier IDs, or RCS message IDs as the primary join key unless the exported evidence lacks Sent IDs. Provider IDs are useful for escalation, but the Sent API and dashboard track status by Sent message ID. ### 3. Normalize lifecycle stages to Sent’s documented statuses -Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, `DELIVERED`, and `READ` for WhatsApp and RCS. Keep failed and error states in a separate terminal bucket using the exact status/error fields present in the evidence. +Use Sent’s documented delivery lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, `SENT`, and `DELIVERED`. Treat `READ` as a separate engagement measure for WhatsApp and RCS, never as an SMS delivery requirement. Keep terminal failures, deferred/in-flight messages, inbound `RECEIVED` messages, and malformed/unknown records in separate buckets using only fields present in the evidence. | Stage | Interpretation | Common diagnostic question | |---|---|---| @@ -62,7 +64,7 @@ Use Sent’s documented lifecycle as the first-pass funnel: `QUEUED`, `ROUTED`, | `ROUTED` | Sent selected a channel/provider path. | Did routing choose the expected channel or fallback path? | | `SENT` | The message left Sent/provider processing toward the destination network. | Are provider accepts high but downstream delivery low? | | `DELIVERED` | Delivery was confirmed where supported. | Did the destination network confirm receipt? | -| `READ` | WhatsApp/RCS read receipt was observed where available. | Did users open the message after delivery? | +| `READ` | WhatsApp/RCS engagement receipt was observed where available. | Did users open the message after delivery? | | Error/failure | A terminal or recoverable error occurred. | Is the root cause compliance, payload, throughput, opt-out, or provider outage? | ### 4. Check webhook health before diagnosing delivery @@ -83,7 +85,7 @@ SMS, WhatsApp, and RCS fail differently. Do not average them together unless the ### 6. Quantify impact before recommending fixes -Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. +Report raw counts and rates together. A 40% failure rate over 15 messages is a different decision than a 4% failure rate over 150,000 messages. Reconcile the global totals with every channel × direction group, retaining explicit `unknown` groups instead of silently dropping incomplete dimensions. Include exclusions such as pending messages, test traffic, sandbox sends, retries, and duplicate channel fan-out. A practical analysis table should include: sent count, latest status distribution, failure count, failure-rate delta versus baseline, top exact error strings/codes, first observed timestamp, affected templates, affected countries, and affected profiles. @@ -108,7 +110,10 @@ Do not mistake broadcast for fallback. Omitted `channel` or `["sent"]` enables a - [ ] The analysis uses Sent `message_id` values as the primary unit. - [ ] The cohort is pinned by time window, profile/sender identity, template, channel, and recipient segment. - [ ] Status math uses the latest known status per Sent message ID. +- [ ] Transition math uses observed activity histories and never backfills stages from a latest-only status. - [ ] Pending or in-flight messages are either excluded or reported separately. +- [ ] SMS delivery analysis stops at `DELIVERED`; WhatsApp/RCS `READ` is labeled engagement. +- [ ] Global and channel × direction totals reconcile, including malformed and explicit `unknown` buckets. - [ ] Webhook configuration, event history, and endpoint test results are checked when the symptom is missing callbacks. - [ ] Channel-specific failures are split before aggregate rates are reported. - [ ] Provider or carrier codes are quoted exactly as observed and not invented from a lookup table. @@ -132,7 +137,7 @@ Use the `sent` skill for shared Sent terminology and routing. |---|---|---| | `references/mdr-status-codes.md` | Lookup table | Normalize observed SMS, WhatsApp, and RCS provider errors without putting long code dictionaries in the skill body. | | `references/performance-diagnosis-playbook.md` | Worked examples | Decision tree for which signal to investigate first, channel-specific diagnostic patterns, cross-skill handoff matrix, and escalation criteria. | -| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), prints per-stage counts and drop-off percentages, exits non-zero on anomalies. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N` to tune, default 20; pass `--show-errors` to also tally `ERR_*` codes parsed from FAILED message `description` fields). | +| `scripts/analyze_mdr_funnel.py` | Validation script | Reads an MDR export (CSV or JSON), groups channel × direction outcomes, separates delivery transitions from engagement, and retains malformed/unknown rows. Run from the skill root: `python scripts/analyze_mdr_funnel.py path/to/mdr.csv` (use `--threshold N`, `--show-errors`, or `--format json`). Exit `0` means no observed transition breach, `2` means bad input/no usable cohort, and `3` means an observed breach. JSON uses `null` where a denominator is unavailable; text uses `N/A`. | | `scripts/fixtures/good.json` | Fixture | Synthetic healthy-funnel MDR export. | | `scripts/fixtures/bad.json` | Fixture | Synthetic MDR export with deliberate >50% SENT→DELIVERED drop. | diff --git a/skills/messaging-performance-analyzer/references/mdr-status-codes.md b/skills/messaging-performance-analyzer/references/mdr-status-codes.md index 36ead53..44d8175 100644 --- a/skills/messaging-performance-analyzer/references/mdr-status-codes.md +++ b/skills/messaging-performance-analyzer/references/mdr-status-codes.md @@ -9,7 +9,7 @@ Authoritative upstream sources (for the downstream provider codes that may appea ## Message status lifecycle -Sent normalizes all channels into a single state machine. Only one terminal at a time; only the **latest** status per `message_id` is meaningful when computing funnel counts. +Sent normalizes channels into a shared delivery state machine. The latest status per `message_id` determines its current outcome, while an activity history supplies the evidence for transition counts. Do not backfill earlier stages from a latest-only record. ``` QUEUED -> ROUTED -> SENT -> DELIVERED -> READ (WhatsApp & RCS only) @@ -27,7 +27,7 @@ FAILED FAILED FAILED FAILED | `FAILED` | Terminal failure; the per-message reason is in the `description` field. | | `RECEIVED` | Inbound message from end user. | -A message can transition `SENT -> DELIVERED -> FAILED` (e.g. expired WhatsApp window, capability lost on RCS); count the latest status, not the journey. +A message can transition `SENT -> DELIVERED -> FAILED`; classify its terminal outcome from the final observed event while retaining the actual journey for transition analysis. `READ` is WhatsApp/RCS engagement, not an SMS delivery stage. ## Synchronous errors (HTTP response body) @@ -207,7 +207,10 @@ Carriers don't share an enum; the categories you actually need to triage on: ## Counting rules - **Use Sent `message_id`** as the primary unit. Provider IDs (carrier message IDs, `wamid`, RBM `messageId`) are useful for escalation but are **not** in the v3 docs as join keys. -- **Use the latest status** (`max(timestamp)`) — a `FAILED` after `DELIVERED` means `FAILED`; a `READ` after `DELIVERED` means `READ`. -- **Exclude pending** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from rate denominators — they're indeterminate. +- **Use the latest observed event for the outcome** — a `FAILED` after `DELIVERED` is a terminal failure; a `READ` after `DELIVERED` is a delivered message with observed engagement. +- **Use only explicit history for transitions.** A latest-only `DELIVERED` record does not prove the export observed `QUEUED`, `ROUTED`, or `SENT`; render those transition denominators as unavailable. +- **Separate pending/deferred records** (`QUEUED`/`ROUTED`/`SENT` with no terminal status after the analysis window closes) from terminal rate denominators. +- **Group by channel and direction.** Keep missing or unsupported dimensions in `unknown` buckets so totals reconcile instead of silently excluding them. +- **Stop SMS at `DELIVERED`.** Calculate `READ` engagement only for WhatsApp and RCS. - **Separate channel fan-out.** `POST /v3/messages` with `"channel": ["sms","whatsapp","rcs"]` creates one message per channel; each has its own `message_id` and its own lifecycle. Don't double-count at the recipient level unless the user explicitly asks for recipient-level rollup. - **Honor a minimum cohort size** before drawing conclusions about small rate shifts. A working heuristic is ≥1,000 messages per cohort; below that, noise dominates. This is an analyst rule of thumb, not a Sent API rule. diff --git a/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md b/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md index a24bd47..a8b5787 100644 --- a/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md +++ b/skills/messaging-performance-analyzer/references/performance-diagnosis-playbook.md @@ -129,9 +129,11 @@ When escalating, include: account / profile ID, channel, cohort definition (temp Repeat until the symptom is explained or scoped: 1. Pin the cohort (channel × template × country × profile × window). -2. Compute the funnel; identify the broken lifecycle stage (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`/`READ`). -3. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. -4. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. -5. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. -6. Hand off via the matrix above, or escalate to Sent support with the required evidence. -7. Quantify the diagnosis — never "looks better now" without a recomputed funnel. +2. Split channel × direction groups, reconcile them to the input total, and separate progression, terminal failure, deferred, inbound, and malformed/unknown outcomes. +3. Compute delivery transitions only from explicit activity histories (`QUEUED`/`ROUTED`/`SENT`/`DELIVERED`). If the export contains latest-only rows, report their outcomes without inventing prior transitions. +4. For WhatsApp/RCS, report `READ` separately as engagement. Stop SMS delivery analysis at `DELIVERED`. +5. If the gate is between `QUEUED` and `SENT`: check synchronous codes on recent request envelopes. +6. If the gate is at `FAILED` after `SENT`: fetch a sample of failed message IDs, read `description` for `ERR_*` codes. +7. If the symptom is missing customer-side data: prove webhook health via `is_active`, `consecutive_failures`, and `/v3/webhooks/{id}/events` before blaming delivery. +8. Hand off via the matrix above, or escalate to Sent support with the required evidence. +9. Quantify the diagnosis — never "looks better now" without a recomputed funnel. diff --git a/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py b/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py index 2df3b5e..50df69f 100644 --- a/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py +++ b/skills/messaging-performance-analyzer/scripts/analyze_mdr_funnel.py @@ -1,21 +1,15 @@ #!/usr/bin/env python3 -"""Analyze a Sent MDR export and report funnel drop-off by lifecycle stage. +"""Analyze Sent MDR records by channel, direction, lifecycle, and outcome. -Lifecycle stages (in order): QUEUED, ROUTED, SENT, DELIVERED, READ. - -The script counts distinct messages that reached at least each stage, -computes the drop-off percentage between adjacent stages, and flags any -stage where drop-off exceeds the configured threshold. +The analyzer uses only statuses that are present in each record. A record with +only ``status=DELIVERED`` contributes one observed DELIVERED status; it does not +implicitly contribute QUEUED, ROUTED, or SENT transitions. Full ``statuses`` +histories can contribute explicit transition rates. Exit codes: - 0 - healthy funnel (no stage drops more than --threshold) - 2 - bad arguments / malformed input - 3 - unhealthy funnel (one or more stages exceed the threshold) - -Examples: - python analyze_mdr_funnel.py path/to/mdr.json - python analyze_mdr_funnel.py path/to/mdr.csv --threshold 15 - python analyze_mdr_funnel.py path/to/mdr.json --show-errors + 0 - usable cohort with no threshold breach, or usable outcome-only data + 2 - bad arguments, malformed input, or no usable analysis cohort + 3 - one or more explicit delivery/engagement transitions breach threshold """ from __future__ import annotations @@ -25,217 +19,463 @@ import json import re import sys -from collections import Counter +from collections import Counter, defaultdict +from dataclasses import dataclass from pathlib import Path -from typing import Iterable +from typing import Any, Iterable + + +DELIVERY_STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED") +ENGAGEMENT_STAGE = "READ" +KNOWN_STATUSES = set(DELIVERY_STAGES) | {ENGAGEMENT_STAGE, "FAILED", "DEFERRED", "RECEIVED"} +OUTCOMES: tuple[str, ...] = ( + "progression", + "terminal_failure", + "deferred", + "inbound", + "malformed", + "unknown", +) +SUPPORTED_CHANNELS = {"sms", "whatsapp", "rcs"} +READ_CHANNELS = {"whatsapp", "rcs"} +OUTBOUND_DIRECTIONS = {"outbound", "outgoing", "mt"} +INBOUND_DIRECTIONS = {"inbound", "incoming", "mo"} +_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") +_STATUS_KEYS = ("status", "stage", "latest_status") -# Ordered lifecycle stages. A message that reached stage N is counted as -# having reached every earlier stage as well. -STAGES: tuple[str, ...] = ("QUEUED", "ROUTED", "SENT", "DELIVERED", "READ") -STAGE_RANK: dict[str, int] = {s: i for i, s in enumerate(STAGES)} + +class InputError(Exception): + """Raised for missing or structurally invalid input. Maps to exit code 2.""" + + +@dataclass(frozen=True) +class NormalizedRecord: + """One input row normalized without inventing lifecycle evidence.""" + + index: int + channel: str + direction: str + observed_statuses: tuple[str, ...] + has_history: bool + outcome: str + diagnostics: tuple[str, ...] + error_codes: tuple[str, ...] def _parse_args(argv: list[str]) -> argparse.Namespace: - parser = argparse.ArgumentParser( - prog="analyze_mdr_funnel.py", - description=( - "Analyze a Sent MDR export and report funnel drop-off by " - "lifecycle stage (QUEUED -> ROUTED -> SENT -> DELIVERED -> READ)." - ), - ) - parser.add_argument( - "path", - help="Path to the MDR export (CSV or JSON; format auto-detected by extension).", - ) + parser = argparse.ArgumentParser(prog="analyze_mdr_funnel.py", description=__doc__) + parser.add_argument("path", help="MDR export path (.json or .csv)") parser.add_argument( "--threshold", type=float, default=20.0, - help=( - "Drop-off percentage threshold per stage. Any stage with a " - "drop-off greater than this value triggers a non-zero exit. " - "Default: 20." - ), + help="Explicit transition drop-off percentage that triggers exit 3 (default: 20)", ) parser.add_argument( "--show-errors", action="store_true", - help=( - "When set, also summarise the count of each Sent send-time error " - "code (ERR_*) parsed from the 'description' field of FAILED " - "messages. Non-FAILED rows and rows without a description are " - "ignored. Does not change exit codes." - ), + help="Include ERR_* codes from FAILED record descriptions", + ) + parser.add_argument( + "--format", + choices=("text", "json"), + default="text", + help="Output format (default: text)", ) return parser.parse_args(argv) -class InputError(Exception): - """Raised for malformed or missing input. Mapped to exit code 2.""" - - -def _load_messages(path: Path) -> list[dict]: +def _load_messages(path: Path) -> list[object]: if not path.is_file(): raise InputError(f"file not found: {path}") - ext = path.suffix.lower() - if ext == ".json": + extension = path.suffix.lower() + if extension == ".json": try: - with path.open("r", encoding="utf-8") as fh: - data = json.load(fh) + data = json.loads(path.read_text(encoding="utf-8")) + except OSError as exc: + raise InputError(f"could not read {path}: {exc}") from exc except json.JSONDecodeError as exc: raise InputError(f"invalid JSON in {path}: {exc}") from exc if isinstance(data, dict) and "messages" in data: data = data["messages"] if not isinstance(data, list): - raise InputError("JSON must be a list of message records or {\"messages\": [...]}") - messages = data - if ext == ".csv": + raise InputError('JSON must be a list of message records or {"messages": [...]}') + return data + if extension == ".csv": try: - with path.open("r", encoding="utf-8", newline="") as fh: - messages = list(csv.DictReader(fh)) - except (csv.Error, UnicodeError) as exc: + with path.open("r", encoding="utf-8", newline="") as handle: + return list(csv.DictReader(handle)) + except (OSError, csv.Error, UnicodeError) as exc: raise InputError(f"invalid CSV in {path}: {exc}") from exc - elif ext != ".json": - raise InputError(f"unsupported file extension '{ext}'; expected .json or .csv") - for index, record in enumerate(messages, 1): - if not isinstance(record, dict): - raise InputError(f"record {index} must be an object") - return messages - - -def _latest_stage(record: dict) -> str | None: - """Return the furthest stage a message reached, or None if unknown. - - Accepts either a single `status`/`stage` field, or a `statuses` list of - {stage, ...} entries (treated as the lifecycle history; the max-rank stage - is the latest in funnel-progression sense). - """ - if "statuses" in record and isinstance(record["statuses"], list): - ranks = [ - STAGE_RANK[s["stage"].upper()] - for s in record["statuses"] - if isinstance(s, dict) and isinstance(s.get("stage"), str) and s["stage"].upper() in STAGE_RANK - ] - if not ranks: - return None - return STAGES[max(ranks)] - for key in ("status", "stage", "latest_status"): - val = record.get(key) - if isinstance(val, str) and val.upper() in STAGE_RANK: - return val.upper() - return None - - -def compute_funnel(messages: Iterable[dict]) -> dict[str, int]: - """Return {stage: count_of_messages_that_reached_at_least_this_stage}.""" - counts = {s: 0 for s in STAGES} - for record in messages: - latest = _latest_stage(record) - if latest is None: - continue - for stage in STAGES[: STAGE_RANK[latest] + 1]: - counts[stage] += 1 - return counts - - -def stage_dropoffs(counts: dict[str, int]) -> list[tuple[str, str, float]]: - """Return [(from_stage, to_stage, dropoff_pct)] for adjacent stages.""" - out: list[tuple[str, str, float]] = [] - for i in range(len(STAGES) - 1): - a, b = STAGES[i], STAGES[i + 1] - if counts[a] == 0: - pct = 0.0 - else: - pct = (counts[a] - counts[b]) / counts[a] * 100.0 - out.append((a, b, pct)) - return out + raise InputError(f"unsupported file extension '{extension}'; expected .json or .csv") + + +def _field(record: dict[str, Any], name: str) -> Any: + if name in record: + return record[name] + payload = record.get("payload") + return payload.get(name) if isinstance(payload, dict) else None + + +def _normalize_channel(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("channel missing; grouped as unknown") + return "unknown" + channel = value.strip().lower() + if channel not in SUPPORTED_CHANNELS: + diagnostics.append(f"unsupported channel {value!r}; grouped as unknown") + return "unknown" + return channel + + +def _normalize_direction(value: Any, diagnostics: list[str]) -> str: + if not isinstance(value, str) or not value.strip(): + diagnostics.append("direction missing; grouped as unknown") + return "unknown" + direction = value.strip().lower() + if direction in OUTBOUND_DIRECTIONS: + return "outbound" + if direction in INBOUND_DIRECTIONS: + return "inbound" + diagnostics.append(f"unsupported direction {value!r}; grouped as unknown") + return "unknown" + + +def _normalize_statuses(record: dict[str, Any], diagnostics: list[str]) -> tuple[tuple[str, ...], bool]: + if "statuses" in record: + history = record["statuses"] + if not isinstance(history, list): + diagnostics.append("statuses must be an array") + return (), False + statuses: list[str] = [] + for position, entry in enumerate(history, 1): + if not isinstance(entry, dict): + diagnostics.append(f"statuses[{position}] must be an object") + continue + value = next((entry.get(key) for key in _STATUS_KEYS if isinstance(entry.get(key), str)), None) + if value is None or not value.strip(): + diagnostics.append(f"statuses[{position}] has no status/stage value") + continue + statuses.append(value.strip().upper()) + return tuple(statuses), True + for key in _STATUS_KEYS: + value = _field(record, key) + if isinstance(value, str) and value.strip(): + return (value.strip().upper(),), False + diagnostics.append("record has no observed status") + return (), False + + +def normalize_record(record: object, index: int) -> NormalizedRecord: + if not isinstance(record, dict): + return NormalizedRecord( + index=index, + channel="unknown", + direction="unknown", + observed_statuses=(), + has_history=False, + outcome="malformed", + diagnostics=("record must be an object",), + error_codes=(), + ) -def _print_report(counts: dict[str, int], drops: list[tuple[str, str, float]]) -> None: - print("Funnel:") - for stage in STAGES: - print(f" {stage:<10} {counts[stage]}") - print("\nStage drop-off:") - for a, b, pct in drops: - print(f" {a} -> {b}: {pct:.1f}%") + diagnostics: list[str] = [] + channel = _normalize_channel(_field(record, "channel"), diagnostics) + direction = _normalize_direction(_field(record, "direction"), diagnostics) + statuses, has_history = _normalize_statuses(record, diagnostics) + unknown_statuses = sorted({status for status in statuses if status not in KNOWN_STATUSES}) + if unknown_statuses: + diagnostics.append("unknown status values: " + ", ".join(unknown_statuses)) + + known = tuple(status for status in statuses if status in KNOWN_STATUSES) + latest = known[-1] if known else None + if direction == "inbound" or latest == "RECEIVED": + direction = "inbound" + outcome = "inbound" + elif latest == "FAILED": + outcome = "terminal_failure" + elif latest == "DEFERRED" or latest in {"QUEUED", "ROUTED", "SENT"}: + outcome = "deferred" + elif latest in {"DELIVERED", "READ"}: + outcome = "progression" + else: + outcome = "unknown" + + if channel == "sms" and "READ" in known: + diagnostics.append("READ is not used for SMS delivery or engagement analysis") + description = _field(record, "description") + error_codes = tuple(sorted(set(_ERR_CODE_RE.findall(description)))) if isinstance(description, str) else () + return NormalizedRecord( + index=index, + channel=channel, + direction=direction, + observed_statuses=statuses, + has_history=has_history, + outcome=outcome, + diagnostics=tuple(diagnostics), + error_codes=error_codes, + ) -# Sent's documented send-time per-message error codes appear in the message -# `description` field on FAILED rows. See references/mdr-status-codes.md. -_ERR_CODE_RE = re.compile(r"\bERR_[A-Z0-9_]+\b") -_STATUS_KEYS = ("status", "stage", "latest_status") +def _completed_transition(statuses: tuple[str, ...], source: str, target: str) -> bool: + try: + source_index = statuses.index(source) + except ValueError: + return False + return target in statuses[source_index + 1 :] + + +def _transition( + records: list[NormalizedRecord], source: str, target: str, threshold: float +) -> dict[str, object]: + terminal = {"progression", "terminal_failure"} + eligible_records = [ + record + for record in records + if record.has_history and record.outcome in terminal and source in record.observed_statuses + ] + completed = sum( + _completed_transition(record.observed_statuses, source, target) for record in eligible_records + ) + eligible = len(eligible_records) + dropoff = round((eligible - completed) / eligible * 100.0, 4) if eligible else None + return { + "from": source, + "to": target, + "eligible": eligible, + "completed": completed, + "dropoff_pct": dropoff, + "exceeds_threshold": dropoff is not None and dropoff > threshold, + } + + +def _totals(records: Iterable[NormalizedRecord]) -> dict[str, int]: + records = list(records) + counts = Counter(record.outcome for record in records) + return { + "input": len(records), + "usable": sum(counts[name] for name in ("progression", "terminal_failure", "deferred")), + **{name: counts[name] for name in OUTCOMES}, + } + + +def _group_report( + channel: str, + direction: str, + records: list[NormalizedRecord], + threshold: float, +) -> dict[str, object]: + delivery_statuses = { + stage: sum(stage in record.observed_statuses for record in records) for stage in DELIVERY_STAGES + } + transitions = [ + _transition(records, source, target, threshold) + for source, target in zip(DELIVERY_STAGES, DELIVERY_STAGES[1:]) + ] + engagement: dict[str, object] | None = None + if channel in READ_CHANNELS: + read_transition = _transition(records, "DELIVERED", "READ", threshold) + engagement = { + "delivered": read_transition["eligible"], + "read": read_transition["completed"], + "read_rate_pct": ( + round(100.0 - float(read_transition["dropoff_pct"]), 4) + if read_transition["dropoff_pct"] is not None + else None + ), + "dropoff_pct": read_transition["dropoff_pct"], + "exceeds_threshold": read_transition["exceeds_threshold"], + } + return { + "channel": channel, + "direction": direction, + "totals": _totals(records), + "progression": {"observed": delivery_statuses, "transitions": transitions}, + "engagement": engagement, + } + + +def build_report( + records: list[NormalizedRecord], + threshold: float, + include_errors: bool, +) -> dict[str, object]: + grouped: dict[tuple[str, str], list[NormalizedRecord]] = defaultdict(list) + for record in records: + grouped[(record.channel, record.direction)].append(record) + groups = [ + _group_report(channel, direction, grouped[(channel, direction)], threshold) + for channel, direction in sorted(grouped) + ] + anomalies: list[dict[str, object]] = [] + evaluated = 0 + for group in groups: + for transition in group["progression"]["transitions"]: + if transition["dropoff_pct"] is not None: + evaluated += 1 + if transition["exceeds_threshold"]: + anomalies.append( + { + "kind": "delivery", + "channel": group["channel"], + "direction": group["direction"], + **transition, + } + ) + engagement = group["engagement"] + if engagement is not None: + if engagement["dropoff_pct"] is not None: + evaluated += 1 + if engagement["exceeds_threshold"]: + anomalies.append( + { + "kind": "engagement", + "channel": group["channel"], + "direction": group["direction"], + "from": "DELIVERED", + "to": "READ", + "eligible": engagement["delivered"], + "completed": engagement["read"], + "dropoff_pct": engagement["dropoff_pct"], + "exceeds_threshold": True, + } + ) + totals = _totals(records) + healthy: bool | None + if not totals["usable"] or not evaluated: + healthy = None + else: + healthy = not anomalies + diagnostics = [ + {"record": record.index, "messages": list(record.diagnostics)} + for record in records + if record.diagnostics + ] + error_codes = Counter( + code + for record in records + if record.outcome == "terminal_failure" + for code in record.error_codes + ) + return { + "schema_version": 1, + "threshold_pct": threshold, + "healthy": healthy, + "totals": totals, + "groups": groups, + "anomalies": anomalies, + "diagnostics": diagnostics, + "error_codes": dict(sorted(error_codes.items())) if include_errors else None, + } -def _is_failed(record: dict) -> bool: - # FAILED isn't in STAGES, so _latest_stage() returns None for it. Check - # the explicit status keys and the statuses history directly. - for key in _STATUS_KEYS: - val = record.get(key) - if isinstance(val, str) and val.upper() == "FAILED": - return True - history = record.get("statuses") - if isinstance(history, list): - for entry in history: - if isinstance(entry, dict) and isinstance(entry.get("stage"), str) \ - and entry["stage"].upper() == "FAILED": - return True - return False +def analyze(records: Iterable[object], threshold: float, include_errors: bool) -> dict[str, object]: + normalized = [normalize_record(record, index) for index, record in enumerate(records, 1)] + return build_report(normalized, threshold, include_errors) def summarise_errors(messages: Iterable[dict]) -> Counter: - """Return a Counter of ERR_* codes parsed from FAILED messages' descriptions.""" + """Return ERR_* code counts from records whose observed outcome is FAILED.""" counter: Counter = Counter() - for record in messages: - if not isinstance(record, dict) or not _is_failed(record): - continue - description = record.get("description") - if not isinstance(description, str): - continue - # Dedupe within a row so one message with the same code mentioned - # twice doesn't double-count. - counter.update(set(_ERR_CODE_RE.findall(description))) + for index, record in enumerate(messages, 1): + normalized = normalize_record(record, index) + if normalized.outcome == "terminal_failure": + counter.update(normalized.error_codes) return counter -def _print_error_summary(counter: Counter) -> None: - print("\nERR_* codes (from FAILED descriptions):") - if not counter: - print(" (none found)") - return - for code, count in counter.most_common(): - print(f" {code:<32} {count}") +def _percent(value: object) -> str: + return "N/A" if value is None else f"{float(value):.1f}%" + + +def render_text(report: dict[str, object]) -> str: + totals = report["totals"] + lines = [ + f"Records: {totals['input']} input, {totals['usable']} usable", + "Outcomes: " + ", ".join(f"{name}={totals[name]}" for name in OUTCOMES), + ] + for group in report["groups"]: + lines.extend( + ( + "", + f"Group: channel={group['channel']} direction={group['direction']}", + " Outcomes: " + ", ".join(f"{name}={group['totals'][name]}" for name in OUTCOMES), + " Observed: " + + ", ".join( + f"{stage}={group['progression']['observed'][stage]}" for stage in DELIVERY_STAGES + ), + " Delivery transitions:", + ) + ) + for transition in group["progression"]["transitions"]: + lines.append( + f" {transition['from']} -> {transition['to']}: " + f"{_percent(transition['dropoff_pct'])} " + f"({transition['completed']}/{transition['eligible']} completed)" + ) + if group["engagement"] is not None: + engagement = group["engagement"] + lines.append(" Engagement:") + lines.append( + " DELIVERED -> READ: " + f"{_percent(engagement['dropoff_pct'])} " + f"({engagement['read']}/{engagement['delivered']} read)" + ) + if report["error_codes"] is not None: + lines.append("") + lines.append("ERR_* codes:") + if report["error_codes"]: + lines.extend(f" {code}: {count}" for code, count in report["error_codes"].items()) + else: + lines.append(" (none found)") + if report["diagnostics"]: + lines.append("") + lines.append(f"Validation diagnostics: {len(report['diagnostics'])} record(s)") + for diagnostic in report["diagnostics"]: + lines.append(f" record {diagnostic['record']}: " + "; ".join(diagnostic["messages"])) + lines.append("") + if not totals["usable"]: + lines.append("INSUFFICIENT: no usable analysis cohort exists.") + elif report["healthy"] is None: + lines.append("INDETERMINATE: outcomes are usable, but no explicit transition denominator exists.") + elif report["healthy"]: + lines.append(f"OK: no explicit transition exceeded {report['threshold_pct']}% drop-off.") + else: + lines.append( + f"FAIL: {len(report['anomalies'])} explicit transition(s) exceeded " + f"{report['threshold_pct']}% drop-off." + ) + return "\n".join(lines) def main(argv: list[str] | None = None) -> int: args = _parse_args(sys.argv[1:] if argv is None else argv) + if not 0 <= args.threshold <= 100: + print("error: --threshold must be between 0 and 100", file=sys.stderr) + return 2 try: - messages = _load_messages(Path(args.path)) + records = _load_messages(Path(args.path)) except InputError as exc: print(f"error: {exc}", file=sys.stderr) return 2 - if not messages: - print("error: no messages found in input", file=sys.stderr) - return 2 - - counts = compute_funnel(messages) - drops = stage_dropoffs(counts) - _print_report(counts, drops) - if args.show_errors: - _print_error_summary(summarise_errors(messages)) + report = analyze(records, args.threshold, args.show_errors) + if args.format == "json": + print(json.dumps(report, indent=2, sort_keys=True)) + else: + print(render_text(report)) - anomalies = [(a, b, pct) for a, b, pct in drops if pct > args.threshold] - if anomalies: - print( - f"\nFAIL: {len(anomalies)} stage(s) exceeded the {args.threshold}% drop-off threshold:", - file=sys.stderr, - ) - for a, b, pct in anomalies: - print(f" {a} -> {b}: {pct:.1f}%", file=sys.stderr) + if report["totals"]["usable"] == 0: + if args.format == "text": + print("error: no usable analysis cohort exists", file=sys.stderr) + return 2 + if report["healthy"] is False: + if args.format == "text": + print(f"FAIL: {len(report['anomalies'])} transition threshold breach(es)", file=sys.stderr) return 3 - print(f"\nOK: no stage exceeded the {args.threshold}% drop-off threshold.") return 0 if __name__ == "__main__": - sys.exit(main()) + raise SystemExit(main()) diff --git a/skills/messaging-performance-analyzer/scripts/fixtures/bad.json b/skills/messaging-performance-analyzer/scripts/fixtures/bad.json index 038028d..e8bbd98 100644 --- a/skills/messaging-performance-analyzer/scripts/fixtures/bad.json +++ b/skills/messaging-performance-analyzer/scripts/fixtures/bad.json @@ -1,502 +1,118 @@ [ { - "message_id": "msg_test_001", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_002", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_003", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_004", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_005", - "channel": "whatsapp", - "status": "QUEUED" - }, - { - "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_010", - "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "message_id": "msg_test_101", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_102", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] + }, + { + "message_id": "msg_test_103", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_104", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_ROUTE_DENIED" + }, + { + "message_id": "msg_test_105", + "channel": "whatsapp", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "FAILED"} + ], + "description": "ERR_TEMPLATE_PARAMS_INVALID" + }, + { + "message_id": "msg_test_106", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_107", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_108", + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] + }, + { + "message_id": "msg_test_109", + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"} + ] + }, + { + "message_id": "msg_test_110", + "channel": "whatsapp", + "direction": "inbound", + "status": "RECEIVED" + }, + "malformed synthetic record", + { + "message_id": "msg_test_112", + "channel": "sms", + "direction": "outbound", + "status": "MYSTERY" } ] diff --git a/skills/messaging-performance-analyzer/scripts/fixtures/good.json b/skills/messaging-performance-analyzer/scripts/fixtures/good.json index 56d3784..0b66c89 100644 --- a/skills/messaging-performance-analyzer/scripts/fixtures/good.json +++ b/skills/messaging-performance-analyzer/scripts/fixtures/good.json @@ -2,501 +2,111 @@ { "message_id": "msg_test_001", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_002", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_003", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_004", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_005", "channel": "whatsapp", - "status": "QUEUED" + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_006", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_007", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "sms", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_008", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "rcs", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"}, + {"stage": "READ"} + ] }, { "message_id": "msg_test_009", - "channel": "whatsapp", - "status": "ROUTED" + "channel": "satellite", + "direction": "outbound", + "statuses": [ + {"stage": "QUEUED"}, + {"stage": "ROUTED"}, + {"stage": "SENT"}, + {"stage": "DELIVERED"} + ] }, { "message_id": "msg_test_010", "channel": "whatsapp", - "status": "ROUTED" - }, - { - "message_id": "msg_test_011", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_012", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_013", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_014", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_015", - "channel": "whatsapp", - "status": "SENT" - }, - { - "message_id": "msg_test_016", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_017", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_018", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_019", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_020", - "channel": "whatsapp", - "status": "DELIVERED" - }, - { - "message_id": "msg_test_021", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_022", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_023", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_024", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_025", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_026", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_027", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_028", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_029", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_030", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_031", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_032", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_033", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_034", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_035", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_036", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_037", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_038", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_039", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_040", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_041", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_042", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_043", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_044", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_045", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_046", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_047", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_048", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_049", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_050", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_051", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_052", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_053", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_054", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_055", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_056", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_057", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_058", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_059", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_060", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_061", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_062", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_063", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_064", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_065", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_066", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_067", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_068", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_069", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_070", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_071", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_072", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_073", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_074", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_075", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_076", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_077", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_078", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_079", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_080", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_081", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_082", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_083", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_084", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_085", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_086", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_087", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_088", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_089", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_090", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_091", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_092", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_093", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_094", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_095", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_096", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_097", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_098", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_099", - "channel": "whatsapp", - "status": "READ" - }, - { - "message_id": "msg_test_100", - "channel": "whatsapp", - "status": "READ" + "direction": "inbound", + "status": "RECEIVED" } ]