Skip to content

Latest commit

 

History

History
224 lines (168 loc) · 16.3 KB

File metadata and controls

224 lines (168 loc) · 16.3 KB

Cold-path automation: Event Hubs Capture → Event Grid → Functions

A cold-chain monitoring pipeline where Azure tells you when a batch is ready, instead of your code polling for it.

Devices publish temperature readings into an event hub. Event Hubs Capture archives the stream to Blob Storage as Avro. The moment an archive lands, Event Hubs raises Microsoft.EventHub.CaptureFileCreated to Event Grid, an Event Grid subscription delivers that notification into a second event hub, and a Function App triggered on that hub downloads the archive, aggregates it per device, and writes summaries to a curated hub.

Architecture

The following diagram illustrates the architecture of the solution:

flowchart TB
    devices["Devices<br/>telemetry_producer.py"]

    subgraph ehns["Event Hubs namespace"]
        telemetry[["telemetry hub<br/>4 partitions, Capture enabled"]]
        notifications[["capture-notifications hub<br/>2 partitions"]]
        curated[["curated hub<br/>2 partitions"]]
    end

    subgraph storage["Storage Account"]
        archive[("telemetry-archive container<br/>Avro archives")]
    end

    subgraph eventgrid["Event Grid"]
        topic["System topic<br/>(over the namespace)"]
        subscription["capture-to-eventhub subscription<br/>(Event Hub destination)"]
    end

    subgraph funcapp["Function App"]
        processor["CaptureProcessor<br/>(Event Hubs trigger)"]
    end

    insights["Application Insights<br/>workspace-based"]
    workspace[("Log Analytics<br/>workspace")]

    devices -->|"1: AMQP, partition key = device_id"| telemetry
    telemetry -->|"2: Capture (60 s window, Avro)"| archive
    telemetry -->|"3: Microsoft.EventHub.CaptureFileCreated"| topic
    topic --> subscription
    subscription -->|"4: delivered as an event"| notifications
    notifications -->|"5: trigger (capture-processor group)"| processor
    processor -->|"6: read archive (data.fileUrl)"| archive
    processor -->|"7: per-device summaries (output binding)"| curated
    funcapp -->|"requests · traces · errors<br/>custom events · custom metrics"| insights
    insights -->|"stores its telemetry in"| workspace
    ehns -. "diagnostic settings: resource logs ·<br/>platform metrics of every resource" .-> workspace

    style ehns fill:#ffffff,stroke:#999999,color:#333333
    style storage fill:#ffffff,stroke:#999999,color:#333333
    style eventgrid fill:#ffffff,stroke:#999999,color:#333333
    style funcapp fill:#ffffff,stroke:#999999,color:#333333
Loading

Why this shape

The obvious way to process archives is a timer that lists a container and looks for new blobs. That is a poll: it is late by up to its interval, it re-lists everything each run, and it needs its own bookkeeping to avoid double-processing.

Here the platform does the telling:

Concern How the pipeline handles it
When is a batch ready? Event Hubs says so — CaptureFileCreated carries the file URL, the partition, and the exact sequence range it contains
How does the notification reach the processor? Event Grid delivers it into an event hub, so there is no public webhook to expose and no HTTP endpoint to keep available
What if the processor is down? The notification sits in the hub for the retention window; the processor resumes from its checkpoint and catches up
Is the aggregation correct? Readings are keyed by device, so a device's readings share a partition — and therefore share an archive, in order
Can it be re-run? The archive is durable in Blob Storage; replaying is reading a file, not replaying a stream

That last column is what makes it a cold path: it runs on whole, closed windows rather than on whatever happened to be in the last batch.

What it demonstrates

Capability Where
Event Hubs Capture to Avro in Blob Storage deploy.sh step 4, bicep/modules/event-hubs.bicep
Microsoft.EventHub.CaptureFileCreated system event Raised by Event Hubs; consumed in src/functions/function_app.py
Event Grid system topic over a namespace deploy.sh step 7, bicep/modules/event-grid.bicep
Event Grid subscription with an EventHub destination Same — this is what makes the notification a stream
Event Hubs trigger with Cardinality.MANY function_app.py — batched delivery
Event Hubs output binding function_app.py — writes the curated summaries
Partition keys for correctness telemetry_producer.py — a device's readings stay together
Consumer groups The processor reads the notifications hub with its own group
Least-privilege SAS Send-only for producers, namespace-wide Listen-only for the demo scripts
Avro decoding from Blob Storage function_app.py _read_archive

Prerequisites

  • The LocalStack Azure emulator running, and lstk az start-interception run.
  • lstk (brew install localstack/tap/lstk or npm install -g @localstack/lstk), which routes the Azure CLI to the emulator.
  • Python 3.12+ with src/producers/requirements.txt installed.
  • jq and zip.

Important

Don't start the emulator with DNS_ADDRESS=0. The function container resolves the emulator's hosts through the emulator's DNS server, and cdn.functions.azure.com too, where the Functions host downloads the Event Hubs trigger's extension bundle on every cold container. If either lookup fails, the trigger silently never fires.

Important

Macs with Apple silicon: the function app runs natively on arm64, but the emulator builds each deployment with Microsoft's Oryx build image, which Microsoft publishes for amd64 only (microsoft/Oryx#2133 asks for arm64). Docker Desktop runs that build under amd64 emulation: in Docker Desktop's settings, under General, select the Apple Virtualization framework as the Virtual Machine Manager and turn on Use Rosetta for x86_64/amd64 emulation on Apple Silicon. Under QEMU, which Docker Desktop uses otherwise (Rosetta off, or Docker VMM, which has no Rosetta), the build can crash, because Oryx is a .NET app and .NET does not support QEMU.

Usage

cd samples/eventhubs-eventgrid/python

bash scripts/deploy.sh          # nine steps: storage, hubs, Capture, Event Grid, Function App
bash scripts/validate.sh        # numbered checks over every capability above
bash scripts/run-pipeline.sh    # the end-to-end demo

bash scripts/cleanup.sh         # when you are done

deploy.sh writes scripts/.deployment-env with the resource names and connection strings; the other scripts source it, and so can you:

source scripts/.deployment-env
cd src/producers && python telemetry_producer.py --count 200 --devices 8

Infrastructure as code

The same topology through either IaC path; both then publish the application code with the CLI, and both write the same scripts/.deployment-env, so the demo runs after either one:

cd bicep     && bash deploy.sh    # or:
cd terraform && bash deploy.sh

bash ../scripts/validate.sh
bash ../scripts/run-pipeline.sh

The demo, step by step

scripts/run-pipeline.sh asserts each stage rather than narrating it. It exits non-zero the moment the chain breaks and, on the emulator, prints the processor's container logs too:

  1. Publish telemetry. 120 readings across 6 devices, keyed by device id. One device (DEV-0003 by default) also emits readings above the temperature limit.
  2. Wait for Capture. Polls the blob container until a new archive appears. Nothing downstream can happen until a window closes, so this sets the tempo.
  3. Show the notification. Reads capture-notifications — the hub Event Grid delivered into.
  4. Read the curated summaries. Polls the curated hub for the per-device aggregates the processor produced, and fails if the excursion device is missing from them.
  5. Recap the path an event took.

Application

Producer (src/producers/telemetry_producer.py)

Publishes cold-chain readings with the device id as the partition key, one batch per device. That is not a throughput trick: it is what guarantees a device's readings land in one archive, in order, so the archive-level aggregation is exact.

python telemetry_producer.py [--count 120] [--devices 6] [--excursion-device DEV-0003]

Processor (src/functions/function_app.py)

An Event Hubs-triggered function over capture-notifications. For each notification it:

  1. parses the Event Grid batch array (Event Grid always delivers an array — "an array with a single event" by default),
  2. downloads the archive named by data.fileUrl,
  3. decodes the Avro records,
  4. aggregates per device (count, min/mean/max, excursions), carrying the archive's sequence range through so a downstream consumer can spot a gap between windows,
  5. writes one summary per device via the output binding.

Two details that are easy to get wrong, both commented in the code:

  • The trigger declares cardinality=func.Cardinality.MANY. The default delivers a bare EventHubEvent and the handler fails with "'EventHubEvent' object is not iterable".
  • The handler annotates List[func.EventHubEvent] from typing. The builtin list[...] makes the Functions worker reject the function as an "invalid non-type annotation", and it never loads at all.

Application Insights

The function app sends its telemetry to <prefix>-ehgrid-insights, a workspace-based Application Insights resource that stores it in the <prefix>-ehgrid-log-analytics Log Analytics workspace. The three deployment methods set its APPLICATIONINSIGHTS_CONNECTION_STRING app setting to the connection string of the resource (scripts/deploy.sh passes --app-insights to az functionapp create), and the app uses OpenTelemetry with Azure Functions: host.json sets "telemetryMode": "OpenTelemetry", requirements.txt adds azure-monitor-opentelemetry, and function_app.py calls configure_azure_monitor(logger_name="function_app"), so that the Azure Monitor OpenTelemetry Distro sends the records of the function's function_app logger. The app configures the distro itself, which works on every hosting plan: on the Linux Consumption plan this sample uses, the Python worker sent none of the app's telemetry when the PYTHON_APPLICATIONINSIGHTS_ENABLE_TELEMETRY app setting asked it to.

For each device summary the function sends a DeviceSummarized custom event, with the device, its readings and its excursions in device.id, device.readings and device.excursions, and counts the readings and the excursions in the readings_processed custom metric, by kind (reading, excursion). The workspace keeps the telemetry in AppRequests (the invocations of CaptureProcessor), AppTraces (CAPTURE_BATCH, DEVICE_SUMMARY and the other records of the logger), AppEvents, AppMetrics and AppExceptions (an archive the function can't read).

scripts/query-telemetry.sh reads these tables back, retrying until their records arrive: run it after run-pipeline.sh, which sends the readings that Capture archives. On Azure the telemetry takes a few minutes, on top of the 60-second Capture window: raise RETRY_COUNT there.

Logs and metrics

Each resource that offers diagnostic settings has one that sends its resource logs, its platform metrics or both to the workspace:

Resource Log categories Tables in the workspace
Function App FunctionAppLogs FunctionAppLogs
App Service plan (Consumption) None: metrics only AzureMetrics
Storage account None: metrics only AzureMetrics
Blob, queue and table services StorageRead, StorageWrite, StorageDelete StorageBlobLogs, StorageQueueLogs, StorageTableLogs, AzureMetrics
Event Hubs namespace ArchiveLogs, OperationalLogs and the other eight categories AZMSOperationalLogs, AzureDiagnostics, AzureMetrics
Event Grid system topic DeliveryFailures AegDeliveryFailureLogs, AzureMetrics
Log Analytics workspace Audit LAQueryLogs, AzureMetrics

The namespace writes its operational logs to their resource-specific table, its other categories to AzureDiagnostics; its setting leaves out the runtime audit logs, which Event Hubs writes only in the Premium and Dedicated tiers, and DeliveryFailures records only the deliveries the subscription fails to make. The blob service records the Capture archives Event Hubs writes and the function reads.

scripts/query-logs-and-metrics.sh shows the diagnostic settings and reads back the invocation records, the requests to the blobs, the exported platform metrics and the audit records of the queries run on the workspace, then reads the executions of the function app, the transactions of the storage account and the messages and captures of the namespace from the Azure Monitor metrics API. On Azure the first records of a new diagnostic setting can take up to 90 minutes: raise RETRY_COUNT there. On Azure the script reads the exported platform metrics (AzureMetrics) only once: Azure exports them late and irregularly, so the script shows those that arrived and notes how many are still missing, without failing.

Notes

  • The first deploy.sh run pulls the Functions build image and can take several minutes.
  • deploy.sh is idempotent: every resource is created only when it does not already exist.
  • Capture's minimum interval is 60 seconds, so the demo always takes at least that long.
  • skip_empty_archives is on, so a window with no readings raises no event — otherwise the processor would be woken for empty files.