Skip to content

Latest commit

 

History

History
350 lines (245 loc) · 12 KB

README.md

File metadata and controls

350 lines (245 loc) · 12 KB
Raw

Azure Blob storage to Azure Event Grid and Azure Container App Job

This pattern demonstrates how to use Azure Blob Storage with Azure Event Grid subscriptions. Azure Event Grid is an eventing service for the cloud. We'll demonstrate how an event is published to Azure Storage Queue once we upload a file to Azure Blob Storage. The event will be pulled and processed by an Azure Container App Job.

See the Blob storage events schema article to view the full list of the events that Blob storage supports.

Architecture

Architecture

Prerequisites

  • Azure CLI version 2.53.0 or later.
  • Go version 1.21.1 or later.

A Dev Container is also provided to make it easy to run the code locally. Instructions on how to use the Dev Container can be found at the root of the GitHub repository nickdala/azure-experiments.

Run the sample

1. Clone the repository and change directory

git clone https://github.com/nickdala/azure-experiments.git

cd azure-experiments/samples/blob-event-grid-container-app

2. Login to Azure

Login to Azure using the Azure CLI.

az login

Optional: Set the default subscription. If you have multiple subscriptions, you can list them using az account list.

az account list --output table

az account set --subscription <subscription-id>

3. Install the latest version of the Azure Container Apps CLI extension.

az extension add --name containerapp --upgrade

4. Register the Microsoft.EventGrid, Microsoft.App and Microsoft.OperationalInsights namespaces.

az provider list --query "[?namespace == 'Microsoft.EventGrid']" -o table

You should see something like the following.

Namespace            RegistrationState    RegistrationPolicy
-------------------  -------------------  --------------------
Microsoft.EventGrid  Registered           RegistrationRequired

If the RegistrationState is not Registered, then run the following to register the provider.

az provider register --namespace Microsoft.EventGrid

az provider register --namespace Microsoft.App

az provider register --namespace Microsoft.OperationalInsights

5. Set environment variables

Source the env.sh file in the scripts directory. You can edit the env.sh file to change the default values. Note: Some Azure resources must have globally unique names.

source ./scripts/env.sh

6. Create Azure resources

  1. Create a resource group.

    az group create \
  --name "$RESOURCE_GROUP" \
  --location "$REGION" \
  --tags system="$TAG"

  2. Create a storage account.

    az storage account create \
  --name "$STORAGE_ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$REGION" \
  --tags system="$TAG"

    Before we can create the storage container and the storage queue, we need the storage account connection string.

  3. Get the storage account connection string.

    STORAGE_CONNECTION_STRING=`az storage account show-connection-string --resource-group $RESOURCE_GROUP --name $STORAGE_ACCOUNT_NAME --query connectionString --output tsv`

  4. Create the text storage container.

    az storage container create --name $STORAGE_CONTAINER_NAME_TEXT --account-name $STORAGE_ACCOUNT_NAME --connection-string $STORAGE_CONNECTION_STRING

  5. Create the storage queue.

    az storage queue create --name $QUEUE_NAME --connection-string $STORAGE_CONNECTION_STRING

  6. Create the event subscription.

    First we need the storage account id and the queue endpoint.

    STORAGE_ACCOUNT_ID="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$STORAGE_ACCOUNT_NAME"

QUEUE_ENDPOINT="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$STORAGE_ACCOUNT_NAME/queueservices/default/queues/$QUEUE_NAME"

    Now we can create the event subscription.

    az eventgrid event-subscription create \
--name $EVENT_SUBSCRIPTION_NAME \
--source-resource-id $STORAGE_ACCOUNT_ID \
--endpoint-type storagequeue \
--endpoint $QUEUE_ENDPOINT \
--included-event-types Microsoft.Storage.BlobCreated \
--subject-ends-with .txt

7. Upload Sample Data to Azure Blob Storage

First we need the storage account key.

STORAGE_ACCOUNT_KEY=`az storage account keys list --resource-group $RESOURCE_GROUP --account-name $STORAGE_ACCOUNT_NAME --query "[0].value" --output tsv`

Verify the storage account key.

echo $STORAGE_ACCOUNT_KEY

Upload the sample text file.

az storage blob upload \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --account-key "$STORAGE_ACCOUNT_KEY" \
  --container-name "$STORAGE_CONTAINER_NAME_TEXT" \
  --file "data/sample.txt" \
  --name "sample.txt"

8. Take a look at the Azure Portal

In the Azure Portal, you will see the storage account, the storage container, the storage queue, and the event subscription.

storage-account-event-resources

Let's take a look at the storage container. You should see the sample.txt file we uploaded. Follow the following steps to view the file.

  1. Click on the storage account.
  2. Click on the storage container. storage-container
  3. You should see the sample.txt file. storage-blob-text-sample

The event subscription published an event to the storage queue after the file was uploaded. Let's take a look at the storage queue.

  1. Navigate to the storage account.
  2. Click on the storage queue. storage-queue

Notice that there is a message in the queue. The message is of the type BlobCreated.

storage-queue-blobcreated-event

9. Azure Container Apps

Next, we will build the app to process the message. This app will be deployed as an Azure Container App Job. The source code for the app can be found here. The app will pull the message from the storage queue and log the event id and blob url. Below is the code that pulls the message from the queue.

//Dequeue message from the storage queue.
resp, err := queueClient.DequeueMessage(context.TODO(), &azqueue.DequeueMessageOptions{VisibilityTimeout: to.Ptr(int32(30))})

Deploy the Azure Container App

  1. Create the Azure Container Registry

    az acr create \
  --name "$CONTAINER_REGISTRY_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$REGION" \
  --sku Basic \
  --admin-enabled true \
  --tags system="$TAG"

Build and Deploy the Azure Container App

  1. Login to the Azure Container Registry.

    az acr login --name "$CONTAINER_REGISTRY_NAME"

  2. Build the Docker image.

    az acr build \
  --registry "$CONTAINER_REGISTRY_NAME" \
  --image "$CONTAINER_IMAGE_NAME" \
  --file Dockerfile .

  3. Verify the Docker image was pushed to the Azure Container Registry.

    az acr repository list --name $CONTAINER_REGISTRY_NAME --output table

  4. Create the App Environment.

    az containerapp env create \
    --name "$ACA_ENVIRONMENT" \
    --resource-group "$RESOURCE_GROUP" \
    --location "$REGION" \
    --tags system="$TAG"

  5. Create the Azure Container App Job.

    az containerapp job create \
--name "$ACA_JOB_NAME" \
--resource-group "$RESOURCE_GROUP" \
--environment "$ACA_ENVIRONMENT" \
--trigger-type "Event" \
--replica-timeout "1800" \
--replica-retry-limit "1" \
--replica-completion-count "1" \
--parallelism "1" \
--min-executions "0" \
--max-executions "10" \
--polling-interval "60" \
--scale-rule-name "queue" \
--scale-rule-type "azure-queue" \
--scale-rule-metadata "accountName=$STORAGE_ACCOUNT_NAME" "queueName=$QUEUE_NAME" "queueLength=1" \
--scale-rule-auth "connection=connection-string-secret" \
--image "$CONTAINER_REGISTRY_NAME.azurecr.io/$CONTAINER_IMAGE_NAME" \
--cpu "0.25" \
--memory "0.5Gi" \
--secrets "connection-string-secret=$STORAGE_CONNECTION_STRING" "storage-key=$STORAGE_ACCOUNT_KEY" \
--registry-server "$CONTAINER_REGISTRY_NAME.azurecr.io" \
--env-vars "AZURE_STORAGE_QUEUE_NAME=$QUEUE_NAME" "AZURE_STORAGE_CONNECTION_STRING=secretref:connection-string-secret" "AZURE_STORAGE_ACCOUNT_NAME=$STORAGE_ACCOUNT_NAME" "AZURE_STORAGE_ACCOUNT_KEY=secretref:storage-key"

10. Verify the event was processed by the Azure Container App

Let's now take a look at the logs for the Azure Container App Job. The logs will show that the event was processed by the Azure Container App Job. Navigate to the Azure Container App Job in the Azure Portal. Click on the Container App Job.

container-app-job-resource

This will bring you to the Container App Job resource. Here, you can see things like configuration, secrets, and event scaling. We're going to explore the execution history. Click on the Execution History tab. You should see the following.

container-app-job-execution-history

Click on the Console Logs link. You should see the following.

container-app-job-logs

Query job execution logs

  1. Save the Log Analytics workspace ID for the Container Apps environment to a variable.

    LOG_ANALYTICS_WORKSPACE_ID=`az containerapp env show \
    --name "$ACA_ENVIRONMENT" \
    --resource-group "$RESOURCE_GROUP" \
    --query "properties.appLogsConfiguration.logAnalyticsConfiguration.customerId" \
    --output tsv`

  2. Save the name of the most recent job execution to a variable.

    JOB_EXECUTION_NAME=`az containerapp job execution list \
    --name "$ACA_JOB_NAME" \
    --resource-group "$RESOURCE_GROUP" \
    --query "[0].name" \
    --output tsv`

  3. Run a query against Log Analytics for the job execution using the following command.

    az monitor log-analytics query \
--workspace "$LOG_ANALYTICS_WORKSPACE_ID" \
--analytics-query "ContainerAppConsoleLogs_CL | where ContainerGroupName_s startswith '$JOB_EXECUTION_NAME' | order by _timestamp_d asc" \
--query "[].Log_s"

The following output is an example of the logs printed by the job execution.

[
  "2023/10/17 15:06:41 Getting blob created event",
  "2023/10/17 15:06:41 This is id 0a7343f3-101e-0055-2a0b-01050b06eff5",
  "2023/10/17 15:06:41 This is url https://stblobnickdalae5de41.blob.core.windows.net/text/sample.txt",
  "2023/10/17 15:06:41 This is api PutBlob",
  "2023/10/17 15:06:41 This is event type Microsoft.Storage.BlobCreated",
  "2023/10/17 15:06:41 Done!"
]

9. Cleanup

az group delete --name $RESOURCE_GROUP --yes --no-wait

Resources