Skip to content

output_azure: OAuth authentication support for Azure output plugins #11571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zshuang0316 wants to merge 4 commits into
base: master
Choose a base branch
from
Open

output_azure: OAuth authentication support for Azure output plugins #11571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8ce83a7
azure_auth: add common Azure authentication module
zshuang0316 Mar 17, 2026
d19d342
out_azure_blob: add OAuth authentication support
zshuang0316 Mar 17, 2026
d0497e7
out_azure_kusto: migrate to common Azure authentication
zshuang0316 Mar 17, 2026
a4478a1
tests: add unit tests for Azure Blob OAuth authentication
zshuang0316 Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions include/fluent-bit/flb_azure_auth.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
/* -*- Mode: C; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- */

/* Fluent Bit
* ==========
* Copyright (C) 2015-2026 The Fluent Bit Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#ifndef FLB_AZURE_AUTH_H
#define FLB_AZURE_AUTH_H

#include <fluent-bit/flb_info.h>
#include <fluent-bit/flb_oauth2.h>
#include <fluent-bit/flb_sds.h>

/* Authentication types for Azure services */
typedef enum {
FLB_AZURE_AUTH_KEY = 0, /* Shared Access Key (blob-specific) */
FLB_AZURE_AUTH_SAS, /* Shared Access Signature (blob-specific) */
FLB_AZURE_AUTH_SERVICE_PRINCIPAL, /* Service Principal (Client ID + Secret) */
FLB_AZURE_AUTH_MANAGED_IDENTITY_SYSTEM, /* System-assigned Managed Identity */
FLB_AZURE_AUTH_MANAGED_IDENTITY_USER, /* User-assigned Managed Identity */
FLB_AZURE_AUTH_WORKLOAD_IDENTITY /* Workload Identity (Federated Token) */
} flb_azure_auth_type;

/* Azure Instance Metadata Service (IMDS) endpoint for Managed Identity */
#define FLB_AZURE_IMDS_HOST "169.254.169.254"
#define FLB_AZURE_IMDS_PORT "80"

/* Managed Identity authentication URL template */
#define FLB_AZURE_MSI_AUTH_URL_TEMPLATE \
"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2021-02-01%s%s&resource=%s"

/* Microsoft Authentication Library (MSAL) authorization URL template */
#define FLB_AZURE_MSAL_AUTH_URL_TEMPLATE \
"https://login.microsoftonline.com/%s/oauth2/v2.0/token"

/* Azure Blob Storage resource identifier */
#define FLB_AZURE_BLOB_RESOURCE "https://storage.azure.com"

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the Blob resource ID with trailing slash

For auth_type managed_identity in azure_blob, flb_azure_auth_build_oauth_url() passes this constant directly as the IMDS resource query value, so the token is requested for https://storage.azure.com instead of the Storage data-plane audience https://storage.azure.com/. That audience mismatch causes managed-identity Blob requests to fail authentication even when RBAC is correct; use a separate MSI resource ID with the trailing slash while keeping the current no-slash base for /.default scopes.

Useful? React with 👍 / 👎.

@zshuang0316 zshuang0316 Jun 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already have it tested without the slash, and no slash is consistent with azure_kusto, so will not change


/* Azure Kusto resource identifier for IMDS (MSI) authentication */
#define FLB_AZURE_KUSTO_RESOURCE "https://api.kusto.windows.net"

/* Azure Kusto scope for MSAL (service principal / workload identity) */
#define FLB_AZURE_KUSTO_SCOPE "https://help.kusto.windows.net/.default"

/* Default Workload Identity token file path */
#define FLB_AZURE_WORKLOAD_IDENTITY_TOKEN_FILE \
"/var/run/secrets/azure/tokens/azure-identity-token"

/**
* Get an OAuth2 access token using Azure Managed Identity (MSI)
*
* This function retrieves an access token from the Azure Instance Metadata Service (IMDS)
* for use with Azure services. Supports both system-assigned and user-assigned managed identities.
*
* @param ctx OAuth2 context containing connection and token information
* @return Access token string on success, NULL on failure
*/
char *flb_azure_msi_token_get(struct flb_oauth2 *ctx);

/**
* Get an OAuth2 access token using Azure Workload Identity
*
* This function exchanges a federated token (JWT) for an Azure AD access token
* using the OAuth2 client credentials flow with client assertion.
*
* @param ctx OAuth2 context for token management
* @param token_file Path to the file containing the federated token
* @param client_id Client ID of the Azure AD application
* @param tenant_id Tenant ID of the Azure AD directory
* @param resource Resource scope for the token (e.g., "https://storage.azure.com/")
* @return 0 on success, -1 on failure
*/
int flb_azure_workload_identity_token_get(struct flb_oauth2 *ctx,
const char *token_file,
const char *client_id,
const char *tenant_id,
const char *resource);

/**
* Build OAuth URL for Azure authentication
*
* Creates the appropriate OAuth2 endpoint URL based on the authentication type.
* For Managed Identity, uses IMDS endpoint. For Service Principal and Workload Identity,
* uses Azure AD OAuth2 endpoint.
*
* @param auth_type Type of authentication to use
* @param tenant_id Azure AD tenant ID (required for Service Principal and Workload Identity)
* @param client_id Client ID (optional, used for user-assigned managed identity)
* @param resource Resource scope for the token (e.g., storage.azure.com)
* @return Allocated SDS string with OAuth URL, or NULL on failure
*/
flb_sds_t flb_azure_auth_build_oauth_url(flb_azure_auth_type auth_type,
const char *tenant_id,
const char *client_id,
const char *resource);

#endif /* FLB_AZURE_AUTH_H */
80 changes: 69 additions & 11 deletions plugins/out_azure_blob/azure_blob.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -738,8 +738,13 @@ static int create_blob(struct flb_azure_blob *ctx, const char *path_prefix, char
}

/* Prepare headers and authentication */
azb_http_client_setup(ctx, c, -1, FLB_TRUE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
ret = azb_http_client_setup(ctx, c, -1, FLB_TRUE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
if (ret == -1) {
flb_plg_error(ctx->ins, "failed to setup HTTP client for create_append_blob");
status = FLB_RETRY;
goto cleanup_create;
}

/* Send HTTP request */
ret = flb_http_do(c, &b_sent);
Expand Down Expand Up @@ -817,8 +822,13 @@ static int delete_blob(struct flb_azure_blob *ctx,
}

/* Prepare headers and authentication */
azb_http_client_setup(ctx, c, -1, FLB_TRUE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
ret = azb_http_client_setup(ctx, c, -1, FLB_TRUE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
if (ret == -1) {
flb_plg_error(ctx->ins, "failed to setup HTTP client for delete_blob");
status = FLB_RETRY;
goto cleanup_delete;
}

/* Send HTTP request */
ret = flb_http_do(c, &b_sent);
Expand Down Expand Up @@ -1000,8 +1010,17 @@ static int http_send_blob(struct flb_config *config, struct flb_azure_blob *ctx,
}

/* Prepare headers and authentication */
azb_http_client_setup(ctx, c, (ssize_t) payload_size, FLB_FALSE,
content_type, content_encoding);
ret = azb_http_client_setup(ctx, c, (ssize_t) payload_size, FLB_FALSE,
content_type, content_encoding);
if (ret == -1) {
flb_plg_error(ctx->ins, "failed to setup HTTP client for http_send_blob");
flb_http_client_destroy(c);
if (compressed == FLB_TRUE) {
flb_free(payload_buf);
}
flb_upstream_conn_release(u_conn);
return FLB_RETRY;
}

/* Send HTTP request */
ret = flb_http_do(c, &b_sent);
Expand Down Expand Up @@ -1233,8 +1252,15 @@ static int create_container(struct flb_azure_blob *ctx, char *name)
}

/* Prepare headers and authentication */
azb_http_client_setup(ctx, c, -1, FLB_FALSE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
ret = azb_http_client_setup(ctx, c, -1, FLB_FALSE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
if (ret == -1) {
flb_plg_error(ctx->ins, "failed to setup HTTP client for create_container");
flb_http_client_destroy(c);
flb_upstream_conn_release(u_conn);
flb_sds_destroy(uri);
return FLB_FALSE;
}

/* Send HTTP request */
ret = flb_http_do(c, &b_sent);
Expand Down Expand Up @@ -1326,8 +1352,15 @@ static int ensure_container(struct flb_azure_blob *ctx)
flb_http_strip_port_from_host(c);

/* Prepare headers and authentication */
azb_http_client_setup(ctx, c, -1, FLB_FALSE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
ret = azb_http_client_setup(ctx, c, -1, FLB_FALSE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
if (ret == -1) {
flb_plg_error(ctx->ins, "failed to setup HTTP client for container_check");
flb_http_client_destroy(c);
flb_upstream_conn_release(u_conn);
flb_sds_destroy(uri);
return FLB_FALSE;
}

/* Send HTTP request */
ret = flb_http_do(c, &b_sent);
Expand Down Expand Up @@ -2514,7 +2547,7 @@ static struct flb_config_map config_map[] = {
{
FLB_CONFIG_MAP_STR, "auth_type", "key",
0, FLB_TRUE, offsetof(struct flb_azure_blob, auth_type),
"Set the auth type: key or sas"
"Set the auth type: key, sas, service_principal, managed_identity, or workload_identity"
},

{
Expand All @@ -2523,6 +2556,31 @@ static struct flb_config_map config_map[] = {
"Azure Blob SAS token"
},

/* OAuth authentication parameters */
{
FLB_CONFIG_MAP_STR, "tenant_id", NULL,
0, FLB_TRUE, offsetof(struct flb_azure_blob, tenant_id),
"Azure AD tenant ID (required for service_principal and workload_identity auth)"
},

{
FLB_CONFIG_MAP_STR, "client_id", NULL,
0, FLB_TRUE, offsetof(struct flb_azure_blob, client_id),
"Azure AD client ID / Application ID (required for OAuth-based authentication)"
},

{
FLB_CONFIG_MAP_STR, "client_secret", NULL,
0, FLB_TRUE, offsetof(struct flb_azure_blob, client_secret),
"Azure AD client secret (required for service_principal auth)"
},

{
FLB_CONFIG_MAP_STR, "workload_identity_token_file", NULL,
0, FLB_TRUE, offsetof(struct flb_azure_blob, workload_identity_token_file),
"Path to workload identity token file (for workload_identity auth)"
},

{
FLB_CONFIG_MAP_STR, "database_file", NULL,
0, FLB_TRUE, offsetof(struct flb_azure_blob, database_file),
Expand Down
23 changes: 19 additions & 4 deletions plugins/out_azure_blob/azure_blob.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@
#include <fluent-bit/flb_sds.h>
#include <fluent-bit/flb_sqldb.h>
#include <fluent-bit/flb_time.h>
#include <fluent-bit/flb_azure_auth.h>
#ifdef FLB_HAVE_TLS
#include <fluent-bit/flb_oauth2.h>
#endif
Comment thread
coderabbitai[bot] marked this conversation as resolved.

/* Content-Type */
#define AZURE_BLOB_CT "Content-Type"
Expand All @@ -51,9 +55,6 @@
#define AZURE_BLOB_APPENDBLOB 0
#define AZURE_BLOB_BLOCKBLOB 1

#define AZURE_BLOB_AUTH_KEY 0
#define AZURE_BLOB_AUTH_SAS 1

struct flb_azure_blob {
int auto_create_container;
int emulator_mode;
Expand All @@ -69,6 +70,13 @@ struct flb_azure_blob {
flb_sds_t date_key;
flb_sds_t auth_type;
flb_sds_t sas_token;

/* OAuth authentication fields */
flb_sds_t tenant_id;
flb_sds_t client_id;
flb_sds_t client_secret;
flb_sds_t workload_identity_token_file;

flb_sds_t database_file;
size_t part_size;
time_t upload_parts_timeout;
Expand Down Expand Up @@ -116,7 +124,7 @@ struct flb_azure_blob {
* Internal use
*/
int btype; /* blob type */
int atype; /* auth type */
flb_azure_auth_type atype; /* auth type (uses flb_azure_auth_type enum from flb_azure_auth.h) */
flb_sds_t real_endpoint;
flb_sds_t base_uri;
flb_sds_t shared_key_prefix;
Expand All @@ -125,6 +133,13 @@ struct flb_azure_blob {
unsigned char *decoded_sk; /* decoded shared key */
size_t decoded_sk_size; /* size of decoded shared key */

#ifdef FLB_HAVE_TLS
/* OAuth2 authentication */
flb_sds_t oauth_url;
struct flb_oauth2 *o;
pthread_mutex_t token_mutex;
Comment thread
coderabbitai[bot] marked this conversation as resolved.
#endif

#ifdef FLB_HAVE_SQLDB
/*
* SQLite by default is not built with multi-threading enabled, and
Expand Down
2 changes: 1 addition & 1 deletion plugins/out_azure_blob/azure_blob_appendblob.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ flb_sds_t azb_append_blob_uri(struct flb_azure_blob *ctx,
}
}

if (ctx->atype == AZURE_BLOB_AUTH_SAS && ctx->sas_token) {
if (ctx->atype == FLB_AZURE_AUTH_SAS && ctx->sas_token) {
if (flb_sds_printf(&uri, "&%s", ctx->sas_token) == NULL) {
flb_sds_destroy(uri);
return NULL;
Expand Down
20 changes: 14 additions & 6 deletions plugins/out_azure_blob/azure_blob_blockblob.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ flb_sds_t azb_block_blob_blocklist_uri(struct flb_azure_blob *ctx,
flb_sds_printf(&uri, "/%s?comp=blocklist", name);
}

if (ctx->atype == AZURE_BLOB_AUTH_SAS && ctx->sas_token) {
if (ctx->atype == FLB_AZURE_AUTH_SAS && ctx->sas_token) {
flb_sds_printf(&uri, "&%s", ctx->sas_token);
}

Expand Down Expand Up @@ -124,7 +124,7 @@ flb_sds_t azb_block_blob_uri(struct flb_azure_blob *ctx,
}
}

if (ctx->atype == AZURE_BLOB_AUTH_SAS && ctx->sas_token) {
if (ctx->atype == FLB_AZURE_AUTH_SAS && ctx->sas_token) {
flb_sds_printf(&uri, "&%s", ctx->sas_token);
}

Expand Down Expand Up @@ -165,7 +165,7 @@ flb_sds_t azb_block_blob_uri_commit(struct flb_azure_blob *ctx,
flb_sds_printf(&uri, "/%s.%s.%" PRIu64 "%s?comp=blocklist", tag, str, ms, ext);
}

if (ctx->atype == AZURE_BLOB_AUTH_SAS && ctx->sas_token) {
if (ctx->atype == FLB_AZURE_AUTH_SAS && ctx->sas_token) {
flb_sds_printf(&uri, "&%s", ctx->sas_token);
}

Expand Down Expand Up @@ -310,16 +310,24 @@ int azb_block_blob_put_block_list(struct flb_azure_blob *ctx, flb_sds_t uri, flb
}

/* Prepare headers and authentication */
azb_http_client_setup(ctx, c, flb_sds_len(payload),
FLB_FALSE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
ret = azb_http_client_setup(ctx, c, flb_sds_len(payload),
FLB_FALSE,
AZURE_BLOB_CT_NONE, AZURE_BLOB_CE_NONE);
if (ret == -1) {
flb_plg_error(ctx->ins, "failed to setup HTTP client for block_blob_commit");
flb_http_client_destroy(c);
flb_upstream_conn_release(u_conn);
return FLB_RETRY;
}

/* Send HTTP request */
ret = flb_http_do(c, &b_sent);

/* Validate HTTP status */
if (ret == -1) {
flb_plg_error(ctx->ins, "error sending block_blob");
flb_http_client_destroy(c);
flb_upstream_conn_release(u_conn);
return FLB_RETRY;
}

Expand Down
Loading
Loading