Skip to content

Commit

Permalink
azure exporter: Enable features from upstream exporter (#5707)
Browse files Browse the repository at this point in the history 
* Update azure exporter for ValidateDimension support

* Add support for subscription scoped metric gathering

* Add docs for validate_dimensions

* Fix conditional for when to use subscription scope

* changelog entry

* Fix crow error from prom common upgrade

* Code review feedback

* Cleanup merge + missing review comments + fix deprecations from go.mod update
  • Loading branch information
@kgeckhart
kgeckhart authored Dec 21, 2023
1 parent d113bc7 commit 5e9df54
Show file tree
Hide file tree
Showing 18 changed files with 287 additions and 149 deletions.
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ Main (unreleased)
- The `target` block in `prometheus.exporter.blackbox` requires a mandatory `name`
argument instead of a block label. (@hainenber)

- In the azure exporter, dimension options will no longer be validated by the Azure API. (@kgeckhart)
- This change will not break any existing configurations and you can opt in to validation via the `validate_dimensions` configuration option.
- Before this change, pulling metrics for azure resources with variable dimensions required one configuration per metric + dimension combination to avoid an error.
- After this change, you can include all metrics and dimensions in a single configuration and the Azure APIs will only return dimensions which are valid for the various metrics.

### Enhancements

- Flow Windows service: Support environment variables. (@jkroepke)
Expand Down Expand Up @@ -63,6 +68,10 @@ Main (unreleased)

- Added 'country' mmdb-type to log pipeline-stage geoip. (@superstes)

- Azure exporter enhancements for flow and static mode, (@kgeckhart)
- Allows for pulling metrics at the Azure subscription level instead of resource by resource
- Disable dimension validation by default to reduce the number of exporter instances needed for full dimension coverage

### Bugfixes

- Update `pyroscope.ebpf` to fix a logical bug causing to profile to many kthreads instead of regular processes https://github.com/grafana/pyroscope/pull/2778 (@korniltsev)
Expand Down
6 changes: 4 additions & 2 deletions component/otelcol/config_filter_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,13 @@ package otelcol_test
import (
"testing"

"k8s.io/utils/ptr"

"github.com/grafana/agent/component/otelcol"

"github.com/grafana/river"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/collector/pdata/plog"
"k8s.io/utils/pointer"
)

func TestConvertMatchProperties(t *testing.T) {
Expand Down Expand Up @@ -49,7 +51,7 @@ func TestConvertMatchProperties(t *testing.T) {
Libraries: []otelcol.InstrumentationLibrary{
{
Name: "mongo-java-driver",
Version: pointer.String("3.8.0"),
Version: ptr.To("3.8.0"),
},
},
SpanKinds: []string{"span1"},
Expand Down
8 changes: 8 additions & 0 deletions component/prometheus/exporter/azure/azure.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ type Arguments struct {
MetricNameTemplate string `river:"metric_name_template,attr,optional"`
MetricHelpTemplate string `river:"metric_help_template,attr,optional"`
AzureCloudEnvironment string `river:"azure_cloud_environment,attr,optional"`
ValidateDimensions bool `river:"validate_dimensions,attr,optional"`
Regions []string `river:"regions,attr,optional"`
}

var DefaultArguments = Arguments{
Expand All @@ -43,6 +45,10 @@ var DefaultArguments = Arguments{
MetricHelpTemplate: "Azure metric {metric} for {type} with aggregation {aggregation} as {unit}",
IncludedResourceTags: []string{"owner"},
AzureCloudEnvironment: "azurecloud",
// Dimensions do not always apply to all metrics for a service, which requires you to configure multiple exporters
// to fully monitor a service which is tedious. Turning off validation eliminates this complexity. The underlying
// sdk will only give back the dimensions which are valid for particular metrics.
ValidateDimensions: false,
}

// SetToDefault implements river.Defaulter.
Expand Down Expand Up @@ -72,5 +78,7 @@ func (a *Arguments) Convert() *azure_exporter.Config {
MetricNameTemplate: a.MetricNameTemplate,
MetricHelpTemplate: a.MetricHelpTemplate,
AzureCloudEnvironment: a.AzureCloudEnvironment,
ValidateDimensions: a.ValidateDimensions,
Regions: a.Regions,
}
}
2 changes: 2 additions & 0 deletions converter/internal/staticconvert/internal/build/azure_exporter.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,7 @@ func toAzureExporter(config *azure_exporter.Config) *azure.Arguments {
MetricNameTemplate: config.MetricNameTemplate,
MetricHelpTemplate: config.MetricHelpTemplate,
AzureCloudEnvironment: config.AzureCloudEnvironment,
ValidateDimensions: config.ValidateDimensions,
Regions: config.Regions,
}
}
57 changes: 40 additions & 17 deletions docs/sources/flow/reference/components/prometheus.exporter.azure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,30 @@ title: prometheus.exporter.azure

# prometheus.exporter.azure

The `prometheus.exporter.azure` component embeds [`azure-metrics-exporter`](https://github.com/webdevops/azure-metrics-exporter) to collect metrics from [Azure Monitor](https://azure.microsoft.com/en-us/products/monitor). The exporter uses [Azure Resource Graph](https://azure.microsoft.com/en-us/get-started/azure-portal/resource-graph/#overview) queries to identify resources for gathering metrics.
The `prometheus.exporter.azure` component embeds [`azure-metrics-exporter`](https://github.com/webdevops/azure-metrics-exporter) to collect metrics from [Azure Monitor](https://azure.microsoft.com/en-us/products/monitor).

The exporter supports all metrics defined by Azure Monitor. You can find the complete list of available metrics in the [Azure Monitor documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported).
Metrics for this integration are exposed with the template `azure_{type}_{metric}_{aggregation}_{unit}` by default. As an example,
the Egress metric for BlobService would be exported as `azure_microsoft_storage_storageaccounts_blobservices_egress_total_bytes`.

The exporter offers the following two options for gathering metrics.

1. (Default) Use an [Azure Resource Graph](https://azure.microsoft.com/en-us/get-started/azure-portal/resource-graph/#overview) query to identify resources for gathering metrics.
1. This query will make one API call per resource identified.
1. Subscriptions with a reasonable amount of resources can hit the [12000 requests per hour rate limit](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#subscription-and-tenant-limits) Azure enforces.
1. Set the regions to gather metrics from and get metrics for all resources across those regions.
1. This option will make one API call per subscription, dramatically reducing the number of API calls.
1. This approach does not work with all resource types, and Azure does not document which resource types do or do not work.
1. A resource type that is not supported produces errors that look like `Resource type: microsoft.containerservice/managedclusters not enabled for Cross Resource metrics`.
1. If you encounter one of these errors you must use the default Azure Resource Graph based option to gather metrics.

## Authentication

{{< param "PRODUCT_NAME" >}} must be running in an environment with access to Azure. The exporter uses the Azure SDK for go and supports [authentication](https://learn.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication?tabs=bash#2-authenticate-with-azure).

The account used by {{< param "PRODUCT_NAME" >}} needs:

- [Read access to the resources that will be queried by Resource Graph](https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview#permissions-in-azure-resource-graph)
- When using an Azure Resource Graph query, [read access to the resources that will be queried by Resource Graph](https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview#permissions-in-azure-resource-graph)
- Permissions to call the [Microsoft.Insights Metrics API](https://learn.microsoft.com/en-us/rest/api/monitor/metrics/list) which should be the `Microsoft.Insights/Metrics/Read` permission

## Usage
Expand Down Expand Up @@ -51,23 +62,27 @@ prometheus.exporter.azure LABEL {
You can use the following arguments to configure the exporter's behavior.
Omitted fields take their default values.

| Name | Type | Description | Default | Required |
| ----------------------------- | -------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------- | -------- |
| `subscriptions` | `list(string)` | List of subscriptions to scrap metrics from. | | yes |
| `resource_type` | `string` | The Azure Resource Type to scrape metrics for. | | yes |
| `metrics` | `list(string)` | The metrics to scrape from resources. | | yes |
| `resource_graph_query_filter` | `string` | The [Kusto query][] filter to apply when searching for resources. | | no |
| `metric_aggregations` | `list(string)` | Aggregations to apply for the metrics produced. | | no |
| `timespan` | `string` | [ISO8601 Duration][] over which the metrics are being queried. | `"PT1M"` (1 minute) | no |
| `included_dimensions` | `list(string)` | List of dimensions to include on the final metrics. | | no |
| `included_resource_tags` | `list(string)` | List of resource tags to include on the final metrics. | `["owner"]` | no |
| `metric_namespace` | `string` | Namespace for `resource_type` which have multiple levels of metrics. | | no |
| `azure_cloud_environment` | `string` | Name of the cloud environment to connect to. | `"azurecloud"` | no |
| `metric_name_template` | `string` | Metric template used to expose the metrics. | `"azure_{type}_{metric}_{aggregation}_{unit}"` | no |
| `metric_help_template` | `string` | Description of the metric. | `"Azure metric {metric} for {type} with aggregation {aggregation} as {unit}"` | no |
| Name | Type | Description | Default | Required |
|-------------------------------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|----------|
| `subscriptions` | `list(string)` | List of subscriptions to scrape metrics from. | | yes |
| `resource_type` | `string` | The Azure Resource Type to scrape metrics for. | | yes |
| `metrics` | `list(string)` | The metrics to scrape from resources. | | yes |
| `resource_graph_query_filter` | `string` | The [Kusto query][] filter to apply when searching for resources. Can't be used if `regions` is set. | | no |
| `regions` | `list(string)` | The list of regions for gathering metrics and enables gathering metrics for all resources in the subscription. Can't be used if `resource_graph_query_filter` is set. | | no |
| `metric_aggregations` | `list(string)` | Aggregations to apply for the metrics produced. | | no |
| `timespan` | `string` | [ISO8601 Duration][] over which the metrics are being queried. | `"PT1M"` (1 minute) | no |
| `included_dimensions` | `list(string)` | List of dimensions to include on the final metrics. | | no |
| `included_resource_tags` | `list(string)` | List of resource tags to include on the final metrics. | `["owner"]` | no |
| `metric_namespace` | `string` | Namespace for `resource_type` which have multiple levels of metrics. | | no |
| `azure_cloud_environment` | `string` | Name of the cloud environment to connect to. | `"azurecloud"` | no |
| `metric_name_template` | `string` | Metric template used to expose the metrics. | `"azure_{type}_{metric}_{aggregation}_{unit}"` | no |
| `metric_help_template` | `string` | Description of the metric. | `"Azure metric {metric} for {type} with aggregation {aggregation} as {unit}"` | no |
| `validate_dimensions` | `bool` | Enable dimension validation in the azure sdk | `false` | no |

The list of available `resource_type` values and their corresponding `metrics` can be found in [Azure Monitor essentials][].

The list of available `regions` to your subscription can be found by running the azure CLI command `az account list-locations --query '[].name'`.

The `resource_graph_query_filter` can be embedded into a template query of the form `Resources | where type =~ "<resource_type>" <resource_graph_query_filter> | project id, tags`.

Valid values for `metric_aggregations` are `minimum`, `maximum`, `average`, `total`, and `count`. If no aggregation is specified, the value is retrieved from the metric. For example, the aggregation value of the metric `Availability` in [Microsoft.ClassicStorage/storageAccounts](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-classicstorage-storageaccounts-metrics) is `average`.
Expand All @@ -78,6 +93,8 @@ Tags in `included_resource_tags` will be added as labels with the name `tag_<tag

Valid values for `azure_cloud_environment` are `azurecloud`, `azurechinacloud`, `azuregovernmentcloud` and `azurepprivatecloud`.

`validate_dimensions` is disabled by default to reduce the number of Azure exporter instances requires when a `resource_type` has metrics with varying dimensions. When `validate_dimensions` is enabled you will need one exporter instance per metric + dimension combination which is more tedious to maintain.

[Kusto query]: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/
[Azure Monitor essentials]: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported
[ISO8601 Duration]: https://en.wikipedia.org/wiki/ISO_8601#Durations
Expand Down Expand Up @@ -107,6 +124,9 @@ debug metrics.
prometheus.exporter.azure "example" {
subscriptions = SUBSCRIPTIONS
resource_type = "Microsoft.Storage/storageAccounts"
regions = [
"westeurope",
]
metric_namespace = "Microsoft.Storage/storageAccounts/blobServices"
metrics = [
"Availability",
Expand All @@ -120,8 +140,11 @@ prometheus.exporter.azure "example" {
"SuccessServerLatency",
"Transactions",
]
included_dimensions = [
"ApiName",
"TransactionType",
]
timespan = "PT1H"
resource_graph_query_filter = "where location == 'westeurope'"
}
// Configure a prometheus.scrape component to send metrics to.
Expand Down
Loading

0 comments on commit 5e9df54

Please sign in to comment.