Skip to content

Add cloud_provider block to database_observability.postgres #4675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 27, 2025
Merged

Add cloud_provider block to database_observability.postgres #4675

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .../reference/components/database_observability/database_observability.postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,39 @@ You can use the following blocks with `database_observability.postgres`:

| Block | Description | Required |
|------------------------------------|---------------------------------------------------|----------|
| [`cloud_provider`][cloud_provider] | Provide Cloud Provider information. | no |
| `cloud_provider` > [`aws`][aws] | Provide AWS database host information. | no |
| [`query_details`][query_details] | Configure the queries collector. | no |
| [`query_samples`][query_samples] | Configure the query samples collector. | no |
| [`schema_details`][schema_details] | Configure the schema and table details collector. | no |
| [`explain_plans`][explain_plans] | Configure the explain plans collector. | no |

The > symbol indicates deeper levels of nesting.
For example, `cloud_provider` > `aws` refers to a `aws` block defined inside an `cloud_provider` block.

[cloud_provider]: #cloud_provider
[aws]: #aws
[query_details]: #query_details
[query_samples]: #query_samples
[schema_details]: #schema_details
[explain_plans]: #explain_plans

### `cloud_provider`

The `cloud_provider` block has no attributes.
It contains zero or more [`aws`][aws] blocks.
You use the `cloud_provider` block to provide information related to the cloud provider that hosts the database under observation.
This information is appended as labels to the collected metrics.
The labels make it easier for you to filter and group your metrics.

### `aws`

The `aws` block supplies the [ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html) identifier for the database being monitored.

| Name | Type | Description | Default | Required |
|-------|----------|---------------------------------------------------------|---------|----------|
| `arn` | `string` | The ARN associated with the database under observation. | | yes |

### `query_details`

| Name | Type | Description | Default | Required |
Expand Down Expand Up @@ -96,6 +119,12 @@ database_observability.postgres "orders_db" {
data_source_name = "postgres://user:pass@localhost:5432/mydb"
forward_to = [loki.write.logs_service.receiver]
enable_collectors = ["query_details", "query_samples", "schema_details"]

cloud_provider {
aws {
arn = "your-rds-db-arn"
}
}
}

prometheus.scrape "orders_db" {
Expand Down
3 changes: 1 addition & 2 deletions internal/component/database_observability/mysql/collector/connection_info.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,9 @@ import (
"strings"

"github.com/go-sql-driver/mysql"
"github.com/grafana/alloy/internal/component/database_observability"
"github.com/prometheus/client_golang/prometheus"
"go.uber.org/atomic"

"github.com/grafana/alloy/internal/component/database_observability"
)

const ConnectionInfoName = "connection_info"
Expand Down
57 changes: 37 additions & 20 deletions internal/component/database_observability/postgres/collector/connection_info.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"regexp"
"strings"

"github.com/grafana/alloy/internal/component/database_observability"
"github.com/prometheus/client_golang/prometheus"
"go.uber.org/atomic"
)
Expand All @@ -22,13 +23,15 @@ type ConnectionInfoArguments struct {
DSN string
Registry *prometheus.Registry
EngineVersion string
CloudProvider *database_observability.CloudProvider
}

type ConnectionInfo struct {
DSN string
Registry *prometheus.Registry
EngineVersion string
InfoMetric *prometheus.GaugeVec
CloudProvider *database_observability.CloudProvider

running *atomic.Bool
}
Expand All @@ -38,7 +41,7 @@ func NewConnectionInfo(args ConnectionInfoArguments) (*ConnectionInfo, error) {
Namespace: "database_observability",
Name: "connection_info",
Help: "Information about the connection",
}, []string{"provider_name", "provider_region", "db_instance_identifier", "engine", "engine_version"})
}, []string{"provider_name", "provider_region", "provider_account", "db_instance_identifier", "engine", "engine_version"})

args.Registry.MustRegister(infoMetric)

Expand All @@ -47,6 +50,7 @@ func NewConnectionInfo(args ConnectionInfoArguments) (*ConnectionInfo, error) {
Registry: args.Registry,
EngineVersion: args.EngineVersion,
InfoMetric: infoMetric,
CloudProvider: args.CloudProvider,
running: &atomic.Bool{},
}, nil
}
Expand All @@ -56,34 +60,45 @@ func (c *ConnectionInfo) Name() string {
}

func (c *ConnectionInfo) Start(ctx context.Context) error {
c.running.Store(true)

var (
providerName = "unknown"
providerRegion = "unknown"
providerAccount = "unknown"
dbInstanceIdentifier = "unknown"
engine = "postgres"
engineVersion = "unknown"
)

parts, err := ParseURL(c.DSN)
if err != nil {
return err
}

if host, ok := parts["host"]; ok {
if strings.HasSuffix(host, "rds.amazonaws.com") {
if c.CloudProvider != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think when we get back to #4524 we can consolidate this logic a little more

if c.CloudProvider.AWS != nil {
providerName = "aws"
matches := rdsRegex.FindStringSubmatch(host)
if len(matches) > 3 {
dbInstanceIdentifier = matches[1]
providerRegion = matches[3]
providerAccount = c.CloudProvider.AWS.ARN.AccountID
providerRegion = c.CloudProvider.AWS.ARN.Region

// We only support RDS database for now. Resource types and ARN formats are documented at: https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonrds.html#amazonrds-resources-for-iam-policies
if resource := c.CloudProvider.AWS.ARN.Resource; strings.HasPrefix(resource, "db:") {
dbInstanceIdentifier = strings.TrimPrefix(resource, "db:")
}
} else if strings.HasSuffix(host, "postgres.database.azure.com") {
providerName = "azure"
matches := azureRegex.FindStringSubmatch(host)
if len(matches) > 1 {
dbInstanceIdentifier = matches[1]
}
} else {
Comment on lines +72 to +83
Copy link

Copilot AI Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The logic for determining connection metadata is split between cloud provider configuration (lines 72-83) and DSN parsing (lines 84-103). When cloud provider info is available, DSN-based detection is completely bypassed. Consider whether DSN parsing should be used as a fallback for missing cloud provider fields (like region or instance identifier) rather than being mutually exclusive, or add a comment explaining why this all-or-nothing approach is intentional.

Copilot uses AI. Check for mistakes.
parts, err := ParseURL(c.DSN)
if err != nil {
return err
}
if host, ok := parts["host"]; ok {
if strings.HasSuffix(host, "rds.amazonaws.com") {
providerName = "aws"
matches := rdsRegex.FindStringSubmatch(host)
if len(matches) > 3 {
dbInstanceIdentifier = matches[1]
providerRegion = matches[3]
}
} else if strings.HasSuffix(host, "postgres.database.azure.com") {
providerName = "azure"
matches := azureRegex.FindStringSubmatch(host)
if len(matches) > 1 {
dbInstanceIdentifier = matches[1]
}
}
}
}
Expand All @@ -93,7 +108,9 @@ func (c *ConnectionInfo) Start(ctx context.Context) error {
engineVersion = matches[1]
}

c.InfoMetric.WithLabelValues(providerName, providerRegion, dbInstanceIdentifier, engine, engineVersion).Set(1)
c.running.Store(true)

c.InfoMetric.WithLabelValues(providerName, providerRegion, providerAccount, dbInstanceIdentifier, engine, engineVersion).Set(1)
return nil
}

Expand Down
28 changes: 24 additions & 4 deletions internal/component/database_observability/postgres/collector/connection_info_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@ import (
"strings"
"testing"

"github.com/aws/aws-sdk-go-v2/aws/arn"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
"github.com/stretchr/testify/require"
"go.uber.org/goleak"

"github.com/grafana/alloy/internal/component/database_observability"
)

func TestConnectionInfo(t *testing.T) {
Expand All @@ -17,32 +20,48 @@ func TestConnectionInfo(t *testing.T) {
const baseExpectedMetrics = `
# HELP database_observability_connection_info Information about the connection
# TYPE database_observability_connection_info gauge
database_observability_connection_info{db_instance_identifier="%s",engine="%s",engine_version="%s",provider_name="%s",provider_region="%s"} 1
database_observability_connection_info{db_instance_identifier="%s",engine="%s",engine_version="%s",provider_account="%s",provider_name="%s",provider_region="%s"} 1
`

testCases := []struct {
name string
dsn string
engineVersion string
cloudProvider *database_observability.CloudProvider
expectedMetrics string
}{
{
name: "generic dsn",
dsn: "postgres://user:pass@localhost:5432/mydb",
engineVersion: "15.4",
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "unknown", "postgres", "15.4", "unknown", "unknown"),
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "unknown", "postgres", "15.4", "unknown", "unknown", "unknown"),
},
{
name: "AWS/RDS dsn",
dsn: "postgres://user:[email protected]:5432/mydb",
engineVersion: "15.4",
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "products-db", "postgres", "15.4", "aws", "us-east-1"),
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "products-db", "postgres", "15.4", "unknown", "aws", "us-east-1"),
},
{
name: "AWS/RDS dsn with cloud provider info supplied",
dsn: "postgres://user:[email protected]:5432/mydb",
engineVersion: "15.4",
cloudProvider: &database_observability.CloudProvider{
AWS: &database_observability.AWSCloudProviderInfo{
ARN: arn.ARN{
Region: "us-east-1",
AccountID: "some-account-123",
Resource: "db:products-db",
},
},
},
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "products-db", "postgres", "15.4", "some-account-123", "aws", "us-east-1"),
},
{
name: "Azure flexibleservers dsn",
dsn: "postgres://user:[email protected]:5432/mydb",
engineVersion: "15.4",
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "products-db", "postgres", "15.4", "azure", "unknown"),
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "products-db", "postgres", "15.4", "unknown", "azure", "unknown"),
},
}

Expand All @@ -53,6 +72,7 @@ func TestConnectionInfo(t *testing.T) {
DSN: tc.dsn,
Registry: reg,
EngineVersion: tc.engineVersion,
CloudProvider: tc.cloudProvider,
})
require.NoError(t, err)
require.NotNil(t, collector)
Expand Down
25 changes: 24 additions & 1 deletion internal/component/database_observability/postgres/component.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"sync"
"time"

"github.com/aws/aws-sdk-go-v2/aws/arn"
"github.com/blang/semver/v4"
"github.com/lib/pq"
"github.com/prometheus/client_golang/prometheus"
Expand Down Expand Up @@ -67,11 +68,19 @@ type Arguments struct {
EnableCollectors []string `alloy:"enable_collectors,attr,optional"`
DisableCollectors []string `alloy:"disable_collectors,attr,optional"`

CloudProvider *CloudProvider `alloy:"cloud_provider,block,optional"`
QuerySampleArguments QuerySampleArguments `alloy:"query_samples,block,optional"`
QueryTablesArguments QueryTablesArguments `alloy:"query_details,block,optional"`
SchemaDetailsArguments SchemaDetailsArguments `alloy:"schema_details,block,optional"`
ExplainPlanArguments ExplainPlanArguments `alloy:"explain_plans,block,optional"`
}

type CloudProvider struct {
AWS *AWSCloudProviderInfo `alloy:"aws,block,optional"`
}

ExplainPlanArguments ExplainPlanArguments `alloy:"explain_plans,block,optional"`
type AWSCloudProviderInfo struct {
ARN string `alloy:"arn,attr"`
}

type QuerySampleArguments struct {
Expand Down Expand Up @@ -342,6 +351,19 @@ func (c *Component) startCollectors(systemID string, engineVersion string) error
startErrors = append(startErrors, errorString)
}

var cloudProviderInfo *database_observability.CloudProvider
if c.args.CloudProvider != nil && c.args.CloudProvider.AWS != nil {
arn, err := arn.Parse(c.args.CloudProvider.AWS.ARN)
if err != nil {
level.Error(c.opts.Logger).Log("msg", "failed to parse AWS cloud provider ARN", "err", err)
}
cloudProviderInfo = &database_observability.CloudProvider{
AWS: &database_observability.AWSCloudProviderInfo{
ARN: arn,
},
}
Comment on lines +356 to +364
Copy link

Copilot AI Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When ARN parsing fails, the code logs an error but continues to use the uninitialized arn variable in the CloudProvider struct (line 362). This will result in an empty ARN being passed to the collector. Either return early after logging the error, or skip setting cloudProviderInfo when parsing fails.

Suggested change
arn, err := arn.Parse(c.args.CloudProvider.AWS.ARN)
if err != nil {
level.Error(c.opts.Logger).Log("msg", "failed to parse AWS cloud provider ARN", "err", err)
}
cloudProviderInfo = &database_observability.CloudProvider{
AWS: &database_observability.AWSCloudProviderInfo{
ARN: arn,
},
}
parsedARN, err := arn.Parse(c.args.CloudProvider.AWS.ARN)
if err != nil {
level.Error(c.opts.Logger).Log("msg", "failed to parse AWS cloud provider ARN", "err", err)
} else {
cloudProviderInfo = &database_observability.CloudProvider{
AWS: &database_observability.AWSCloudProviderInfo{
ARN: parsedARN,
},
}
}

Copilot uses AI. Check for mistakes.
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bunch of copy-paste from database_observability.mysql, but I think it's fine for now.

entryHandler := addLokiLabels(loki.NewEntryHandler(c.handler.Chan(), func() {}), c.instanceKey, systemID)

collectors := enableOrDisableCollectors(c.args)
Expand Down Expand Up @@ -384,6 +406,7 @@ func (c *Component) startCollectors(systemID string, engineVersion string) error
DSN: string(c.args.DataSourceName),
Registry: c.registry,
EngineVersion: engineVersion,
CloudProvider: cloudProviderInfo,
})
if err != nil {
logStartError(collector.ConnectionInfoName, "create", err)
Expand Down
Loading