Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open Telemetry : Adds implementation for network level Metrics #4872

Open
wants to merge 58 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
3d1d6ba
Added request level metrics
sourabh1007 Sep 13, 2024
60c411b
add IsClientMetricsEnabled option
sourabh1007 Sep 13, 2024
55b6505
added contract file
sourabh1007 Sep 13, 2024
6fb216c
wip
sourabh1007 Sep 16, 2024
841c7bc
adding test
sourabh1007 Sep 18, 2024
ee32e43
added documentation
sourabh1007 Oct 14, 2024
1ddbcbe
emit metrics
sourabh1007 Oct 16, 2024
7e9a344
fixed dimensions
sourabh1007 Oct 17, 2024
bc6f273
nonworking changes
sourabh1007 Oct 17, 2024
278c5c5
final commit
sourabh1007 Oct 18, 2024
8bae2a9
remove unnecessary dependencies
sourabh1007 Oct 18, 2024
cebace3
contract update
sourabh1007 Oct 18, 2024
6c32e52
fix merges
sourabh1007 Oct 18, 2024
f6974ea
remove console
sourabh1007 Oct 18, 2024
161abe9
add noops if disables
sourabh1007 Oct 18, 2024
2569ed9
added null check
sourabh1007 Oct 18, 2024
a3ee34d
[INTERNAL] CI: Fixes emulator set-up to leverage central SDK teams sc…
kirankumarkolli Oct 18, 2024
de96acc
VectorIndexDefinition: Adds Support for Partitioned DiskANN (#4792)
kundadebdatta Oct 18, 2024
2af0b05
Azurecore: Fixes upgrading azure core dependency to latest (#4819)
kirankumarkolli Oct 18, 2024
8d80c1c
DeleteAllItemsByPartitionKeyStreamAsync: Adds DeleteAllItemsByPartiti…
kirankumarkolli Oct 18, 2024
316b3d8
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Oct 22, 2024
c6a33b0
rename file
sourabh1007 Oct 22, 2024
4fd1193
refactor code
sourabh1007 Oct 23, 2024
033fda4
refactor code
sourabh1007 Oct 23, 2024
e92477a
perf tests
sourabh1007 Oct 23, 2024
cf5bb03
updated contracts
sourabh1007 Oct 24, 2024
6338908
code refactor
sourabh1007 Oct 24, 2024
321520c
refactored code
sourabh1007 Oct 25, 2024
51d485c
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Oct 25, 2024
7d03b8f
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Oct 26, 2024
1614b75
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Nov 1, 2024
9081e1a
added region contacted as dimension
sourabh1007 Nov 1, 2024
66efcf9
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Nov 6, 2024
ba63724
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Nov 12, 2024
a451d49
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Nov 14, 2024
736c292
perf fix
sourabh1007 Nov 14, 2024
6d6957b
inc perf test
sourabh1007 Nov 14, 2024
36d0ee1
perf results
sourabh1007 Nov 14, 2024
2509f3a
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Nov 14, 2024
36079d4
refactor according to versioning
sourabh1007 Nov 18, 2024
b7ae941
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Nov 19, 2024
d01a3c4
Merge branch 'master' into users/sourabhjain/otelmetriccpu
sourabh1007 Nov 20, 2024
e0efe5b
fix test
sourabh1007 Nov 20, 2024
369160b
refactor code
sourabh1007 Nov 20, 2024
1f64899
added try catch
sourabh1007 Nov 20, 2024
79b7307
added console
sourabh1007 Nov 20, 2024
f184813
updated contract
sourabh1007 Nov 20, 2024
ef2f2a5
code refatoring
sourabh1007 Nov 20, 2024
731951a
fix tetss
sourabh1007 Nov 20, 2024
a9eeba6
refactor code
sourabh1007 Nov 20, 2024
963824c
first drfat
sourabh1007 Nov 8, 2024
35c3141
updated
sourabh1007 Nov 18, 2024
eb289e5
fix test
sourabh1007 Nov 20, 2024
1280e13
updated sln
sourabh1007 Nov 21, 2024
675c49f
refactor code
sourabh1007 Nov 21, 2024
b4e2045
test fix
sourabh1007 Nov 21, 2024
3f30e87
Merge branch 'master' into users/sourabhjain/otelnetworkmetrics
sourabh1007 Nov 24, 2024
e4a7459
cleanup
sourabh1007 Nov 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<ClientOfficialVersion>3.46.0</ClientOfficialVersion>
<ClientPreviewVersion>3.47.0</ClientPreviewVersion>
<ClientPreviewSuffixVersion>preview.0</ClientPreviewSuffixVersion>
<DirectVersion>3.37.1</DirectVersion>
<DirectVersion>3.37.2</DirectVersion>
<EncryptionOfficialVersion>2.0.4</EncryptionOfficialVersion>
<EncryptionPreviewVersion>2.1.0</EncryptionPreviewVersion>
<EncryptionPreviewSuffixVersion>preview4</EncryptionPreviewSuffixVersion>
Expand Down
3 changes: 2 additions & 1 deletion Microsoft.Azure.Cosmos/src/DocumentClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -959,7 +959,8 @@ internal virtual void Initialize(Uri serviceEndpoint,
if (this.cosmosClientTelemetryOptions.IsClientMetricsEnabled)
{
CosmosDbOperationMeter.Initialize();

CosmosDbNetworkMeter.Initialize();

CosmosDbOperationMeter.AddInstanceCount(this.ServiceEndpoint);
}

Expand Down
17 changes: 15 additions & 2 deletions Microsoft.Azure.Cosmos/src/Resource/ClientContextCore.cs
Original file line number Diff line number Diff line change
Expand Up @@ -533,16 +533,23 @@ private async Task<TResult> RunWithDiagnosticsHelperAsync<TResult>(
&& (!this.ClientOptions.CosmosClientTelemetryOptions.DisableDistributedTracing || this.ClientOptions.CosmosClientTelemetryOptions.IsClientMetricsEnabled))
{
// Extracts and records telemetry data from the result of the operation.
OpenTelemetryAttributes response = openTelemetry?.Item2(result);
OpenTelemetryAttributes otelAttributes = openTelemetry?.Item2(result);

// Records the telemetry attributes for Distributed Tracing (if enabled)
recorder.Record(response);
recorder.Record(otelAttributes);

// Records metrics such as request units, latency, and item count for the operation.
CosmosDbOperationMeter.RecordTelemetry(getOperationName: getOperationName,
accountName: this.client.Endpoint,
containerName: containerName,
databaseName: databaseName,
attributes: otelAttributes);

CosmosDbNetworkMeter.RecordTelemetry(getOperationName: getOperationName,
accountName: this.client.Endpoint,
containerName: containerName,
databaseName: databaseName,
attributes: otelAttributes);
attributes: response);
}
return result;
Expand Down Expand Up @@ -584,6 +591,12 @@ private async Task<TResult> RunWithDiagnosticsHelperAsync<TResult>(
containerName: containerName,
databaseName: databaseName,
ex: cosmosException);
CosmosDbNetworkMeter.RecordTelemetry(getOperationName: getOperationName,
accountName: this.client.Endpoint,
containerName: containerName,
databaseName: databaseName,
ex: cosmosException);

}

throw;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ namespace Microsoft.Azure.Cosmos.Telemetry
using System;
using System.Collections.Generic;
using global::Azure.Core;
using Microsoft.Azure.Cosmos.Tracing.TraceData;

internal sealed class AppInsightClassicAttributeKeys : IActivityAttributePopulator
{
Expand Down Expand Up @@ -162,12 +163,29 @@ public void PopulateAttributes(DiagnosticScope scope, QueryTextMode? queryTextMo
}
}

public KeyValuePair<string, object>[] PopulateNetworkMeterDimensions(string operationName, Uri accountName, string containerName, string databaseName, OpenTelemetryAttributes attributes, CosmosException ex, ClientSideRequestStatisticsTraceDatum.StoreResponseStatistics tcpStats = null, ClientSideRequestStatisticsTraceDatum.HttpResponseStatistics? httpStats = null)
{
return new KeyValuePair<string, object>[]
{
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.ContainerName, containerName),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.DbName, databaseName),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.ServerAddress, accountName?.Host),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.DbOperation, operationName),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.StatusCode, (int)(attributes?.StatusCode ?? ex?.StatusCode)),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.SubStatusCode, attributes?.SubStatusCode ?? ex?.SubStatusCode)
};
}

public KeyValuePair<string, object>[] PopulateOperationMeterDimensions(string operationName, string containerName, string databaseName, Uri accountName, OpenTelemetryAttributes attributes, CosmosException ex)
{
return new KeyValuePair<string, object>[]
{
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.ContainerName, containerName),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.DbName, databaseName),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.ServerAddress, accountName?.Host),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.DbOperation, operationName),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.StatusCode, (int)(attributes?.StatusCode ?? ex?.StatusCode)),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.SubStatusCode, attributes?.SubStatusCode ?? ex?.SubStatusCode)
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.ServerAddress, accountName.Host),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.DbOperation, operationName),
new KeyValuePair<string, object>(AppInsightClassicAttributeKeys.StatusCode, (int)(attributes?.StatusCode ?? ex?.StatusCode)),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ namespace Microsoft.Azure.Cosmos
public sealed class CosmosDbClientMetrics
{
/// <summary>
/// OperationMetrics
/// Operation Metrics
/// </summary>
public static class OperationMetrics
{
Expand Down Expand Up @@ -101,6 +101,120 @@ public static class Description
}
}

/// <summary>
/// Network Metrics
/// </summary>
public static class NetworkMetrics
{
/// <summary>
/// the name of the operation meter
/// </summary>
public const string MeterName = "Azure.Cosmos.Client.Request";

/// <summary>
/// Version of the operation meter
/// </summary>
public const string Version = "1.0.0";

/// <summary>
/// Metric Names
/// </summary>
public static class Name
{
/// <summary>
/// Network Call Latency
/// </summary>
public const string Latency = "db.client.cosmosdb.request.duration";

/// <summary>
/// Request Payload Size
/// </summary>
public const string RequestBodySize = "db.client.cosmosdb.request.body.size";

/// <summary>
/// Request Payload Size
/// </summary>
public const string ResponseBodySize = "db.client.cosmosdb.response.body.size";

/// <summary>
/// Channel Aquisition Latency
/// </summary>
public const string ChannelAquisitionLatency = "db.client.cosmosdb.request.channel_aquisition.duration";

/// <summary>
/// Backend Server Latency
/// </summary>
public const string BackendLatency = "db.server.cosmosdb.request.duration";

/// <summary>
/// Transit Time Latency
/// </summary>
public const string TransitTimeLatency = "db.client.cosmosdb.request.transit.duration";

/// <summary>
/// Received Time Latency
/// </summary>
public const string ReceivedTimeLatency = "db.client.cosmosdb.request.received.duration";
}

/// <summary>
/// Unit for metrics
/// </summary>
public static class Unit
{
/// <summary>
/// Unit representing bytes
/// </summary>
public const string Bytes = "bytes";

/// <summary>
/// Unit representing time in seconds
/// </summary>
public const string Sec = "s";
}

/// <summary>
/// Provides descriptions for metrics.
/// </summary>
public static class Description
{
/// <summary>
/// Network Call Latency
/// </summary>
public const string Latency = "Duration of client requests.";

/// <summary>
/// Request Payload Size
/// </summary>
public const string RequestBodySize = "Size of client request body.";

/// <summary>
/// Request Payload Size
/// </summary>
public const string ResponseBodySize = "Size of client response body.";

/// <summary>
/// Channel Aquisition Latency
/// </summary>
public const string ChannelAquisitionLatency = "The duration of the successfully established outbound TCP connections. i.e. Channel Aquisition Time (for direct mode).";

/// <summary>
/// Backend Server Latency
/// </summary>
public const string BackendLatency = "Backend Latency (for direct mode).";

/// <summary>
/// Transit Time Latency
/// </summary>
public const string TransitTimeLatency = "Time spent on the wire (for direct mode).";

/// <summary>
/// Received Time Latency
/// </summary>
public const string ReceivedTimeLatency = "Time spent on 'Received' stage (for direct mode).";
}
}

/// <summary>
/// Buckets
/// </summary>
Expand Down
Loading
Loading