[DOC] Tempo 2.3 release notes

docs/sources/tempo/release-notes/v2-2.md

@@ -108,19 +108,16 @@ The following fields were removed or renamed.
    </td>
   </tr>
   <tr>
-   <td><code> ```</code>
+   <td>
 <p>
 <code>  query_frontend:</code>
 <p>
 <code>    tolerate_failed_blocks: &lt;int></code>
-<p>
-<code>  ```</code>
    </td>
    <td>Remove support for `tolerant_failed_blocks` [PR <a href="https://github.com/grafana/tempo/pull/2416">2416</a>]
    </td>
   </tr>
   <tr>
-   <td><code>```</code>
 <p>
 <code>storage:</code>
 <p>
@@ -130,7 +127,6 @@ The following fields were removed or renamed.
 <p>
 <code>      insecure_skip_verify: true   // renamed to tls_insecure_skip_verify</code>
 <p>
-<code>```</code>
    </td>
    <td>Renamed `insecure_skip_verify` to `tls_insecure_skip_verify` [PR <a href="https://github.com/grafana/tempo/pull/2407">2407</a>]
    </td>

docs/sources/tempo/release-notes/v2-3.md

@@ -0,0 +1,203 @@
+---
+title: Version 2.3 release notes
+menuTitle: V2.3
+description: Release notes for Grafana Tempo 2.3
+weight: 45
+---
+
+# Version 2.3 release notes
+
+The Tempo team is pleased to announce the release of Tempo 2.3.
+
+This release gives you:
+
+* New [structural operators]({{< relref "../traceql#experimental-structural" >}}) in TraceQL:
+    * Ancestor (`<<`)
+    * Parent (`<`)
+    * Not-child (`!>`) (experimental)
+    * Not descendant (`!>>`) (experimental)
+    * Not ancestor (`!<<`) (experimental)
+    * Not parent (`!<`) (experimental)
+    * Not siblings (`!~`) (experimental)
+
+* TraceQL support for [searching quoted attributes]({{< relref "../traceql#experimental-structural" >}}) (for example, `{ span."name with spaces" = "foo" }`).
+* [Dedicated attribute columns]({{< relref "../operations/dedicated_columns" >}}) for improving query performance on your most frequently queried attributes
+
+Tempo 2.3 introduces [vParquet3]({{< relref "../configuration/parquet" >}}), a Parquet version designed to be more compatible with other Parquet implementations, available as a production-ready option.
+This block format improves query performance relative to previous formats. 
+
+Read the **Tempo 2.3 blog post** for more examples and details about these improvements.
+
+These release notes highlight the most important features and bugfixes. For a complete list, refer to the [Tempo changelog](https://github.com/grafana/tempo/releases).
+
+## Features and enhancements
+
+The most important features and enhancements in Tempo 2.3 are highlighted below.
+
+### vParquet3 promoted to stable
+
+[vParquet3]({{< relref "../configuration/parquet" >}}), an updated Parquet block format introduced in Tempo 2.2, has been promoted to stable and is production ready.
+We're excited about vParquet3 relative to prior formats because of its support for dedicated attribute columns, which help speed up queries on your largest and most queried attributes. [[PR 2649](https://github.com/grafana/tempo/pull/2649)]
+
+To read more about the design of vParquet3, refer to [the design proposal](https://github.com/grafana/tempo/blob/main/docs/design-proposals/2023-05%20vParquet3.md).
+For general information, refer to [the Apache Parquet schema]({{< relref "../operations/schema" >}}).
+
+#### Dedicated attribute columns
+
+Dedicated attribute columns improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list.
+
+Introduced with vParquet3, dedicated attribute columns are only available when using this storage format.
+
+To learn more about this new feature, refer to the [Dedicated attribute columns]({{< relref "../operations/dedicated_columns" >}}) documentation.
+
+### TraceQL
+
+Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data.
+To learn more about the TraceQL syntax, see the [TraceQL documentation]({{< relref "../traceql" >}}).
+For information on planned future extensions to the TraceQL language, refer to [future work]({{< relref "../traceql/architecture" >}}).
+
+We’ve made the following improvements to TraceQL:
+
+* Added two structural operators, ancestor (`<<`) and parent (`<`) ([documentation]({{< relref "../traceql#experimental-structural" >}})) [[PR 2877](https://github.com/grafana/tempo/pull/2877)]
+
+* Added experimental not structural operators: `!>>`, `!>`, `!<<`, `!<`, and `!~` ([documentation]({{< relref "../traceql#experimental-structural" >}})) [[PR 2993](https://github.com/grafana/tempo/pull/2993)]
+
+* Added quoted attribute syntax so you can search for attributes with spaces and other special characters in their names ([documentation]({{< relref "../traceql#experimental-structural" >}})) [[PR 3004](https://github.com/grafana/tempo/pull/3004)]
+
+* Improved the performance of TraceQL [`select()` queries]({{< relref "../traceql#selection" >}}). Metrics-summary now also correctly handles missing attributes. [[PR 2765](https://github.com/grafana/tempo/pull/2765)]
+
+* Added support for searching by OpenTelemetry’s [span status message ](https://github.com/open-telemetry/opentelemetry-proto/blob/afcd2aa7f728216d5891ffc0d83f09f0278a6611/opentelemetry/proto/trace/v1/trace.proto#L260)using `statusMessage` intrinsic attribute ([documentation]({{< relref "../traceql#intrinsic-fields" >}})) [[PR 2848](https://github.com/grafana/tempo/pull/2848)]
+
+* Fixed cases where an empty filter (`{}`) didn't return expected results [[PR 2498](https://github.com/grafana/tempo/issues/2498)]
+
+### Metrics-generator
+
+We’ve made the following improvements to metrics-generator:
+
+* Added a scope query parameter to `/api/overrides` so users can choose between fetching the overrides stored by the API and the merged overrides (those actually used by Tempo) [[PR 2915](https://github.com/grafana/tempo/pull/2915), [#3018](https://github.com/grafana/tempo/pull/3018)]
+* Added `TempoUserConfigurableOverridesReloadFailing` alert [[PR 2784](https://github.com/grafana/tempo/pull/2784)]
+* Added a metrics-generator configuration option to enable/disable X-Scope-OrgID headers on remote write to better support single-tenant setups. [[PR 2974](https://github.com/grafana/tempo/pull/2974)]
+* Allowed metrics-generator ingestion slack to be configured on a per-tenant, rather than global, basis [[PR 2589](https://github.com/grafana/tempo/pull/2589)]
+* Added several metrics-generator fields to user-configurable overrides, including histogram buckets, collection-interval, and span metrics filter policies [[PR 2906](https://github.com/grafana/tempo/pull/2906), [2711](https://github.com/grafana/tempo/pull/2711), [2928](https://github.com/grafana/tempo/pull/2928), [2899](https://github.com/grafana/tempo/pull/2899)]
+
+## Upgrade considerations
+
+When [upgrading]({{< relref "../setup/upgrade" >}}) to Tempo 2.3, be aware of these considerations and breaking changes.
+
+### Transition to vParquet 3
+
+Although the vParquet3 format isn't yet the default, it's production ready and we highly recommend switching to it for improved query performance and [dedicated attribute columns]({{< relref "../operations/dedicated_columns" >}}).
+
+Upgrading to Tempo 2.3 doesn’t modify the Parquet block format. You can use Tempo 2.3 with vParquet2 or vParquet3. vParquet2 remains the default backend for Tempo 2.3; vParquet3 is available as a stable option.
+
+{{% admonition type="note" %}}
+Tempo 2.2 can’t read data stored in vParquet3.
+{{% /admonition %}}
+
+For information on upgrading, refer to [Change the block format to vParquet3]({{< relref "../setup/upgrade" >}}) upgrade documentation.
+
+### Azure SDK v2
+
+If you are using Azure storage, we recommend using the v2 SDK, [azure-sdk-for-go](https://github.com/Azure/azure-sdk-for-go).
+You can use the `use_v2_sdk` configure option for switching. For more information, refer to the [Storack block configuration example documentation]({{< relref "../configuration#storage-block-configuration-example" >}}). [[PR 2952](https://github.com/grafana/tempo/issues/2952)]
+
+### Produce debug metrics with the distributor
+
+You can now enable a new configuration block for the distributor to produce debug metrics. These metrics can be particularly useful when tracking down a process that's creating an exceptional amount of spans. [[PR 3008](https://github.com/grafana/tempo/pull/3008)]
+
+```yaml
+distributor:
+  metric_received_spans:
+    enabled: false
+    root_only: false
+```
+
+### Changes to the Overrides module configuration
+
+We’ve added a new `defaults` block to the overrides module for configuring global or per-tenant settings. The Overrides change to indented syntax. For more information, read the [Overrides configuration documentation]({{< relref "../configuration#overrides" >}}).
+
+You can also use the Tempo CLI to migrate configurations. Refer to the [documentation]({{< relref "../operations/tempo_cli#migrate-overrides-config-command" >}}). [[PR 2688](https://github.com/grafana/tempo/pull/2688)]
+
+The old configuration block looked like this:
+
+```yaml
+overrides:
+  ingestion_rate_strategy: local
+  ingestion_rate_limit_bytes: 12345
+  ingestion_burst_size_bytes: 67890
+  max_search_duration: 17s
+  forwarders: ['foo']
+  metrics_generator_processors: [service-graphs, span-metrics]
+```
+
+Here is an example of the new configuration block:
+
+```yaml
+overrides:
+  defaults:
+    ingestion:
+      rate_strategy: local
+      rate_limit_bytes: 12345
+      burst_size_bytes: 67890
+    read:
+      max_search_duration: 17s
+    forwarders: ['foo']
+    metrics_generator:
+      processors: [service-graphs, span-metrics]
+
+```
+
+### Removed or renamed configuration parameters
+
+<table>
+  <tr>
+   <td><strong>Parameter</strong>
+   </td>
+   <td><strong>Comments</strong>
+   </td>
+  </tr>
+  <tr>
+   <td><code>distributor.log_received_traces</code>
+   </td>
+   <td>Use the <code>distributor.log_received_spans</code> configuration block instead. [PR <a href="https://github.com/grafana/tempo/pull/3008">#3008</a>]
+   </td>
+  </tr>
+  <tr>
+   <td><code>tempo_query_frontend_queries_total{op="searchtags|metrics"}</code>
+   </td>
+   <td>Removed deprecated frontend metrics configuration option
+   </td>
+  </tr>
+</table>
+
+## Security fixes
+
+The following vulnerabilities have been addressed:
+
+* Updated Alpine image version to 3.18 to patch [CVE-2022-48174](https://nvd.nist.gov/vuln/detail/CVE-2022-48174) [PR 3046](https://github.com/grafana/tempo/pull/3046)
+* Corrected a HTTP rapid reset issue vulnerability ([CVE-2023-39325](https://github.com/advisories/GHSA-4374-p667-p6c8)) [PR 3017](https://github.com/grafana/tempo/pull/3017)
+
+## Bugfixes
+
+For a complete list, refer to the [Tempo changelog](https://github.com/grafana/tempo/releases).
+
+* Loaded defaults for the internal server [PR 3041](https://github.com/grafana/tempo/pull/3041)
+* Fixed  pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions [PR 3012](https://github.com/grafana/tempo/pull/3012)
+* Fixed a panic in metrics-summary API PR [#2738](https://github.com/grafana/tempo/pull/2738)
+* Fixed a rare deadlock when uploading blocks to Azure Blob Storage [PR 2129](https://github.com/grafana/tempo/issues/2129)
+* Only search ingester blocks that fall within the request time range. [PR 2783](https://github.com/grafana/tempo/pull/2783)
+* Aligned `tempo_query_frontend_queries_total` and `tempo_query_frontend_queries_within_slo_total`. [PR 2840](https://github.com/grafana/tempo/pull/2840)
+  This query now correctly tells you `%age` of requests that are within SLO:
+
+  ```
+  sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op)
+  /
+  sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
+  ```
+
+* Fixed support for blob storage in Azure Stack Hub as a backend. [PR 2853](https://github.com/grafana/tempo/pull/2853)
+* Respected spss on GRPC streaming. [PR 2971](https://github.com/grafana/tempo/pull/2840)
+* Moved empty root span substitution from `querier` to `query-frontend`. [PR 2671](https://github.com/grafana/tempo/issues/2671)
+* Ingester errors correctly propagate on the query path [PR 2935](https://github.com/grafana/tempo/issues/2935)
+* Fixed an issue where ingester didn’t stop a query after timeout [PR 3031](https://github.com/grafana/tempo/pull/3031)
+* Reordered the S3 credential chain and upgraded `minio-go`. `native_aws_auth_enabled` is deprecated [PR 3006](https://github.com/grafana/tempo/pull/3006)

docs/sources/tempo/setup/upgrade.md

@@ -15,6 +15,110 @@ This upgrade guide applies to on-premise installations and not for Grafana Cloud
 
 >**TIP**: You can check your configuration options using the [`status` API endpoint]({{< relref "../api_docs#status" >}}) in your Tempo installation.
 
+## Upgrade to Tempo 2.3
+
+Tempo 2.3 has several considerations for any upgrade:
+
+* vParquet3 is available as a stable, production-read block format
+* Configuration option to use Azure SDK v2
+* New `defaults` block in Overrides module configuration
+* Several configuration parameters have been renamed or removed.
+
+For a complete list of changes, enhancements, and bug fixes, refer to the [Tempo 2.3 changelog](https://github.com/grafana/tempo/releases).
+
+### Production-ready vParquet3 block format
+
+vParquet3 provides improved query performance and [dedicated attribute columns]({{< relref "../operations/dedicated_columns" >}}).
+
+This block format is required for using dedicated attribute columns.
+
+While vParquet2 remains the default backend for Tempo 2.3, vParquet3 is available as a stable option. 
+Both work with Tempo 2.3.
+
+Upgrading to Tempo 2.3 doesn’t modify the Parquet block format. 
+
+{{% admonition type="note" %}}
+Tempo 2.2 can’t read data stored in vParquet3.
+{{% /admonition %}}
+
+Recommended update process:
+
+1. Upgrade your Tempo installation to version 2.3, remaining on vParquet2.
+2. Verify the upgrade is stable and performs as expected. If you notice any issues, you can downgrade to version 2.2, and data remains readable.
+3. [Change the block format to vParquet3]({{< relref "../configuration/parquet" >}}).
+
+If you notice any issues on step 3 using the new block format, you can downgrade to vParquet2.
+All your data remains readable in Tempo 2.3.
+However, if you have vParquet3 blocks and have to downgrade to Tempo 2.2, you will have data loss. 
+
+### Use Azure SDK v2
+
+If you are using Azure storage, we recommend using the v2 SDK, [azure-sdk-for-go](https://github.com/Azure/azure-sdk-for-go).
+You can use the `use_v2_sdk` configure option for switching.
+
+For more information, refer to the [Storage block configuration example documentation]({{< relref "../configuration#storage-block-configuration-example" >}}).
+
+### New `defaults` block in Overrides module configuration
+
+The Overrides module has a new `defaults` block for configuring global or per-tenant settings.
+The Overrides format now includes changes to indented syntax.
+For more information, read the [Overrides configuration documentation]({{< relref "../configuration#overrides" >}}).
+
+You can also use the Tempo CLI to migrate configurations. Refer to the [tempo-cli documentation]({{< relref "../operations/tempo_cli#migrate-overrides-config-command" >}}).
+
+The old configuration block looked like this:
+
+```yaml
+overrides:
+  ingestion_rate_strategy: local
+  ingestion_rate_limit_bytes: 12345
+  ingestion_burst_size_bytes: 67890
+  max_search_duration: 17s
+  forwarders: ['foo']
+  metrics_generator_processors: [service-graphs, span-metrics]
+```
+
+The new configuration block looks like this:
+
+```yaml
+overrides:
+  defaults:
+    ingestion:
+      rate_strategy: local
+      rate_limit_bytes: 12345
+      burst_size_bytes: 67890
+    read:
+      max_search_duration: 17s
+    forwarders: ['foo']
+    metrics_generator:
+      processors: [service-graphs, span-metrics]
+
+```
+
+### Removed or renamed configuration parameters
+
+<table>
+  <tr>
+   <td><strong>Parameter</strong>
+   </td>
+   <td><strong>Comments</strong>
+   </td>
+  </tr>
+  <tr>
+   <td><code>distributor.log_received_traces</code>
+   </td>
+   <td>Use the <code>distributor.log_received_spans</code> configuration block instead. [PR <a href="https://github.com/grafana/tempo/pull/3008">#3008</a>]
+   </td>
+  </tr>
+  <tr>
+   <td><code>tempo_query_frontend_queries_total{op="searchtags|metrics"}</code>
+   </td>
+   <td>Removed deprecated frontend metrics configuration option
+   </td>
+  </tr>
+</table>
+
+
 ## Upgrade to Tempo 2.2
 
 Tempo 2.2 has several considerations for any upgrade:
@@ -28,7 +132,7 @@ For a complete list of changes, enhancements, and bug fixes, refer to the [Tempo
 
 While not a breaking change, upgrading to Tempo 2.2 will by default change Tempo’s block format to vParquet2.
 
-To stay on a previous block format, read the[Parquet configuration documentation]({{< relref "../configuration/parquet#choose-a-different-block-format" >}}).
+To stay on a previous block format, read the [Parquet configuration documentation]({{< relref "../configuration/parquet#choose-a-different-block-format" >}}).
 We strongly encourage upgrading to vParquet2 as soon as possible as this is required for using structural operators in your TraceQL queries and provides query performance improvements, in particular on queries using the `duration` intrinsic.
 
 ### Updated JSonnet supports `statefulset` for the metrics-generator
@@ -92,8 +196,7 @@ All Prometheus metrics exposed by Tempo on its `/metrics` endpoint that were pre
 
 Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR [2008](https://github.com/grafana/tempo/pull/2008))
 
-The ``query_frontend_result_metrics_inspected_bytes`` metric was removed in favor of ``query_frontend_bytes_processed_per_second`.`
-
+The `query_frontend_result_metrics_inspected_bytes` metric was removed in favor of `query_frontend_bytes_processed_per_second`.
 
 ## Upgrade from Tempo 1.5 to 2.0
 
@@ -109,7 +212,9 @@ Tempo 2.0 marks a major milestone in Tempo’s development. When planning your u
 
 Once you upgrade to Tempo 2.0, there is no path to downgrade.
 
->**Note**: There is a potential issue loading Tempo 1.5's experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues please [report them](https://github.com/grafana/tempo/issues/new?assignees=&labels=&template=bug_report.md&title=) so we can isolate and fix this issue.
+{{% admonition type="note" %}}
+There is a potential issue loading Tempo 1.5's experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues please [report them](https://github.com/grafana/tempo/issues/new?assignees=&labels=&template=bug_report.md&title=) so we can isolate and fix this issue.
+{{% /admonition %}}
 
 ### Check Tempo installation resource allocation