diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 2da8649f7e7..d66bef330a5 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -7074,11 +7074,6 @@ menu: url: logs/log_configuration/rehydrating parent: log_configuration weight: 210 - - name: Archive Search - identifier: log_configuration_archive_search - url: logs/log_configuration/archive_search - parent: log_configuration - weight: 211 - name: Forwarding identifier: log_configuration_forwarding url: logs/log_configuration/forwarding_custom_destinations/ @@ -7174,6 +7169,11 @@ menu: url: logs/explorer/saved_views/ parent: log_explorer weight: 514 + - name: Archive Search + identifier: log_explorer_archive_search + url: logs/explorer/archive_search/ + parent: log_explorer + weight: 515 - name: Error Tracking url: logs/error_tracking/ parent: log_management diff --git a/content/en/glossary/terms/archive_search.md b/content/en/glossary/terms/archive_search.md new file mode 100644 index 00000000000..c4699b6d4b8 --- /dev/null +++ b/content/en/glossary/terms/archive_search.md @@ -0,0 +1,7 @@ +--- +id: archive_search +title: Archive Search +core_product: + - log management +--- +Archive Search lets you query logs directly from long-term cloud storage archives without re-indexing them. Results stream in real time and you are charged only for the data scanned. Use **Search & Rehydration** mode when you need full platform access or longer retention. See [Archive Search](/logs/explorer/archive_search/). diff --git a/content/en/glossary/terms/rehydration.md b/content/en/glossary/terms/rehydration.md index 219ef73d7e7..71317e7405b 100644 --- a/content/en/glossary/terms/rehydration.md +++ b/content/en/glossary/terms/rehydration.md @@ -4,4 +4,8 @@ title: Rehydration core_product: - log management --- -Rehydration is when archived logs are recalled back into Datadog. \ No newline at end of file +
+Archive Search is the recommended way to query archived logs. Use its Search & Rehydration mode when you need full platform access or longer retention. +
+ +Rehydration is when archived logs are re-indexed back into Datadog for full platform access. \ No newline at end of file diff --git a/content/en/logs/log_configuration/archive_search.md b/content/en/logs/explorer/archive_search.md similarity index 99% rename from content/en/logs/log_configuration/archive_search.md rename to content/en/logs/explorer/archive_search.md index 66f165cd66a..d01dbb42172 100644 --- a/content/en/logs/log_configuration/archive_search.md +++ b/content/en/logs/explorer/archive_search.md @@ -1,6 +1,8 @@ --- title: Archive Search description: Instantly search and analyze logs from long-term archives without re-indexing. +aliases: + - /logs/log_configuration/archive_search/ further_reading: - link: "/logs/explorer/" tag: "Documentation" diff --git a/content/en/logs/guide/best-practices-for-log-management.md b/content/en/logs/guide/best-practices-for-log-management.md index 386f2e08088..6395e2d76e6 100644 --- a/content/en/logs/guide/best-practices-for-log-management.md +++ b/content/en/logs/guide/best-practices-for-log-management.md @@ -68,11 +68,11 @@ If you want to retain logs for an extended time while maintaining querying speed ### Set up multiple archives for long-term storage -If you want to store your logs for longer periods of time, set up [Log Archives][2] to send your logs to a storage-optimized system, such as Amazon S3, Azure Storage, or Google Cloud Storage. When you want to use Datadog to analyze those logs, use [Log Rehydration][3]™ to capture those logs back in Datadog. With multiple archives, you can both segment logs for compliance reasons and keep rehydration costs under control. +If you want to store your logs for longer periods of time, set up [Log Archives][2] to send your logs to a storage-optimized system, such as Amazon S3, Azure Storage, or Google Cloud Storage. Use [Archive Search][32] to query these logs directly from storage in real time. When you need full platform access or longer retention, use Archive Search's **Search & Rehydration** mode. With multiple archives, you can segment logs for compliance reasons and keep costs under control. -#### Set up max scan size to manage expensive rehydrations +#### Set up max scan size to manage costs -Set a limit on the volume of logs that can be rehydrated at one time. When setting up an archive, you can define the maximum volume of log data that can be scanned for Rehydration. See [Define maximum scan size][4] for more information. +Set a limit on the volume of logs that can be scanned at one time. When setting up an archive, you can define the maximum volume of log data that can be scanned per Archive Search query or rehydration. See [Define maximum scan size][4] for more information. ### Set up RBAC for custom roles @@ -225,7 +225,6 @@ If you want to see user activities, such as who changed the retention of an inde [1]: https://app.datadoghq.com/logs/pipelines/indexes [2]: /logs/log_configuration/archives/ -[3]: /logs/log_configuration/rehydrating/ [4]: /logs/log_configuration/archives/?tab=awss3#define-maximum-scan-size [5]: /account_management/rbac/?tab=datadogapplication#datadog-default-roles [6]: https://app.datadoghq.com/ @@ -253,4 +252,5 @@ If you want to see user activities, such as who changed the retention of an inde [28]: /monitors/configuration/?tab=thresholdalert#evaluation-window [29]: /observability_pipelines/ [30]: /logs/log_configuration/flex_logs/ -[31]: https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&group_by=%40asset.new_value.query.indexes](https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&agg_m=count&agg_m_source=base&agg_q=%40asset.new_value.query.indexes&agg_q_source=base&agg_t=count&audit__diff=unified&cols=log_usr.id%2Clog_action%2Clog_evt.name&fromUser=true&messageDisplay=expanded-md&refresh_mode=sliding&stream_sort=desc&top_n=10&top_o=top&viz=query_table&x_missing=true&from_ts=1768733389060&to_ts=1771325389060&live=true \ No newline at end of file +[31]: https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&agg_m=count&agg_m_source=base&agg_q=%40asset.new_value.query.indexes&agg_q_source=base&agg_t=count&audit__diff=unified&cols=log_usr.id%2Clog_action%2Clog_evt.name&fromUser=true&messageDisplay=expanded-md&refresh_mode=sliding&stream_sort=desc&top_n=10&top_o=top&viz=query_table&x_missing=true&live=true +[32]: /logs/explorer/archive_search/ diff --git a/content/en/logs/log_configuration/_index.md b/content/en/logs/log_configuration/_index.md index dd14a6411c3..ba08bf2d57b 100644 --- a/content/en/logs/log_configuration/_index.md +++ b/content/en/logs/log_configuration/_index.md @@ -32,7 +32,7 @@ Datadog Logging without Limits* decouples log ingestion and indexing. Choose whi - [Generate metrics from ingested logs][6] as cost-efficient way to summarize log data from an entire ingested stream. - Institute fine-grained control over your log management budget with [log indexes][7]. - Forward ingested logs to your own cloud-hosted storage bucket to keep as an [archive][8] for future troubleshooting or compliance audits. -- [Rehydrate an archive][9] to analyze or investigate log events that are older or excluded from indexing. +- [Search an archive][12] with Archive Search to query logs in real time directly from storage. Use **Search & Rehydration** mode when you need full platform access or longer retention. - Restrict [logs data access][10] with restriction queries. ## Log Explorer @@ -54,6 +54,6 @@ Once you've completed configuration, start investigating and troubleshooting log [6]: /logs/log_configuration/logs_to_metrics/ [7]: /logs/log_configuration/indexes [8]: /logs/log_configuration/archives/ -[9]: /logs/log_configuration/rehydrating [10]: /logs/guide/logs-rbac/ [11]: /logs/explorer/ +[12]: /logs/explorer/archive_search/ diff --git a/content/en/logs/log_configuration/archives.md b/content/en/logs/log_configuration/archives.md index e8b8fabcd96..95883bcdd0a 100644 --- a/content/en/logs/log_configuration/archives.md +++ b/content/en/logs/log_configuration/archives.md @@ -159,7 +159,7 @@ Only Datadog users with the [`logs_write_archive` permission][5] can create, mod ] } ``` - * The `GetObject` and `ListBucket` permissions allow for [rehydrating from archives][2]. + * The `GetObject` and `ListBucket` permissions allow for [searching archives][2]. * The `PutObject` permission is sufficient for uploading archives. * Ensure that the resource value under the `s3:PutObject` and `s3:GetObject` actions ends with `/*` because these permissions are applied to objects within the buckets. @@ -174,11 +174,11 @@ Only Datadog users with the [`logs_write_archive` permission][5] can create, mod [1]: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html -[2]: /logs/archives/rehydrating/ +[2]: /logs/explorer/archive_search/ {{% /tab %}} {{% tab "Azure Storage" %}} -1. Grant the Datadog app permission to write to and rehydrate from your storage account. +1. Grant the Datadog app permission to write to and read from your storage account. 2. Select your storage account from the [Storage Accounts page][1], go to {{< ui >}}Access Control (IAM){{< /ui >}}, and select {{< ui >}}Add{{< /ui >}} > {{< ui >}}Add Role Assignment{{< /ui >}}. 3. Input the Role called **Storage Blob Data Contributor**, select the Datadog app which you created to integrate with Azure, and save. @@ -298,7 +298,7 @@ To configure compression, select {{< ui >}}Compression Type{{< /ui >}} when crea You can either select a storage class for your archive or [set a lifecycle configuration on your S3 bucket][1] to automatically transition your log archives to optimal storage classes. -[Rehydration][2] only supports the following storage classes: +[Archive Search][2] only supports the following storage classes: * S3 Standard * S3 Standard-IA @@ -306,33 +306,33 @@ You can either select a storage class for your archive or [set a lifecycle confi * S3 Glacier Instant Retrieval * S3 Intelligent-Tiering, only if [the optional asynchronous archive access tiers][3] are both disabled. -If you wish to rehydrate from archives in another storage class, you must first move them to one of the supported storage classes above. +If your archive uses another storage class, you must first move it to one of the supported storage classes above. [1]: https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-set-lifecycle-configuration-intro.html -[2]: /logs/archives/rehydrating/ +[2]: /logs/explorer/archive_search/ [3]: https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ {{% /tab %}} {{% tab "Azure Storage" %}} -Archiving and [Rehydration][1] only supports the following access tiers: +Archiving and [Archive Search][1] only support the following access tiers: - Hot access tier - Cool access tier -If you wish to rehydrate from archives in another access tier, you must first move them to one of the supported tiers above. +If your archive uses another access tier, you must first move it to one of the supported tiers above. -[1]: /logs/archives/rehydrating/ +[1]: /logs/explorer/archive_search/ {{% /tab %}} {{% tab "Google Cloud Storage" %}} -Archiving and [Rehydration][1] supports the following access tiers: +Archiving and [Archive Search][1] support the following access tiers: - Standard - Nearline - Coldline - Archive -[1]: /logs/archives/rehydrating/ +[1]: /logs/explorer/archive_search/ {{% /tab %}} {{< /tabs >}} @@ -486,7 +486,7 @@ This directory structure simplifies the process of querying your historical log [13]: /account_management/rbac/permissions#logs_read_data [14]: /logs/explorer/live_tail/ [15]: /events/explorer/ -[16]: /logs/log_configuration/archive_search/?tab=amazons3 +[16]: /logs/explorer/archive_search/?tab=amazons3 [17]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html [18]: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html diff --git a/content/en/logs/log_configuration/rehydrating.md b/content/en/logs/log_configuration/rehydrating.md index 4f36c503a47..bc7e5d6bdcb 100644 --- a/content/en/logs/log_configuration/rehydrating.md +++ b/content/en/logs/log_configuration/rehydrating.md @@ -8,8 +8,16 @@ further_reading: - link: "logs/archives" tag: "Documentation" text: "Log Archives documentation" +- link: "/logs/explorer/archive_search/" + tag: "Documentation" + text: "Archive Search" --- +
+Archive Search is the recommended way to access archived logs.
+It streams results in real time directly from your archive without re-indexing, and charges only for the data scanned. When you need full platform access or longer retention, use Archive Search's Search & Rehydration mode. +
+ ## Overview Log Rehydration* enables you to capture log events from customer-owned storage-optimized archives back into Datadog's search-optimized [Log Explorer][1], so that you can use Datadog to analyze or investigate log events that are either old or were excluded from indexing. diff --git a/content/en/observability_pipelines/destinations/datadog_archives.md b/content/en/observability_pipelines/destinations/datadog_archives.md index e06be1d0574..c478dd99b2a 100644 --- a/content/en/observability_pipelines/destinations/datadog_archives.md +++ b/content/en/observability_pipelines/destinations/datadog_archives.md @@ -11,7 +11,7 @@ products: ## Overview -Use the Datadog Archives destination to send logs to Amazon S3 for [archiving][1] in Datadog-rehydratable format. You can [rehydrate][2] these logs later when you want to analyze and investigate them in Datadog. +Use the Datadog Archives destination to send logs to Amazon S3 for [archiving][1] in Datadog-rehydratable format. You can then query these logs with [Archive Search][16]. Use Archive Search's **Search & Rehydration** mode when you need to re-index results for full platform access. **Note**: Use the [Amazon S3][12] destination if you want to send your logs to Amazon S3 in JSON or Parquet format. @@ -195,4 +195,5 @@ A batch of events is flushed when one of these parameters is met. See [event bat [12]: /observability_pipelines/destinations/amazon_s3/ [13]: https://app.datadoghq.com/observability-pipelines [14]: /api/latest/observability-pipelines/ +[16]: /logs/explorer/archive_search/ [15]: https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline \ No newline at end of file