Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7074,11 +7074,6 @@ menu:
url: logs/log_configuration/rehydrating
parent: log_configuration
weight: 210
- name: Archive Search
identifier: log_configuration_archive_search
url: logs/log_configuration/archive_search
parent: log_configuration
weight: 211
- name: Forwarding
identifier: log_configuration_forwarding
url: logs/log_configuration/forwarding_custom_destinations/
Expand Down Expand Up @@ -7174,6 +7169,11 @@ menu:
url: logs/explorer/saved_views/
parent: log_explorer
weight: 514
- name: Archive Search
identifier: log_explorer_archive_search
url: logs/explorer/archive_search/
parent: log_explorer
weight: 515
- name: Error Tracking
url: logs/error_tracking/
parent: log_management
Expand Down
7 changes: 7 additions & 0 deletions content/en/glossary/terms/archive_search.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
id: archive_search
title: Archive Search
core_product:
- log management
---
Archive Search lets you query logs directly from long-term cloud storage archives without re-indexing them. Results stream in real time and you are charged only for the data scanned. Use **Search & Rehydration** mode when you need full platform access or longer retention. See [Archive Search](/logs/explorer/archive_search/).
6 changes: 5 additions & 1 deletion content/en/glossary/terms/rehydration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,8 @@ title: Rehydration
core_product:
- log management
---
Rehydration is when archived logs are recalled back into Datadog.
<div class="alert alert-info">
<a href="/logs/explorer/archive_search/">Archive Search</a> is the recommended way to query archived logs. Use its <strong>Search &amp; Rehydration</strong> mode when you need full platform access or longer retention.
</div>

Rehydration is when archived logs are re-indexed back into Datadog for full platform access.
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
title: Archive Search
description: Instantly search and analyze logs from long-term archives without re-indexing.
aliases:
- /logs/log_configuration/archive_search/
further_reading:
- link: "/logs/explorer/"
tag: "Documentation"
Expand Down
10 changes: 5 additions & 5 deletions content/en/logs/guide/best-practices-for-log-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,11 @@ If you want to retain logs for an extended time while maintaining querying speed

### Set up multiple archives for long-term storage

If you want to store your logs for longer periods of time, set up [Log Archives][2] to send your logs to a storage-optimized system, such as Amazon S3, Azure Storage, or Google Cloud Storage. When you want to use Datadog to analyze those logs, use [Log Rehydration][3]™ to capture those logs back in Datadog. With multiple archives, you can both segment logs for compliance reasons and keep rehydration costs under control.
If you want to store your logs for longer periods of time, set up [Log Archives][2] to send your logs to a storage-optimized system, such as Amazon S3, Azure Storage, or Google Cloud Storage. Use [Archive Search][32] to query these logs directly from storage in real time. When you need full platform access or longer retention, use Archive Search's **Search & Rehydration** mode. With multiple archives, you can segment logs for compliance reasons and keep costs under control.

#### Set up max scan size to manage expensive rehydrations
#### Set up max scan size to manage costs

Set a limit on the volume of logs that can be rehydrated at one time. When setting up an archive, you can define the maximum volume of log data that can be scanned for Rehydration. See [Define maximum scan size][4] for more information.
Set a limit on the volume of logs that can be scanned at one time. When setting up an archive, you can define the maximum volume of log data that can be scanned per Archive Search query or rehydration. See [Define maximum scan size][4] for more information.

### Set up RBAC for custom roles

Expand Down Expand Up @@ -225,7 +225,6 @@ If you want to see user activities, such as who changed the retention of an inde

[1]: https://app.datadoghq.com/logs/pipelines/indexes
[2]: /logs/log_configuration/archives/
[3]: /logs/log_configuration/rehydrating/
[4]: /logs/log_configuration/archives/?tab=awss3#define-maximum-scan-size
[5]: /account_management/rbac/?tab=datadogapplication#datadog-default-roles
[6]: https://app.datadoghq.com/
Expand Down Expand Up @@ -253,4 +252,5 @@ If you want to see user activities, such as who changed the retention of an inde
[28]: /monitors/configuration/?tab=thresholdalert#evaluation-window
[29]: /observability_pipelines/
[30]: /logs/log_configuration/flex_logs/
[31]: https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&group_by=%40asset.new_value.query.indexes](https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&agg_m=count&agg_m_source=base&agg_q=%40asset.new_value.query.indexes&agg_q_source=base&agg_t=count&audit__diff=unified&cols=log_usr.id%2Clog_action%2Clog_evt.name&fromUser=true&messageDisplay=expanded-md&refresh_mode=sliding&stream_sort=desc&top_n=10&top_o=top&viz=query_table&x_missing=true&from_ts=1768733389060&to_ts=1771325389060&live=true
[31]: https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&agg_m=count&agg_m_source=base&agg_q=%40asset.new_value.query.indexes&agg_q_source=base&agg_t=count&audit__diff=unified&cols=log_usr.id%2Clog_action%2Clog_evt.name&fromUser=true&messageDisplay=expanded-md&refresh_mode=sliding&stream_sort=desc&top_n=10&top_o=top&viz=query_table&x_missing=true&live=true
[32]: /logs/explorer/archive_search/
4 changes: 2 additions & 2 deletions content/en/logs/log_configuration/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Datadog Logging without Limits* decouples log ingestion and indexing. Choose whi
- [Generate metrics from ingested logs][6] as cost-efficient way to summarize log data from an entire ingested stream.
- Institute fine-grained control over your log management budget with [log indexes][7].
- Forward ingested logs to your own cloud-hosted storage bucket to keep as an [archive][8] for future troubleshooting or compliance audits.
- [Rehydrate an archive][9] to analyze or investigate log events that are older or excluded from indexing.
- [Search an archive][12] with Archive Search to query logs in real time directly from storage. Use **Search & Rehydration** mode when you need full platform access or longer retention.
- Restrict [logs data access][10] with restriction queries.

## Log Explorer
Expand All @@ -54,6 +54,6 @@ Once you've completed configuration, start investigating and troubleshooting log
[6]: /logs/log_configuration/logs_to_metrics/
[7]: /logs/log_configuration/indexes
[8]: /logs/log_configuration/archives/
[9]: /logs/log_configuration/rehydrating
[10]: /logs/guide/logs-rbac/
[11]: /logs/explorer/
[12]: /logs/explorer/archive_search/
24 changes: 12 additions & 12 deletions content/en/logs/log_configuration/archives.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ Only Datadog users with the [`logs_write_archive` permission][5] can create, mod
]
}
```
* The `GetObject` and `ListBucket` permissions allow for [rehydrating from archives][2].
* The `GetObject` and `ListBucket` permissions allow for [searching archives][2].
* The `PutObject` permission is sufficient for uploading archives.
* Ensure that the resource value under the `s3:PutObject` and `s3:GetObject` actions ends with `/*` because these permissions are applied to objects within the buckets.

Expand All @@ -174,11 +174,11 @@ Only Datadog users with the [`logs_write_archive` permission][5] can create, mod


[1]: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html
[2]: /logs/archives/rehydrating/
[2]: /logs/explorer/archive_search/
{{% /tab %}}
{{% tab "Azure Storage" %}}

1. Grant the Datadog app permission to write to and rehydrate from your storage account.
1. Grant the Datadog app permission to write to and read from your storage account.
2. Select your storage account from the [Storage Accounts page][1], go to {{< ui >}}Access Control (IAM){{< /ui >}}, and select {{< ui >}}Add{{< /ui >}} > {{< ui >}}Add Role Assignment{{< /ui >}}.
3. Input the Role called **Storage Blob Data Contributor**, select the Datadog app which you created to integrate with Azure, and save.

Expand Down Expand Up @@ -298,41 +298,41 @@ To configure compression, select {{< ui >}}Compression Type{{< /ui >}} when crea

You can either select a storage class for your archive or [set a lifecycle configuration on your S3 bucket][1] to automatically transition your log archives to optimal storage classes.

[Rehydration][2] only supports the following storage classes:
[Archive Search][2] only supports the following storage classes:

* S3 Standard
* S3 Standard-IA
* S3 One Zone-IA
* S3 Glacier Instant Retrieval
* S3 Intelligent-Tiering, only if [the optional asynchronous archive access tiers][3] are both disabled.

If you wish to rehydrate from archives in another storage class, you must first move them to one of the supported storage classes above.
If your archive uses another storage class, you must first move it to one of the supported storage classes above.

[1]: https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-set-lifecycle-configuration-intro.html
[2]: /logs/archives/rehydrating/
[2]: /logs/explorer/archive_search/
[3]: https://aws.amazon.com/s3/storage-classes/intelligent-tiering/
{{% /tab %}}
{{% tab "Azure Storage" %}}

Archiving and [Rehydration][1] only supports the following access tiers:
Archiving and [Archive Search][1] only support the following access tiers:

- Hot access tier
- Cool access tier

If you wish to rehydrate from archives in another access tier, you must first move them to one of the supported tiers above.
If your archive uses another access tier, you must first move it to one of the supported tiers above.

[1]: /logs/archives/rehydrating/
[1]: /logs/explorer/archive_search/
{{% /tab %}}
{{% tab "Google Cloud Storage" %}}

Archiving and [Rehydration][1] supports the following access tiers:
Archiving and [Archive Search][1] support the following access tiers:

- Standard
- Nearline
- Coldline
- Archive

[1]: /logs/archives/rehydrating/
[1]: /logs/explorer/archive_search/
{{% /tab %}}

{{< /tabs >}}
Expand Down Expand Up @@ -486,7 +486,7 @@ This directory structure simplifies the process of querying your historical log
[13]: /account_management/rbac/permissions#logs_read_data
[14]: /logs/explorer/live_tail/
[15]: /events/explorer/
[16]: /logs/log_configuration/archive_search/?tab=amazons3
[16]: /logs/explorer/archive_search/?tab=amazons3
[17]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html
[18]: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html

Expand Down
8 changes: 8 additions & 0 deletions content/en/logs/log_configuration/rehydrating.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,16 @@ further_reading:
- link: "logs/archives"
tag: "Documentation"
text: "Log Archives documentation"
- link: "/logs/explorer/archive_search/"
tag: "Documentation"
text: "Archive Search"
---

<div class="alert alert-info">
<strong><a href="/logs/explorer/archive_search/">Archive Search</a> is the recommended way to access archived logs.</strong><br>
It streams results in real time directly from your archive without re-indexing, and charges only for the data scanned. When you need full platform access or longer retention, use Archive Search's <strong>Search &amp; Rehydration</strong> mode.
</div>

## Overview

Log Rehydration* enables you to capture log events from customer-owned storage-optimized archives back into Datadog's search-optimized [Log Explorer][1], so that you can use Datadog to analyze or investigate log events that are either old or were excluded from indexing.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ products:

## Overview

Use the Datadog Archives destination to send logs to Amazon S3 for [archiving][1] in Datadog-rehydratable format. You can [rehydrate][2] these logs later when you want to analyze and investigate them in Datadog.
Use the Datadog Archives destination to send logs to Amazon S3 for [archiving][1] in Datadog-rehydratable format. You can then query these logs with [Archive Search][16]. Use Archive Search's **Search & Rehydration** mode when you need to re-index results for full platform access.

**Note**: Use the [Amazon S3][12] destination if you want to send your logs to Amazon S3 in JSON or Parquet format.

Expand Down Expand Up @@ -195,4 +195,5 @@ A batch of events is flushed when one of these parameters is met. See [event bat
[12]: /observability_pipelines/destinations/amazon_s3/
[13]: https://app.datadoghq.com/observability-pipelines
[14]: /api/latest/observability-pipelines/
[16]: /logs/explorer/archive_search/
[15]: https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline
Loading