skywalking ttl

[xxlaila]

Skywalking version: 10.0.1
The Elasticsearch version is: 7.14

The TTL expiration time for skywalking has been set to 7 days,

recordDataTTL: ${SW_CORE_RECORD_DATA_TTL:7} # Unit is day
metricsDataTTL: ${SW_CORE_METRICS_DATA_TTL:7} # Unit is day

1. Is there a mechanism for skywalking to clear the historical data of Elasticsearch that has not been cleared?

• In the configuration file, if enableDataKeeperExecutor is set to true, it will be automatically cleared. DataKeeperExecutePeriod: 5, executed every 5 minutes

2. I manually deleted historical data in Elasticsearch, but it was automatically created. What's going on? How can I troubleshoot this issue to avoid it

[wu-sheng]

Is there a mechanism for skywalking to clear the historical data of Elasticsearch that has not been cleared?

Elasticsearch never removed the expired data. OAP does. I am not sure what you are asking here.

In the configuration file, if enableDataKeeperExecutor is set to true, it will be automatically cleared. DataKeeperExecutePeriod: 5, executed every 5 minutes

You don't need to worry about that. Elasticsearch storage provider has its own implementation, which will execute only once per day.

I manually deleted historical data in Elasticsearch, but it was automatically created. What's going on? How can I troubleshoot this issue to avoid it

I guess you removed all indcies? At least the latest day index should be kept. Otherwise, query would trigger error.

---

If you want to read codes, go for HistoryDeleteEsDAO.

[xxlaila]

Sorry, maybe I didn't describe it clearly，
The OAP configuration file sets the following parameters
enableDataKeeperExecutor: ${SW_CORE_ENABLE_DATA_KEEPER_EXECUTOR:true} # Turn it off then automatically metrics data delete will be close.
dataKeeperExecutePeriod: ${SW_CORE_DATA_KEEPER_EXECUTE_PERIOD:5} # H
recordDataTTL: ${SW_CORE_RECORD_DATA_TTL:7} # Unit is day
metricsDataTTL: ${SW_CORE_METRICS_DATA_TTL:7} # Unit is day
But in Elasticsearch, the data from 7 days ago is still retained, and historical data has not been deleted. My Elasticsearch does not have a separate deletion strategy configured

[wu-sheng]

That doesn't make sense to me. As you mentioned indices are recreated after manual deletion. Usually, your data timestamp is not correct. The data timestamp drives the indices creation.

[xxlaila]

OAP has set the data expiration time, but historical data has not been deleted. Elasticsearch still has historical data

[wu-sheng]

I manually deleted historical data in Elasticsearch, but it was automatically created. What's going on? How can I troubleshoot this issue to avoid it

This says otherwise. Your target app time could be wrong.