fix: resolve rollover issue with index and ism policy improvents #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Coverage Report
|
jboix
force-pushed
the
fix/rollover-and-sharding
branch
2 times, most recently
from
0ea9280 to
b04b9f3
Compare
MGaetan89
approved these changes
Nov 27, 2024
src/main/kotlin/ch/srgssr/pillarbox/monitoring/event/setup/IndexTemplateSetupTask.kt
Outdated
Show resolved
Hide resolved
src/main/kotlin/ch/srgssr/pillarbox/monitoring/event/setup/IndexTemplateSetupTask.kt
Outdated
Show resolved
Hide resolved
src/test/kotlin/ch/srgssr/pillarbox/monitoring/event/setup/OpenSearchSetupServiceTest.kt
Outdated
Show resolved
Hide resolved
jboix
force-pushed
the
fix/rollover-and-sharding
branch
from
b04b9f3 to
d4f3f29
Compare
MGaetan89
approved these changes
Nov 27, 2024
This commit addresses the issue where rollover was not resolving as intended, leading to a single
index accumulating all documents without purging.
BREAKING CHANGE: The index name has been changed from "actions" to "events", requiring manual
deletion of the old "actions" index.
Key changes:
1. The index configuration now uses 3 shards and 1 replica to distribute the load evenly across
the cluster's 3 data nodes. The index name has been changed to `events` to enforce this setting.
2. An index template with a `rollover_alias` has been created to ensure rollover operations work
correctly and don't hang.
3. Updated ISM policy:
- Rollover now occurs daily, allowing for more granular purging after 14 days.
- Introduced a warm state for indexes older than 7 days an action reduces the number of replicas
to 0.
jboix
force-pushed
the
fix/rollover-and-sharding
branch
from
d4f3f29 to
bcd08f3
Compare
|
🎉 This PR is included in version 3.0.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Addresses the issue where rollover was not resolving as intended, leading to a single index accumulating all documents without purging.
BREAKING CHANGE: The index name has been changed from
actionstoevents, requiring manual deletion of the oldactionsindex. A migration guide is available.Changes Made
eventsto enforce this setting.rollover_aliashas been created to ensure rollover operations work correctly and don't hang.Migration Guide
1. Stop the data-transfer service
2. Clean up OpenSearch cluster
Delete all indexes:
Delete ISM policies:
3. Update the data-transfer service to the latest version
4. Restart the service
Checklist