NONEVM-3034: log pruning background worker #553

jadepark-dev · 2026-01-28T21:51:23Z

log/soft-deleted filter pruning
schema update - add core store migration NONEVM-3034: TON schema update chainlink#20972
add integrations tests

Copilot

Pull request overview

This PR implements a comprehensive log pruning system for the TON log poller, enabling automatic cleanup of old or excessive logs based on configurable retention policies.

Changes:

Added time-based and count-based log pruning mechanisms with configurable intervals and batch sizes
Extended database schema to support log_retention and max_logs_kept fields for filters, plus expires_at for logs
Implemented background pruning worker that runs periodically to clean up expired logs, excess logs, and logs from deleted filters

Reviewed changes

Copilot reviewed 20 out of 22 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
scripts/.core_version	Updated core version reference
pkg/logpoller/store/postgres/models.go	Added retention fields to filter and log models
pkg/logpoller/store/postgres/logs.go	Implemented DeleteExpiredLogs, DeleteExcessLogs, and DeleteLogsForDeletedFilters methods
pkg/logpoller/store/postgres/filters.go	Added validation for retention fields and DeleteEmptyFilters method
pkg/logpoller/store/memory/logs.go	Added no-op implementations for in-memory store
pkg/logpoller/store/memory/filters.go	Added no-op DeleteEmptyFilters for in-memory store
pkg/logpoller/service.go	Integrated background pruning worker with configurable startup delay
pkg/logpoller/pruning.go	New file implementing backgroundPruningRun method
pkg/logpoller/parser.go	Updated to populate ExpiresAt field and refactored filter handling
pkg/logpoller/models/models.go	Changed FilterIndex to map to Filter pointers and added retention fields
pkg/logpoller/metrics.go	Added pruning-specific metrics (logs_deleted, pruning_duration, pruning_errors)
pkg/logpoller/interfaces.go	Added method signatures for pruning operations
pkg/logpoller/filter.go	Updated buildFilterIndex to store Filter pointers instead of IDs
pkg/logpoller/config_test.go	Refactored tests and added pruning configuration validation tests
pkg/logpoller/config.go	Added pruning configuration fields with validation
pkg/ccip/chainaccessor/event.go	Set default CCIP log retention to 30 days
integration-tests/logpoller/testdata/create_ton_logpoller_tables.sql	Added retention columns and pruning index
integration-tests/logpoller/pg_pruning_test.go	New comprehensive pruning integration test
integration-tests/logpoller/pg_filters_test.go	Added tests for retention field validation
integration-tests/go.mod	Added test dependencies

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

patricios-space

LGTM

ogtownsend

small comments otherwise lgtm!

ogtownsend · 2026-02-02T20:49:26Z

pkg/logpoller/store/postgres/logs.go

+}
+
+// DeleteExcessLogs removes logs exceeding max_logs_kept for each filter.
+func (s *pgLogStore) DeleteExcessLogs(ctx context.Context, limit int64) (int64, error) {


This one might end up getting a little expensive/slow, but it's only running every ~10 minutes, so we can keep an eye on it. Probably overkill to create an index for it now

yeah, and for CCIP we don't have actual log count limit per filter

ogtownsend · 2026-02-02T20:50:17Z

integration-tests/go.mod

 // but pin chainlink-ton root module to avoid conflicts with indirect dependency via CLDF.
 replace github.com/smartcontractkit/chainlink-ton/deployment => ../deployment

+// TODO: remove on merge(TBD local package dependency management)


ogtownsend · 2026-02-02T20:56:16Z

pkg/logpoller/config.go

+	PruningInterval   *config.Duration // How often to run pruning (default 10 minutes)
+	PruningBatchSize  int64            // Max rows to delete per batch (default 1000)
+	PruningStartDelay *config.Duration // Delay before first pruning cycle (default 5 minutes)


nit: add new fields to full config test: TestNewDecodedTOMLConfig() in pkg/config/toml_test.go

ogtownsend · 2026-02-02T21:03:46Z

pkg/logpoller/store/postgres/logs.go

+		WHERE chain_id = :chain_id
+		  AND expires_at IS NOT NULL
+		  AND expires_at <= NOW()
+		LIMIT :limit


This could be a post-launch follow-up, but what if:

PruningBatchSize = 1000 , PruningInterval = 10 minutes

something goes down for an extended time

there are ~50k+ expired logs

each pruning cycle deletes only 1k logs, this would take ~8 hours to catch up, but meanwhile we'll be accumulating more and more logs

maybe a future optimization: this could return the number of logs deleted and if numLogsDeleted == limit, then loop immediately and issue another delete operation right away rather than waiting another 10 minutes 🤷

Good point, noted. also for short term we can try to increase the batch size.

krebernisak

Great work!

krebernisak · 2026-02-03T09:28:06Z

pkg/logpoller/metrics.go

+	promTonLpPruningErrors = promauto.NewCounterVec(prometheus.CounterOpts{
+		Name: "ton_logpoller_pruning_errors_total",
+		Help: "Total number of pruning errors",
+	}, []string{"chainID"})


Is node metadata (id, name) added automatically to these metrics?

Copilot

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jadepark-dev added 7 commits January 28, 2026 21:47

feat: log pruning worker

061ba2f

fix: remove error col

1c06c5a

Merge branch 'main' into jade/scheduled-log-pruning

52e1385

chore: add tests

fbb86c4

Merge branch 'main' into jade/scheduled-log-pruning

80c0453

chore: tidy

dccdfe4

Merge branch 'main' into jade/scheduled-log-pruning

077fa79

jadepark-dev marked this pull request as ready for review January 30, 2026 17:32

jadepark-dev requested a review from a team as a code owner January 30, 2026 17:32

jadepark-dev requested review from archseer, Copilot and ogtownsend January 30, 2026 17:32

Copilot AI reviewed Jan 30, 2026

View reviewed changes

chore: bump core

25e2b69

patricios-space previously approved these changes Feb 2, 2026

View reviewed changes

Merge branch 'main' into jade/scheduled-log-pruning

528eb98

ogtownsend previously approved these changes Feb 2, 2026

View reviewed changes

Merge branch 'main' into jade/scheduled-log-pruning

fc6570e

jadepark-dev dismissed stale reviews from ogtownsend and patricios-space via fc6570e February 3, 2026 00:26

jadepark-dev added 4 commits February 3, 2026 00:30

chore: add tests

449a960

fix: local replace for integration test

659716f

chore: lint

116c113

chore: lint

7d44f1e

jadepark-dev enabled auto-merge (squash) February 3, 2026 01:01

jadepark-dev requested a review from ogtownsend February 3, 2026 01:01

ogtownsend previously approved these changes Feb 3, 2026

View reviewed changes

chore: bump core_version

e6358c5

jadepark-dev dismissed ogtownsend’s stale review via e6358c5 February 3, 2026 08:20

krebernisak approved these changes Feb 3, 2026

View reviewed changes

jadepark-dev merged commit 43c83cc into main Feb 3, 2026
36 of 37 checks passed

jadepark-dev deleted the jade/scheduled-log-pruning branch February 3, 2026 09:30

krebernisak requested a review from Copilot February 3, 2026 09:30

Copilot AI reviewed Feb 3, 2026

View reviewed changes

NONEVM-3034: log pruning background worker #553

NONEVM-3034: log pruning background worker #553

Uh oh!

Conversation

jadepark-dev commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

patricios-space left a comment

Choose a reason for hiding this comment

Uh oh!

ogtownsend left a comment

Choose a reason for hiding this comment

Uh oh!

ogtownsend Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

jadepark-dev Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

ogtownsend Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ogtownsend Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ogtownsend Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jadepark-dev Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

krebernisak left a comment

Choose a reason for hiding this comment

Uh oh!

krebernisak Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jadepark-dev commented Jan 28, 2026 •

edited

Loading

ogtownsend Feb 2, 2026 •

edited

Loading