-
Notifications
You must be signed in to change notification settings - Fork 85
datastore: gc enrichments #1528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
6be4a42
to
cdaae49
Compare
datastore/postgres/migrations/matcher/15-uo_enrich-fkey-index.sql
Outdated
Show resolved
Hide resolved
@crozzy thanks for the approval. I want to do one more test prior to merge: Deploy an older database with >25 million rows in the enrichment table, hit Indexer/Matcher with a bunch of scan requests, and then update to a version with the changes from this PR so I can see how long it takes to delete everything while under load. If that goes well, I'll merge |
@crozzy I ran a test where I installed StackRox Scanner V4 DB with CPU limit of 2 and memory limit of 4GiB. I then gave it almost 2000 images to scan (15 at a time), and I found that the deletion took 1 hour and 24 minutes. That is a very long time, though no failures was nice, and I did put a lot of load on the DB. This also made the table go from
to
I also realized
I believe this may also help the deletion query, as one of the conditions looks for matching |
Signed-off-by: RTann <[email protected]> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
I have StackRox installed using Scanner V4, which is based on Claircore. Here's the
EXPLAIN ANALYZE
for the deletion queries:Tested on a local installation with nothing else running, and it took 6 - 8 minutes (sorry I forget). I had someone run this in an active cluster, and it took 7 minutes. All of these were on SSD. I ran it again on an inactive installation (ie one with no other activity happening) and it took 50 minutes on HDD.