Limit stats history #181
Limit stats history #181
Conversation
We only really need data for the past 3 months. We need to keep the audit_resource and resource_compliance tables manageable.
| elapsed_time = time() - start_time; | ||
|
|
||
| if elapsed_time > execution_limit: | ||
| break; |
There was a problem hiding this comment.
should we do something special if we're at risk of the lambda timing out and the expired audits haven't all been deleted? (write an error somewhere / retrigger the lambda) rather than just breaking the loop and exiting - won't that mean we could build up a backlog of expired audits or is that not likely
There was a problem hiding this comment.
Yeah - It definitely can't do them all in one go because of the execution time limit on the lambda.
I'd forgotten to change the frequency. I've just pushed a commit to run this every 15 minutes so it should slowly catch up with itself and then it should be able to keep on top of the deletions each day. The earliest ones there should only be a few audits each day because there weren't many subscribed accounts. I guess we might have to run it more frequently if we keep growing the number of accounts. This should handle <190 accounts maybe.
Hopefully we'll replace CSW with the Config Rules service before that becomes a problem.
There was a problem hiding this comment.
I guess I could do something else to notify us if the oldest audit is >365 days ago which should tell us if it's not keeping up.
There was a problem hiding this comment.
I'll have a think about how to do that and raise it as a separate issue maybe
If each run deletes maybe 2 audits and there's an audit for every subscribed account every day that gives us a maximum of 192 accounts before we need to increase the timeout or the frequency.
Create a scheduled lambda to delete expired audits (> 1 year old)
Change stats queries to generate static stats for past 3 months.
and some version upgrades in the package file