Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External data staleness monitoring #177

Open
konklone opened this issue Apr 9, 2015 · 8 comments
Open

External data staleness monitoring #177

konklone opened this issue Apr 9, 2015 · 8 comments

Comments

@konklone
Copy link
Contributor

konklone commented Apr 9, 2015

We had our first server bug that delayed data updates for a time. We should have something that watches the real-time data feeds and alerts us if it sees stale data.

I suggest performing a HEAD request to https://analytics.usa.gov/data/live/realtime.json and looking for whether its Last-Modified header has become more than 5 or 10 minutes old.

@ArcTanSusan
Copy link
Contributor

Do you have a preference for what tool to use in order to notify folks if data ever is more than 5-10 mins old?

@konklone
Copy link
Contributor Author

konklone commented Dec 8, 2015

We haven't dived into it too much. Got any ideas?

@ArcTanSusan
Copy link
Contributor

Since the dashboard is entirely front-end w/o a backend, we can try a number of email-sending JS tools such as https://github.com/eleith/emailjs.

@konklone
Copy link
Contributor Author

konklone commented Dec 8, 2015

That's only appropriate for back-end use, as if it's used in the front-end it would expose the authentication credentials for the SMTP server.

@ArcTanSusan
Copy link
Contributor

Hmm, this seems to be an ops task. Maybe a cron job that calls a python script to make requests to that url and check for the timestamp headers?

@konklone
Copy link
Contributor Author

konklone commented Dec 8, 2015

Yep, I think it could be relatively self-contained. Ideally, it would be a little general-purpose open source monitoring app that we can host on a Heroku-like system, and which can take enough parameters to do this without much/any custom coding. I'm not sure if something like that exists, but it would be the ideal.

@ArcTanSusan
Copy link
Contributor

@konklone: Thoughts on the nodeJS project https://github.com/iloire/watchmen?

@levinmr
Copy link
Contributor

levinmr commented Aug 19, 2024

@smarina04 this is a potential feature. We currently have no way to indicate if data stopped updating for the site. It could be worth alerting somehow if data reports used for the charts has a timestamp older than expected

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants