Incident report for hub 403 incident on Sep 27 2017 #12
Conversation
|
|
||
| 1. Local development environment being slightly different from production | ||
| environment (single node vs multi node) | ||
| 2. Testing in staging not thorough enough |
There was a problem hiding this comment.
I would add one more thing, which is: "changing lots of things at the same time." It sounds like a reason this was hard to debug is that we have a few different moving pieces that we're juggling (e.g., jupyterhub will be upgraded soon and that'll "just fix" some problems) that caused us to miss this. Not much we can do about that short-term as a lot of this code is evolving rapidly, but good to keep in mind to try keeping our # of changed variables at a minimum on each iteration.
|
|
||
| ## Local development | ||
|
|
||
| 1. Consider figuring out an easy way for people to run a development environment |
There was a problem hiding this comment.
This seems like an important point to me...any other possibilities than using another cloud instance?
There was a problem hiding this comment.
I amended this PR with a link to an issue I opened about this with a possible solution: jupyterhub/binderhub#137
|
Merging this since I'm about to perform some other git surgery. Feel free to continue to leave comments here! |
|
Hooray for incident reports! |
|
An external test battery to run against an existing deployment would also be useful. We could make sure that runs against staging before deploying to prod. |
No description provided.