CA Game Day Exercise Part III

[scleary1cs]

DevEx/OpEx

To ensure we're ready to support unexpected issues in production, as a team we need to run through some sample scenarios.

Potential Exercises:

LIMS / EHR offline
TI or RS offline
Inter-service auth failure
Invalid message contents
Corrupt zip file via SFTP
Changed SFTP encryption key

Completion Criteria

• We've completed at least one exercise:
  • Description of surprise failure here.

Tasks (these need to be verified)

• Determine the scenario
• Determine when this exercise will take place
  • Get support from the RS team
• Document an intro to the session
• Submit sample data for scenarios
• Staging configuration change to support one of the scenarios
• Conduct a blameless post-mortem
• Conduct a retro for the whole exercise

Additional Context

Team has a number of new members since the previous CA Game Day Exercises
#1132.

Additionally, we can verify all of the updates made to our incident management process.

[JohnNKing]

The exercise was initiated today and resolved in short order.

[JohnNKing]

Based on action items from https://www.notion.so/flexion-cdc-ti/Post-mortem-Gameday-3-1365955365f58010b556e4000c23daa6:

• How does ReportStream classify production support issues, incidents, etc?
  • I've created Understand ReportStream Incident Classification #1577 (@brick-green, @GilmoreA6 )
• Shorten/summarize process doc to be easier to consume in the heat of battle
  • I suspect this aligns with the intent of Reorganize the documentation to maximize ease of use #1352 (@csweet92, @jbiskie, @somesylvie)