Skip to content

2021 08 14 (Saturday) Deployment

Mike Marcotte edited this page Aug 16, 2021 · 4 revisions

General Notes

This is fairly large deployment including two batches of work completed by the vendor. It includes some data migrations to fix some data and consolidate some unnecessary records. It also notably includes some more automation for the migration process. We will be paying close attention to how these procedures perform on their maiden voyage.

Due to the high amount of code changes, we will be performing this deployment after hours and on the weekend. It has been tested in the dev, mig, and test environments.

Timeline

  • 22:00 -  made the Pull Request

  • 22:10 - ran ./docker-to-ecr.sh latest

  • 22:15 - ran ./docker-to-ecr.sh prod

    NOTE: There exists some confusion about what the correct procedure is for this, and I have received conflicting guidance. Created this ticket to track that and help bring clarity to me and the team about its usage.

  • 22:18 - ran ./clamav-docker-to-ecr.sh prod

  • 22:24 - ran account specific

    NOTE: had to add -upgrade flag to download the latest provider

    NOTE: Using 8 nodes in prod for the Kibana cluster until make snapshots and are comfortable running without all of the logs to search upon.

  • 22:26 - created missing log groups create-missing-log-groups.sh

  • 22:27 - reran account specific ✅ deploy:account-specific

  • 22:28 - ran environment-specific ✅ deploy:environment-specific

  • 22:30 - merged PR ✅

  • 22:45 - tests pass ✅

  • 22:48 - deploy fails; 👎 error: expected the efcms-****-beta table to have been deleted from us-east-1 before running migration

    NOTE: I believe we have an improvement to this where if the table exists (and is empty) to not fail here and instead just use the empty DDB table.

  • 22:50 - deleted tables

  • 23:00 - restart from failed

  • 23:05 - observed setup of migration worked! ✅ 😎

  • 23:45 - Migration kicked off ✅

  • 00:40 - Severe weather alert for flooding 😨 ⛈️

  • 00:46 - migration finished; ✅ indexing begins 🕐

  • 03:00 - clusters quieted, testing looks good, switched colors

    Comparison between ES Clusters (Beta was destination; Alpha was source)

    Discrepancy with work items is due to the migration script to consolidate those records Discrepancy with the docket entry, user, and user-case is due to activity after switch colors

          indexName      countAlpha countBeta diff 
        'efcms-case'        1990810    1990810    0 
    'efcms-case-deadline'   16318      16318    0 
    'efcms-docket-entry'    18253964  18253968    4 
        'efcms-message'      291752    291752    0 
        'efcms-user'        2001222    2001226    4 
      'efcms-user-case'    1979220    1979228    8 
      'efcms-work-item'    775992    740948  35044

    Total Difference: 35028 (25274250/25309278) 99.86%

Clone this wiki locally