2021 11 20 (Saturday) Deployment

General Notes

This deployment mainly consists of the latest batch of work from Flexion. See the stories below.

We are performing this update after hours, expecting it to conclude between 1am and 2am as we observe low level of activity at this time. We will notify any Court Staff logged in to save their work and log out as the deployment completes.

Bugfixes

• https://github.com/flexion/ef-cms/issues/9123
• https://github.com/flexion/ef-cms/issues/9120

Feature Stories

• https://github.com/flexion/ef-cms/issues/9066
• https://github.com/flexion/ef-cms/issues/9068
• https://github.com/flexion/ef-cms/issues/9076
• https://github.com/flexion/ef-cms/issues/8990
• https://github.com/flexion/ef-cms/issues/9073

Observations

Timeline

• 21:06 - Created the Pull Request
• 21:11 - Run script to setup boolean values in prod deploy table

$ ./scripts/setup-external-order-search-flag.sh prod
$ ./scripts/rename-order-search-flag-to-internal.sh prod
$ ./scripts/setup-internal-opinion-search-flag.sh prod

• 21:13 - Ensure external-order-search-enabled flag is false (SOON!)

• 21:13 - Ensure ES and DynamoDB tables are ready for a Migration;

  • efcms-prod-beta tables deleted east and west
  • efcms-search-prod-beta deleted
  • set destination-table-version to beta
  • set migrate to true
  • confirmed source-table-version is alpha

• 21:18 - Tests pass

• 21:25 - Ran Docker to ECR script

$ ./docker-to-ecr.sh latest

• 21:39 - ... script running painfully slow. DSL Reports Speed Test failed; attempting a reboot.

• 22:35 - ... got the script to complete after tethering off of my phone. have a new modem on the way. 😩

• 21:26 - Merged the PR CircleCI Build

• 21:52 - e2e-cypress-smoketests-local fails, retrying from failed CircleCI Build

• 22:59 - Tests pass; deploy step starts

• 23:21 - Deploy step fails, this is due to manually recreated WebSocket mapping records without reimporting into Terraform.

  │ Error: error creating API Gateway v2 API mapping: ConflictException: ApiMapping key already exists for this domain name
  │   with module.ef-cms_apis.module.api-west-blue.aws_apigatewayv2_api_mapping.websocket_mapping,
  │   with module.ef-cms_apis.module.api-east-blue.aws_apigatewayv2_api_mapping.websocket_mapping,    
  

• 23:29 - Deleted beta tables and clusters as well as blue websockets mapping records in preparation for another attempt.

• 00:11 - Deploy step completes

• 00:18 - Migration starts. 🤞

• 01:11 - Migration completes successfully

• 04:15 - Reindexing appears to be complete; even though the automated script is reporting one missing efcms-case-deadline record. I ran the script to calculate the delta between the two clusters, and I identified pk:case-deadline|d8547822-b38a-4b95-ad5a-a656c939bb10,sk:case-deadline|d8547822-b38a-4b95-ad5a-a656c939bb10 as the missing record. When looking in the source table, the record did not exist. This appears to be another example of this bug that we cannot reliably replicate where records aren't getting reliably removed from ES after being deleted.

• 04:23 - Readonly Smoketests pass

• 04:24 - Switching colors