2021 10 18 (Monday) Deployment
This deployment mainly consists of the latest batch of work from Flexion. See the stories below.
Additionally, it commits the change to the r5.xlarge.elasticsearch cluster to the project for production, test, and migration. We are doing this after encountering an elevated error rate in the previous week, and observing healthy garbage collection since the change.
And to combat the memory issues that we've been observing, we are also turning on logging for Slow queries for Indexing and Searching. We have created a spike to review the logs, and we'll be conducting that Wednesday at 12:30pm.
We are performing this update after midnight to ensure that end users trying to beat a deadline do not miss their opportunity to do so. Additionally, we observe a low level of activity at this time. We will notify any Court Staff logged in to save their work and log out as the deployment completes.
- None
- 8880 Opinion Search: UI Fix
- 8793 Order Search - Automate performance testing to utilize variable data for a simulated load
- 8511 Automate Switch Colors Step in Prod
- 7643 Update or Add New Email: Notice of Change of Email Address - Party
- 7384 Opinion Search: AND keyword search
- 7370 Opinion Search: Locate document by exact keyword/phase
- 23:45 - Created the Pull Request
- 23:50 - Run environment-specific
$ npm run deploy:environment-specific prod
Plan: 4 to add, 1 to change, 0 to destroy.
aws_iam_role.iam_update_petitioner_cases_lambda_role: Creating...
aws_iam_role.reindex_status_role: Creating...
aws_iam_role_policy.iam_cognito_post_authentication_lambda_policy: Modifying... [id=iam_cognito_post_authentication_lambda_role_prod:iam_cognito_post_authentication_lambda_policy_prod]
aws_iam_role_policy.iam_cognito_post_authentication_lambda_policy: Modifications complete after 0s [id=iam_cognito_post_authentication_lambda_role_prod:iam_cognito_post_authentication_lambda_policy_prod]
aws_iam_role.iam_update_petitioner_cases_lambda_role: Creation complete after 1s [id=iam_update_petitioner_cases_lambda_role_prod]
aws_iam_role_policy.iam_update_petitioner_cases_lambda_policy: Creating...
aws_iam_role.reindex_status_role: Creation complete after 1s [id=reindex_status_role_prod]
aws_iam_role_policy.reindex_status_policy: Creating...
aws_iam_role_policy.iam_update_petitioner_cases_lambda_policy: Creation complete after 0s [id=iam_update_petitioner_cases_lambda_role_prod:iam_update_petitioner_cases_lambda_policy_prod]
aws_iam_role_policy.reindex_status_policy: Creation complete after 0s [id=reindex_status_role_prod:reindex_status_policy_prod]
Apply complete! Resources: 4 added, 1 changed, 0 destroyed.
-
00:08 - Adjust required tests for
prodto the updated list:- ci/circleci: e2e-cypress
- ci/circleci: e2e-pa11y
- ci/circleci: build-client-integration
- ci/circleci: e2e-cypress-public
- ci/circleci: e2e-pa11y-public
-
00:09 - Tests pass
-
00:10 - Merged the PR CircleCI Build
-
00:19 - Tests pass; deploy step starts
-
00:42 - Deploy fails (as predicted)
-
01:01 - Run the script
setup-s3-triggers-file.sh
./setup-s3-triggers-file.sh prod
copy: s3://dawson.ustaxcourt.gov.efcms.prod.us-east-1.lambdas/triggers_blue.js.zip to s3://dawson.ustaxcourt.gov.efcms.prod.us-east-1.lambdas/triggers_green.js.zip
- 01:02 - Redeploy from failed CircleCI Build
- 01:14 - Deploy step completes
- 01:21 - Smoketests pass! Observed that
USTC_ADMIN_USERis disabled. - 01:22 - Switch colors...
- 01:32 - Observed that the slow query logs exist
- 01:33 - Disabled
greenapi custom domains east & west
Everything looks good!
