ci: smoke test automation #13162

sobolk · 2023-08-28T20:25:36Z

Description of changes

This PR automates execution of Smoke Tests

Adds re-usable smoke test workflow
Adds a workflow that can be triggered manually by operator
Adds canary test that runs on schedule and runs smoke tests against latest CLI
Adds lightweight smoke tests for Windows that use shell and headless commands instead of e2e framework due to e2e framework not working well on small Windows machines.

Actions used in workflow that are not defined in this repository:

aws-actions/configure-aws-credentials@04b98b3f9e85f563fb061be8751a0352327246b0 # v3.0.1 - to obtain test account access
ruby/setup-ruby@250fcd6a742febb1123a77a841497ccaa8b9e939 - required for Xcode test
actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce #v3.1.2 - uploads test report
actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3

Out of scope
The goal of this PR is to :

establish automation (GH actions)
cover Xcode IOS use case in addition to existing smoke test

Remaining coverage (proper Windows tests and Amplify App) will be pursued in subsequent PRs.

Issue #, if available

Description of how you validated changes

Running workflow from push to smoke-tests-prototype branch.
See https://github.com/aws-amplify/amplify-cli/actions/workflows/smoke-tests-manual.yml

Other triggers like schedule and manual dispatch will be tested after merge (they require workflow to be present in dev branch).

E2E test run as well (since I touched these)

Checklist

PR description included
yarn test passes
Tests are changed or added
Relevant documentation is changed or added (and PR referenced)
New AWS SDK calls or CloudFormation actions have been added to relevant test and service IAM policies
Pull request labels are added

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sobolk · 2023-08-29T17:03:53Z

packages/amplify-e2e-tests/resources/example-ios-app/.gitignore

@@ -0,0 +1,22 @@
+
+
+#amplify-do-not-edit-begin


Borrowed from https://github.com/aws-amplify/amplify-swift/blob/d4f0ca6014758cc66f9df2dd76182af815bb6d15/.github/workflows/canary.yml

sobolk · 2023-08-29T17:04:47Z

packages/amplify-e2e-tests/src/__tests__/smoke-tests/smoketest.test.ts

+describe('Smoke Tests', () => {
+  const createCommands = (amplify: Amplify, directory: string): Command[] => [
    {
-      description: `Install @aws-amplify/cli@${cliVersion}`,
-      run: () => NPM.install(`@aws-amplify/cli@${cliVersion}`, true),
+      description: 'Amplify CLI version',
+      run: () => amplify.version(),
    },


I moved CLI installation to Actions logic, so that we can reuse same smoke test with many installation strategies.

edwardfoyle · 2023-08-29T17:09:28Z

.github/actions/run-smoke-tests-windows/windows-smoke-tests.sh

+
+$AMPLIFY_PATH status
+
+ADD_AUTH_REQUEST=$(cat ../.github/actions/run-smoke-tests-windows/add_auth_request.json)


what path does this script expect to be run from? Could we pass in a base path as an arg to avoid confusion if we change the directory structure of these actions / scripts?

edwardfoyle · 2023-08-29T17:10:52Z

.github/actions/run-smoke-tests/action.yml

+
+    - name: Install Dependencies
+      shell: bash
+      run: yarn install


should this include --frozen-lockfile?

bcd0740fa72c:amplify-cli-nofork sobkamil$ yarn install --frozen-lockfile ➤ YN0050: The --frozen-lockfile option is deprecated; use --immutable and/or --immutable-cache instead

I'll add immutable.

edwardfoyle · 2023-08-29T17:12:31Z

.github/actions/run-smoke-tests/action.yml

+
+    - name: Upload Report
+      uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce #v3.1.2
+      if: always()


is this not the default?

No. By default step is skipped if previous step fails. This makes sure that report is uploaded regardless of test success.

see https://stackoverflow.com/questions/58858429/how-to-run-a-github-actions-step-even-if-the-previous-step-fails-while-still-f

edwardfoyle · 2023-08-29T17:13:03Z

.github/actions/run-smoke-tests/action.yml

+      env:
+        CLI_REGION: ${{ inputs.region }}
+        CI: true
+        CIRCLECI: true


why is this necessary?

This is needed to trigger test profile configuration

amplify-cli/packages/amplify-e2e-tests/src/configure_tests.ts

Line 4 in ce0a0ff

if (isCI()) {

.

edwardfoyle · 2023-08-29T17:23:09Z

.github/workflows/smoke-tests.yml

@@ -0,0 +1,53 @@
+name: Smoke Tests


should this be an action rather than a workflow? Seems like it is being dispatched from the canary and manual workflow

The reason this is "reusable workflow" (see https://docs.github.com/en/actions/using-workflows/reusing-workflows )
Is the matrix element that runs the same steps on cartesian product of versions and agents.
That's not available in actions.

I created "canary" and "manual" as "lightweight entry points"
Think about smoke-tests.yml as base class and canary/manual as lightweight implementation of base that encapsulate invocation and parameters.
In other words smoke-tests.yml contains workflow logic. canary/manual are just different entry points (like cli throw over programatic interface).

Another reason to have canary and manual entry points is that they'll appear as separate top level workflows in GH ui so that person who wants to look at canaries vs person who wants to smoke test RC won't have to reason about modality of parameters.

And yet other reason is that creating single workflow definition with multiple "triggers" would require workarounds to have matrix populated correctly in every case. (Depending on trigger we'd need different logic to populate matrix which is hard problem in declarative yaml).

edwardfoyle

This is going to be a huge win!

Amplifiyer · 2023-08-30T15:12:58Z

.github/workflows/smoke-tests-manual.yml

+  push:
+    branches: [smoke-tests-prototype]


Is this still needed?

Amplifiyer · 2023-08-30T15:14:01Z

.github/workflows/smoke-tests-canaries.yml

+    uses: ./.github/workflows/smoke-tests.yml
+    secrets: inherit
+    with:
+      versions: '["latest"]'


Perhaps we should add RC as well to catch any errors before going to production?

We could. I was thinking that RC will go through "oncall will click a button to trigger smoke tests", but having an active monitoring here won't hurt I guess.

Amplifiyer · 2023-08-30T15:14:42Z

.github/workflows/smoke-tests-canaries.yml

@@ -0,0 +1,17 @@
+name: Smoke Tests - Canaries


What's the notification mechanism for this canary? I.e. how would the oncall find out if a run has failed?

This is a good question.

Options to pursue:

(lightweight) Have a slack bot that listens to this workflow, see https://github.blog/changelog/2022-12-06-github-actions-workflow-notifications-in-slack-and-microsoft-teams/ .

(more involved) Add an action that publishes metric to CloudWatch like our friends did here https://github.com/aws-amplify/amplify-ui/blob/a787311029e6c0c179b552f5abdad1dbed73436a/.github/actions/log-metric/action.yml#L43 .

I'd try that Slack integration first after merging this and explore cloudwatch if this is not working.

Amplifiyer · 2023-08-30T15:50:04Z

.github/workflows/smoke-tests-manual.yml

+  call-smoke-tests:
+    if: github.event_name != 'workflow_dispatch'
+    uses: ./.github/workflows/smoke-tests.yml
+    secrets: inherit


I'm guessing this can be cleaned up as well?

yeah, good catch

Amplifiyer

Great win!

sobolk added 30 commits August 25, 2023 08:21

chore: add very basic smoke test

cd4457b

chore: upload artifacts

a4eba6c

chore: debug logs

ceb1f15

chore: debug logs

f0b847e

chore: undo logs

4d9b826

chore: install amplify

6e604de

chore: does this work

1d21586

chore: nope

bfb018d

chore: creds

b2ffb23

chore: creds

c5783bb

chore: does this work

56b0dc6

chore: does this work

5b92742

chore: don't run twice

919a05a

chore: refactor that

0e07521

chore: refactor that

4ce4efb

chore: refactor that

fcddba9

chore: refactor that

137102f

try this

4ca86f1

try this

8435269

try this

f348811

try this

8257209

try this

f7573ee

try this

2877bbb

try this

d072aff

try this

3c63b83

try this

0eccdae

try this

3cf1d96

try this

4b838c0

try this

06461ac

Undo that.

a8dd6ea

sobolk added 5 commits August 28, 2023 14:17

chore: lint

4fa584b

chore: tests

4643eee

chore: pin actions versions

ef75a4f

Merge branch 'dev' into smoke-tests-prototype

fcfc627

chore: undo debug outputs

893b3da

sobolk marked this pull request as ready for review August 29, 2023 16:22

sobolk requested review from a team as code owners August 29, 2023 16:22

sobolk commented Aug 29, 2023

View reviewed changes

edwardfoyle reviewed Aug 29, 2023

View reviewed changes

sobolk added 3 commits August 29, 2023 10:50

chore: pr feedback

d020721

chore: pr feedback

8afc86d

chore: pr feedback

0bcae81

edwardfoyle previously approved these changes Aug 29, 2023

View reviewed changes

Merge remote-tracking branch 'origin/dev' into smoke-tests-prototype

6ea64ca

sobolk dismissed edwardfoyle’s stale review via 6ea64ca August 29, 2023 20:10

edwardfoyle previously approved these changes Aug 29, 2023

View reviewed changes

Amplifiyer reviewed Aug 30, 2023

View reviewed changes

chore: pr feedback

31ca2ac

sobolk dismissed edwardfoyle’s stale review via 31ca2ac August 30, 2023 15:47

Amplifiyer previously approved these changes Aug 30, 2023

View reviewed changes

edwardfoyle previously approved these changes Aug 30, 2023

View reviewed changes

chore: pr feedback

feea838

sobolk dismissed stale reviews from edwardfoyle and Amplifiyer via feea838 August 30, 2023 16:05

edwardfoyle approved these changes Aug 30, 2023

View reviewed changes

Amplifiyer approved these changes Aug 30, 2023

View reviewed changes

sobolk merged commit 446a4f4 into dev Aug 30, 2023
5 checks passed

sobolk deleted the smoke-tests-prototype branch August 30, 2023 16:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: smoke test automation #13162

ci: smoke test automation #13162

sobolk commented Aug 28, 2023 •

edited

Loading

sobolk Aug 29, 2023

sobolk Aug 29, 2023

edwardfoyle Aug 29, 2023

edwardfoyle Aug 29, 2023

sobolk Aug 29, 2023

edwardfoyle Aug 29, 2023

sobolk Aug 29, 2023

sobolk Aug 29, 2023

edwardfoyle Aug 29, 2023

sobolk Aug 29, 2023

edwardfoyle Aug 29, 2023

sobolk Aug 29, 2023

edwardfoyle left a comment

Amplifiyer Aug 30, 2023

sobolk Aug 30, 2023

Amplifiyer Aug 30, 2023

sobolk Aug 30, 2023

Amplifiyer Aug 30, 2023

sobolk Aug 30, 2023

Amplifiyer Aug 30, 2023

sobolk Aug 30, 2023

Amplifiyer left a comment


		$AMPLIFY_PATH status

		ADD_AUTH_REQUEST=$(cat ../.github/actions/run-smoke-tests-windows/add_auth_request.json)

ci: smoke test automation #13162

ci: smoke test automation #13162

Conversation

sobolk commented Aug 28, 2023 • edited Loading

Description of changes

Issue #, if available

Description of how you validated changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edwardfoyle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Amplifiyer left a comment

Choose a reason for hiding this comment

sobolk commented Aug 28, 2023 •

edited

Loading