Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add integration tests #45

Merged
merged 6 commits into from
Feb 6, 2025
Merged

chore: add integration tests #45

merged 6 commits into from
Feb 6, 2025

Conversation

nadiamoe
Copy link
Member

@nadiamoe nadiamoe commented Oct 25, 2024

This PR adds an integration test suite to crocochrome, which should help catch succinct errors in the crocochrome setup, such as misconfigurations of chromium, permissions, capabilities, etc. All other tests in the codebase run a simple script instead of real chromium.

The suite roughly works like this:

  • It builds the crocochrome container using the Dockerfile, and starts it
  • It starts a paused k6 container (grafana/k6, updated by renovate so we can notice if something breaks)
  • For each test script in the test case:
    • Creates a new session using crocochrome API
    • Execs into the k6 container and uses it to run the script against the WS url provided by crocochrome
    • Check the return code of k6 is 0, and that the string error does not appear in k6 logs
    • Deletes the session using crocochrome API

As normal, if any of these steps fail, the test fails.

All is done with testcontainers in go, so this test will normal run as part of go test without any special configuration, and should work everywhere as long as docker is available. As crocochrome is not multiarch though, running this test on arm might be very slow or not work at all. If this is the case, this test can be skipped by setting the env var TEST_INTEGRATION_SKIP to any non-empty value.

@nadiamoe nadiamoe requested a review from a team as a code owner October 25, 2024 11:50
@nadiamoe nadiamoe changed the title Add integration test Add integration tests Nov 4, 2024
@nadiamoe nadiamoe force-pushed the integration-tests branch 3 times, most recently from efb3e05 to 024c68d Compare November 4, 2024 11:34
@nadiamoe nadiamoe force-pushed the integration-tests branch 3 times, most recently from dfa58fe to 0390e0f Compare December 16, 2024 12:41
Copy link

@The-9880 The-9880 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, a few small comments

"strings"
)

// buildContainer builds the container image for sm-k6-runner by running `docker` commands, generating an image on the

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// buildContainer builds the container image for sm-k6-runner by running `docker` commands, generating an image on the
// buildContainer builds the container image for crocochrome by running `docker` commands, generating an image on the

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hehe, this is a copypaste and it shows. Good catch!

if err != nil {
t.Fatalf("starting crocochrome container: %v", err)
}
testcontainers.CleanupContainer(t, cc)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Not a big deal, but the doc comment suggests we call this before the error check as a defer.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also on L91

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have done the "before error check" part, but the defer one feels like an error in the docs to me: CleanupContainer is already wrapping the remove call in t.Cleanup, there doesn't seem to be any point to defer it further: https://github.com/testcontainers/testcontainers-go/blob/add4ac3f019b773ae08c1af05f57554ba6819430/testing.go#L73

ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)

image, err := buildContainer("..", "crocochrome")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth adding a step to delete the image after the integration test suite runs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and implemented this, but after doing so I'm not sure there is really a point to it.
Even if I run docker rmi ..., all the build cache and intermediate layers will remain, so I wonder what is the point of removing the tag if the users would need to garbage-collect the things that actually take space on its own.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I don't think it's necessary either :)

Copy link
Member Author

@nadiamoe nadiamoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @The-9880 ! Will apply the suggested changes on monday :)

"strings"
)

// buildContainer builds the container image for sm-k6-runner by running `docker` commands, generating an image on the
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hehe, this is a copypaste and it shows. Good catch!

@nadiamoe nadiamoe requested a review from The-9880 February 5, 2025 10:31
@nadiamoe nadiamoe changed the title Add integration tests chore: add integration tests Feb 6, 2025
@nadiamoe nadiamoe merged commit 53cef04 into main Feb 6, 2025
7 checks passed
@nadiamoe nadiamoe deleted the integration-tests branch February 6, 2025 11:22
@sm-release-app sm-release-app bot mentioned this pull request Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants