Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unreliable test: 'mkdir ... creates folder but uses the "fs" module if in C9' #4390

Open
justinmk3 opened this issue Feb 5, 2024 · 3 comments · Fixed by #4429
Open

unreliable test: 'mkdir ... creates folder but uses the "fs" module if in C9' #4390

justinmk3 opened this issue Feb 5, 2024 · 3 comments · Fixed by #4429
Labels
tests-ci-cd

Comments

@justinmk3
Copy link
Contributor

test failure sometimes occurs:

  1) FileSystem
       mkdir()
         creates folder but uses the "fs" module if in C9: 'a':

      AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert_1.default.ok(mkdirSpy.calledOnce)

      + expected - actual

      -false
      +true
      
      at Context.<anonymous> (d:\a\aws-toolkit-vscode\aws-toolkit-vscode\src\test\srcShared\fs.test.ts:177:24)
@nkomonen-amazon
Copy link
Contributor

Looking

nkomonen-amazon added a commit to nkomonen-amazon/aws-toolkit-vscode that referenced this issue Feb 5, 2024
@nkomonen-amazon
fix: flaky fs test
95b6cb1 
Reported by aws#4390, this test is failing on the mkdirSpy call count assertion.

Not exactly sure why this is flaky, but maybe it is due
to the previous folders not being cleaned up.

This commit deletes the created folders after each test finishes.
Additionally, this commit checks the actual spy call count so that
we can better debug the issue

Signed-off-by: nkomonen <[email protected]>
@nkomonen-amazon nkomonen-amazon mentioned this issue Feb 5, 2024
Merged
nkomonen-amazon added a commit to nkomonen-amazon/aws-toolkit-vscode that referenced this issue Feb 5, 2024
@nkomonen-amazon
fix: flaky fs test
fcef50f 
Reported by aws#4390, this test is failing on the mkdirSpy call count assertion.

Not exactly sure why this is flaky, but maybe it is due
to the previous folders not being cleaned up.

This commit deletes the created folders after each test finishes.
Additionally, this commit checks the actual spy call count so that
we can better debug the issue

Signed-off-by: nkomonen <[email protected]>
nkomonen-amazon added a commit to nkomonen-amazon/aws-toolkit-vscode that referenced this issue Feb 6, 2024
@nkomonen-amazon
fix: flaky fs test
c7a1231 
Reported by aws#4390, this test is failing on the mkdirSpy call count assertion.

Not exactly sure why this is flaky, but maybe it is due
to the previous folders not being cleaned up.

This commit deletes the created folders after each test finishes.
Additionally, this commit checks the actual spy call count so that
we can better debug the issue

Signed-off-by: nkomonen <[email protected]>
nkomonen-amazon added a commit to nkomonen-amazon/aws-toolkit-vscode that referenced this issue Feb 6, 2024
@nkomonen-amazon
fix: flaky fs test
22761e1 
Reported by aws#4390, this test is failing on the mkdirSpy call count assertion.

Not exactly sure why this is flaky, but maybe it is due
to the previous folders not being cleaned up.

This commit deletes the created folders after each test finishes.
Additionally, this commit checks the actual spy call count so that
we can better debug the issue

Signed-off-by: nkomonen <[email protected]>
nkomonen-amazon added a commit to nkomonen-amazon/aws-toolkit-vscode that referenced this issue Feb 6, 2024
@nkomonen-amazon
fix: flaky fs test
edb707d 
Reported by aws#4390, this test is failing on the mkdirSpy call count assertion.

Not exactly sure why this is flaky, but maybe it is due
to the previous folders not being cleaned up.

This commit deletes the created folders after each test finishes.
Additionally, this commit checks the actual spy call count so that
we can better debug the issue

Signed-off-by: nkomonen <[email protected]>
nkomonen-amazon added a commit that referenced this issue Feb 6, 2024
@nkomonen-amazon
fix: flaky fs test (#4391)
6982bcc 
Reported by #4390, this test is failing on the mkdirSpy call count
assertion.
Not exactly sure why this is flaky, but maybe it is due to the previous
folders not being cleaned up.

## Solution

This commit deletes the created folders after each test finishes.
Additionally, this commit checks the actual spy call count so that we
can better debug the issue

<!---
    REMINDER:
    - Read CONTRIBUTING.md first.
    - Add test coverage for your changes.
    - Update the changelog using `npm run newChange`.
    - Link to related issues/commits.
    - Testing: how did you test your changes?
    - Screenshots
-->

## License

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

Signed-off-by: nkomonen <[email protected]>
nkomonen-amazon added a commit to nkomonen-amazon/aws-toolkit-vscode that referenced this issue Feb 13, 2024
@nkomonen-amazon
test: try fix flaky test
37b39b5 
Not sure if this fixes aws#4390, but we now
reuse the same sandbox instead of recreating it
each time. We were failing a test where a spied method
was being called more than expected.

Signed-off-by: nkomonen <[email protected]>
@nkomonen-amazon nkomonen-amazon mentioned this issue Feb 13, 2024
Merged
nkomonen-amazon added a commit to nkomonen-amazon/aws-toolkit-vscode that referenced this issue Feb 13, 2024
@nkomonen-amazon
test: try fix flaky test
870f494 
Not sure if this fixes aws#4390, but we now
reuse the same sandbox instead of recreating it
each time. We were failing a test where a spied method
was being called more than expected.

Signed-off-by: nkomonen <[email protected]>
nkomonen-amazon added a commit that referenced this issue Feb 14, 2024
@nkomonen-amazon
test: try fix flaky test (#4429)
dd82987 
Not sure if this fixes #4390, but we now
reuse the same sandbox instead of recreating it
each time. We were failing a test where a spied method
was being called more than expected.

Signed-off-by: nkomonen <[email protected]>
@justinmk3
Copy link
Contributor Author

justinmk3 commented Feb 22, 2024 •
edited
Loading

Similar case is present on latest master:

  1 failing
  1) FileSystem
       mkdir()
         creates folder but uses the "fs" module if in C9: 'a/b/c':

      AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

2 !== 1

      + expected - actual

      -2
      +1
      
      at Context.<anonymous> (src/test/srcShared/fs.test.ts:179:24)

@justinmk3 justinmk3 reopened this Feb 22, 2024
@justinmk3
Copy link
Contributor Author

Asserting spy.callCount is fragile. Can we assert the actual result instead?

@nkomonen-amazon nkomonen-amazon mentioned this issue Feb 23, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests-ci-cd
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: try fix flaky test nkomonen-amazon/aws-toolkit-vscode
2 participants
@justinmk3 @nkomonen-amazon