Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests: use the default mocha reporter for better display for errors #1375

Merged
merged 1 commit into from
Jan 14, 2025

Conversation

fflorent
Copy link
Collaborator

Context

When running yarn test: commands (like yarn test:server), the displayed errors are difficult to read. Especially when comparing two objects deeply (using assert.deepEqual):

I heavily use yarn test:server or yarn test:gen-server instead of yarn test in order to avoid launching a browser for the tests.

An example of the display:

(2) Scim when enabled using GRIST_ENABLE_SCIM=1 DELETE /Users/{id} should return 403 for system users
AssertionError: expected { schemas: [ 'urn:ietf:params:scim:api:messages:2.0:Error' ],
  status: '403',
  detail: 'System user deletion not permitted.' } to deeply equal { schemas: [ 'urn:ietf:params:scim:api:messages:2.0:Error' ],
  detail: 'System user deletion not permitted.' }
    at Proxy.assertEql (node_modules/chai/lib/chai/core/assertions.js:1084:10)
    at Proxy.methodWrapper (node_modules/chai/lib/chai/utils/addMethod.js:57:25)
    at doAsserterAsyncAndAddThen (node_modules/chai-as-promised/lib/chai-as-promised.js:289:22)
    at Proxy.<anonymous> (node_modules/chai-as-promised/lib/chai-as-promised.js:255:20)
    at Proxy.overwritingMethodWrapper (node_modules/chai/lib/chai/utils/overwriteMethod.js:78:33)
    at Function.assert.deepEqual.assert.deepStrictEqual (node_modules/chai/lib/chai/interface/assert.js:232:56)
    at checkOperationOnTechUserDisallowed (test/server/lib/Scim.ts:146:16)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at Context.<anonymous> (test/server/lib/Scim.ts:613:9)

Proposed solution

We remove the test/xunit-file reporter and use the mocha default reporter. The above error is displayed like below, which is handier:

  2) Scim
       when enabled using GRIST_ENABLE_SCIM=1
         DELETE /Users/{id}
           should return 403 for system users:

      AssertionError: expected { schemas: [ 'urn:ietf:params:scim:api:messages:2.0:Error' ],
  status: '403',
  detail: 'System user deletion not permitted.' } to deeply equal { schemas: [ 'urn:ietf:params:scim:api:messages:2.0:Error' ],
  detail: 'System user deletion not permitted.' }
      + expected - actual

         "detail": "System user deletion not permitted."
         "schemas": [
           "urn:ietf:params:scim:api:messages:2.0:Error"
         ]
      -  "status": "403"
       }
      
      at Proxy.assertEql (node_modules/chai/lib/chai/core/assertions.js:1084:10)
      at Proxy.methodWrapper (node_modules/chai/lib/chai/utils/addMethod.js:57:25)
      at doAsserterAsyncAndAddThen (node_modules/chai-as-promised/lib/chai-as-promised.js:289:22)
      at Proxy.<anonymous> (node_modules/chai-as-promised/lib/chai-as-promised.js:255:20)
      at Proxy.overwritingMethodWrapper (node_modules/chai/lib/chai/utils/overwriteMethod.js:78:33)
      at Function.assert.deepEqual.assert.deepStrictEqual (node_modules/chai/lib/chai/interface/assert.js:232:56)
      at checkOperationOnTechUserDisallowed (test/server/lib/Scim.ts:146:16)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at Context.<anonymous> (test/server/lib/Scim.ts:613:9)

Alternatively, we may only use the test/xunit-file reporter when running the tests in Jenkins

Has this been tested?

  • 👍 yes, I added tests to the test suite
  • 💭 no, because this PR is a draft and still needs work
  • 🙅 no, because this is not relevant here
  • 🙋 no, because I need help

"test:server": "TEST_SUITE=server TEST_SUITE_FOR_TIMINGS=server TIMINGS_FILE=test/timings/server.txt ./test/test_env.sh mocha ${DEBUG:+'-b'} -g \"${GREP_TESTS}\" -R test/xunit-file '_build/test/server/**/*.js'",
"test:gen-server": "TEST_SUITE=gen-server TEST_SUITE_FOR_TIMINGS=gen-server TIMINGS_FILE=test/timings/gen-server.txt ./test/test_env.sh mocha ${DEBUG:+'-b'} -g \"${GREP_TESTS}\" -R test/xunit-file '_build/test/gen-server/**/*.js'",
"test:server": "TEST_SUITE=server TEST_SUITE_FOR_TIMINGS=server TIMINGS_FILE=test/timings/server.txt ./test/test_env.sh mocha ${DEBUG:+'-b'} -g \"${GREP_TESTS}\" '_build/test/server/**/*.js'",
"test:gen-server": "TEST_SUITE=gen-server TEST_SUITE_FOR_TIMINGS=gen-server TIMINGS_FILE=test/timings/gen-server.txt ./test/test_env.sh mocha ${DEBUG:+'-b'} -g \"${GREP_TESTS}\" '_build/test/gen-server/**/*.js'",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is this interesting note:

https://github.com/fflorent/grist-core/blob/4a9bdf6c220a0554c684dddf1a8d62ba85d07553/sandbox/grist/runtests.py#L1-L10

I don't have access to Jenkins or arc unit. If you feel like it would be better to use the xunit reporter for Jenkins or whatever, I may add a $REPORTER env variable. Please tell me if you would like that.

Copy link
Member

@paulfitz paulfitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @fflorent. Looks plausible, and since it only touches package.json it shouldn't actually affect any Grist Labs test infrastructure.

The test/xunit-file utility does produce a timings output that was used in the past to balance test jobs, but last time I rebalanced I just eyeballed it and it was fine.

@paulfitz paulfitz merged commit e0772fe into gristlabs:main Jan 14, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants