🛠️ Repo: Consider strategy for asserting compat with third-party modules

[boneskull]

We want to make sure that a change in Mocha won't break compatibility with other modules frequently used alongside Mocha (think CITGM). We already have a few of these checks in our test suite:

• RequireJS
• Chai
• Webpack (very recently)
• CoffeeScript!
• karma-mocha and thus karma
• maybe something else

but what's going to happen if we keep adding these? A lot of dev dependencies we don't use directly! The only ones we use in the above list are karma and karma-mocha.

Here's a suggestion, but open to any ideas:

1. Create a new module, e.g., @mocha/3p-tests. mocha would not be a dependency of this module, nor would mocha have this module as a dependency
2. Remove the dev deps above (sans karma*) from mocha and add them to the deps of @mocha/3p-tests.
3. Extract the actual tests/fixtures/etc. that consume these modules from our test dir into @mocha/3p-tests
4. For some lifecycle script in mocha--maybe preversion or prepublishOnly--run a script which manually installs @mocha/3p-tests into mocha's node_modules, then executes the tests found in e.g., node_modules/@mocha/3p-tests/test. this would allow us to re-use the remainder of mocha's devdeps when running tests, e.g., karma or unexpected.
5. We could also configure CI to run this script as a job. don't know if that needs to happen on every PR though.

to be clear, npm test should not run these tests. this has precedent: CITGM does not run when you issue make test in a Node.js working copy, but a green CITGM run is a prerequisite for a Node.js release.

the package-maintenance WG also may have something in the works here--not sure if it's ready to be used? What of fugl?

[dominykas]

Getting back to this... Just to understand this properly:

this would allow us to re-use the remainder of mocha's devdeps when running tests

Is there a specific reason for this? What if @mocha/3p-tests was just a regular GH repository, with all the devdeps of its own, and with mocha as a devdep? This way:

• running tests to check relationship with 3rd party modules is an npm test in the @mocha/3p-tests
  • wiby can help here today, as that's the design - you run wiby from mocha and it replaces mocha in 3p-tests with the version from git
• you could have a command which helps do that locally - check out a local copy of 3p-tests, npm link [or similar] the local copy of mocha and then npm test in there

If you think that's workable - I'm happy to make the necessary wiby changes (they're pretty small anyways). Also happy to hear alternatives/other suggestions.

---

From wiby perspective, you'd probably need the following:

• Support testing devDependecies (et al) pkgjs/wiby#55
  • devDependencies support
• Example usage with Github Actions (or an alternative CI) pkgjs/wiby#56
  • a GH action which calls wiby test (this one's easy)
  • another GH action which periodically (?) checks reports wiby result as a comment back to the PR (this one's a bit more involved, but still doable - I just haven't worked through this in my head)
• Pretty print results pkgjs/wiby#57
  • pretty print results

[boneskull]

@dominykas thanks. I am also happy to work on wiby itself if you can use the help.

It's probably worth setting up with Mocha and just kind of see where it's at, where it could improve, or even if we could start using it in earnest.

(I'm not sure what I was thinking with the devdeps-must-be-shared stuff. maybe I was thinking that we would want to write specific tests against specific integrations. if we did, then I wouldn't want to maintain two lists of devdeps and try to keep them in sync.)

[dominykas]

OK, cool, let's try to come up with something. I think #56 is where it's at. And I don't see an obvious way forward and it needs a little bit of thought. Kicking off the initial wiby test is the easy part.

The hard part is to collect the results, because you can't trigger an action off a status change in another repo. Which means some sort of polling, but there's no obvious way of implementing that using actions either, or at least no obvious way which wouldn't use up the precious actions minutes.

So I'm going to do some digging and try to come up with some sort of PoC, but leaving this out here in case someone else comes up with an idea.

I am also happy to work on wiby itself if you can use the help.

I created a bunch of issues today, but I think besides the GH action, none of them are on the critical path towards the full PoC - but feel free to poke around and see if you want to pick up any of them. Haven't had a chance to put them into a single coherent story/plan/roadmap yet.