Test Methods in Public Global Environment

[bgribaudo]

The public global environment (e.g. what the .query.pq file can see) contains a number of unit test-related methos (like Fact, FactCheck, FactSummarize, Theory, etc.).

Is the new SDK shipping with a built-in unit test framework, or is this something that leaked through?

[mattmasson]

It's a built-in unit testing framework that is compatible with the samples we posted in the TripPin tutorial (+additional features). But it's not complete, and I'm not convinced it's what we want as part of our official unit testing functionality.

[bgribaudo]

I can imagine pros and cons both ways.

If you continue to include it, should its method names follow the "x.y" naming convention standard (e.g. FactCheck -> Test.FactCheck)?

Alternately, instead of including the framework as built in to the SDK's standard library, what about incorporating it as part of the default template for new projects? That way, someone can get rid of (or modify) it, if desired, but at the same time its presence by default would (hopefully) encourage people toward unit testing.

[mattmasson]

The downside to including the test functions as part of the project template is that it makes it harder for us to augment and improve the functionality going forward. That said, including test templates and commands to help generate tests is definitely something we want to do.

I agree on the naming. One of the original goals was to make the updated test framework 100% compatible with the one from the TripPin samples (as it's actually heavily used by a few MSFT teams). I still see value in that, but we could do it by having a code snippet that redirects the old functions to the new (Fact = (test, expected, actual) => Test.Fact(test, expected, actual);)

[bgribaudo]

If it is going to be built-in, any chance it (the test module/extension) could be open source on GitHub, for reference's sake, as well as to allow user contributions?

[bgribaudo]

Not sure if this would help, but an idea as to an alternative way of pulling this off:

• The default test framework could ship as a .pq extension (vs. baked into the distro or a .mez/.pqx) included as a separate file in the VSIX/tools package.
• The default project template could contain config including this extension, such that it is passed as a second --extension argument to PQTest.

This would give you the ability to ship updates to the default framework (avoiding the maintenance issue with directly including the framework's source in the project template). At the same time, the test framework isn't baked into the SDK's M distro, so the SDK user has the option of editing the project's config to either not include the test framework or to include a different test framework (e.g. their own).

Since the default framework would be in an M extension, its source would be readable for reference, and if it were MIT licensed, cloneable, if the user wants to modify it.

For the current Microsoft projects based on the TripPin-style test framework, they could edit their project configs to include a "compatibility shim" extension that would map from old-style to new-style names, or those shims could be built into the base test extension.