Automated Testing & Code Coverage

Introduction

Note that this guide does not cover more advanced test cases beyond simple DM tests. Please ask on Discord for assistance with advanced testing.

To automate testing of the compiler and runtime, OpenDream (and the underlying Robust Toolbox engine) make use of the NUnit testing framework. If needed, you can find further details and documentation of NUnit on their website. These tests are ran automatically on all pull requests by GitHub Actions.

In most cases, an OpenDream test is simply a self-contained DM code file that attempts to check one of the following pre-defined outcomes:

• Code successfully compiles and runs without error.
• Code fails to compile (for testing that we actually compile error on bad code or that a pragma actually emits).
• Runtime error occurs (for testing that we actually runtime on bad code).
• Proc returns TRUE.
• ASSERT() that a value is what we expect it to be.

Robust Toolbox also has its own suite of tests, though running these is typically not recommended as it can take quite a while and generally should not be pertinent to OpenDream development.

Code Coverage

Code coverage measures what percentage of OpenDream's source code is actually executed by automated testing and can be used to guide test development. If a piece of code is not covered, it needs tests written for it.

You can peruse OpenDream's code coverage by file here.

Unit Tests vs Integration Tests

All of OpenDream's tests are either unit tests or integration tests.

This guide will not get too detailed about the differences between unit tests and integration tests, but there are some important things to know:

Unit Tests

• Use essentially just a compiler & interpreter, not a full-fledged server.
• Contained within Content.Tests/DMProject/Tests
• Does not allow creating/using subtypes of /atom. Only /datum.
• Does not have a map to test on.
• Does not support a variety of other things beyond simply compiling and executing a subset of DM code.
• Code execution begins in a user-declared RunTest() proc (/proc/RunTest()).
• One test per DM file.

Integration Tests

• Full-fledged server.
• Contained within Content.IntegrationTests/DMProject/Tests.
• Allows creation of atom types/subtypes.
• Has a map to test on.
• Code execution begins in a user-declared RunTest() override (e.g. /datum/unit_test/your_new_test/RunTest()). Ignore that the typepath says unit_test instead of integration_test.
• One test per DM file.

Generally speaking, you want to be writing unit tests unless you depend on features that necessitate an integration test. Unit tests have less overhead and are faster to execute.

Running Tests

With a GUI

Your IDE or text editor (e.g. Rider, Visual Studio, or VSCode) has the ability to run all or specific tests through a GUI. Refer to your application's documentation for how to do that. Unit tests are in Content.Tests while integration tests are in Content.IntegrationTests.

Note that DM tests are detected automatically by the testing framework, therefore you may not initially see all of the individual tests listed until after the first run.

With the Terminal

• Running dotnet test will run all tests including Robust Toolbox's (not advised).
• Running dotnet test Content.Tests and dotnet test Content.IntegrationTests will run OpenDream's unit tests and integration tests, respectively.

Creating Tests

First determine if you're writing a unit test or an integration test based on the aforementioned differences. Try to create a unit test unless you need features that are restricted to integration tests.

Don't stress about which subdirectory under Tests to put your test file in; it is purely organizational and mostly unimportant.

The entry point for a test is the user-declared RunTest() proc. The testing framework will automatically call RunTest() when your test executes.

Creating Unit Tests

1. Create a new DM file in Content.Tests/DMProject/Tests.
2. Declare the RunTest() proc: /proc/RunTest().
3. Put the code you want to execute within that RunTest() proc.

By default, a test is expected to compile and run without any compile-time or runtime errors. Most testing in this case is done using ASSERT() to check that a value is what you expect it to be. For testing other scenarios, you can use test flags.

Test Flags

The first line of a unit test can be a DM code comment with a test flag that impacts test behavior.

For example, the first line of your file could be: // RETURN TRUE

This will cause the test to fail unless RunTest() returns TRUE.

Here are all of the test flags:

• // IGNORE - This DM file will not be executed as a DM test. Useful for writing helper code that you want to share between multiple tests (just don't forget to #include it in your tests).
• // RETURN TRUE - RunTest() must return TRUE for the test to pass.
• // NO RETURN - RunTest() must not return a value at all for the test to pass.
• // RUNTIME ERROR - The test must create a runtime error to pass. Currently, you cannot control which runtime error you're looking for.
• // COMPILE ERROR OD#### - The test will only pass if the specified compiler error occurs, where OD#### is the WarningCode for that error (e.g. to test the PointlessParentCall pragma you would specify // COMPILE ERROR OD2205 then include #pragma PointlessParentCall error in your code).

Creating Integration Tests

1. Create a new DM file in Content.IntegrationTests/DMProject/Tests.
2. Declare the RunTest() override: /datum/unit_test/your_new_test/RunTest() where your_new_test is the name of the test. Ignore the fact that the typepath says unit_test even though this is an integration test.
3. Put the code you want to execute within that RunTest() override.

Note that integration tests do not have access to unit tests' test flags (e.g. // RUNTIME ERROR or // IGNORE).

Broken Tests

There are a variety of tests that have already been written that should work in OpenDream but currently don't. As a result, they are currently ignored. These can be found in Content.Tests/DMProject/BrokenTests.

These broken tests are a good TODO list of problems in OpenDream that need to be fixed. Simply move them from BrokenTests to Tests and fix whatever bug with OpenDream is causing them to fail.

Note that many of these tests are quite old and should be executed in BYOND first to confirm that they are still accurate and correct behavior.