Camera conformance tests

[marktsuchida]

This is an early experiment on which feedback is very welcome. The idea is to add user- (and developer-) runnable automated tests in MMCore that can be invoked on any real device — cameras for now.

The tests concentrate on device adapter implementation correctness, not correct hardware behavior. This allows the tests to be fully automated.

In the API, this is a single function that takes the device label and optionally the test name to run (default: all). It returns JSON with details of the results. So we can test the tests themselves (as done here), and also potentially collect and organize.

Once this is polished, perhaps we can come up with a workflow for collaborating with users that have the device to fix various issues. At the very least, it will be a tool for bug reporting and one metric for device adapter quality.

It might be nice to add a mode that streams the results progressively so that they can be displayed in a UI.

The 6 tests (mostly 3, with one having 4 variants) implemented so far check that cameras correctly notify the beginning and end of sequence acquisitions (so that autoshutter etc. work correctly) and that they exit the acquisition when there is an error (e.g., sequence buffer overflow). Several of our camera adapters are in fact incorrect in these areas but many of those cannot be fixed without hardware testing.

There are many more potential tests we can add (correct Exposure/Binning properties, correct GetImageWidth(), etc.). For some other devices (esp. stages) we may need to require the user to set motion limits.

The tests run under the current settings (e.g., exposure), so it might be desirable for the results to capture such things.

Note: this adds a dependency (nlohmann_json) to MMCore and only builds using Meson at the moment.

Result of core.runCameraConformanceTests("Camera") on DemoCamera:

Details

{
  "device": {
    "label": "Camera",
    "library": "DemoCamera",
    "name": "DCam"
  },
  "deviceType": "Camera",
  "summary": {
    "durationMs": 8297.412625,
    "passed": 6,
    "total": 6
  },
  "tests": [
    {
      "assertions": [
        {
          "message": "PrepareForAcq called before first InsertImage",
          "passed": true
        }
      ],
      "name": "seq-prepare-before-insert",
      "passed": true
    },
    {
      "assertions": [
        {
          "message": "AcqFinished called after finite sequence completed",
          "passed": true
        }
      ],
      "name": "seq-finished-after-count",
      "passed": true
    },
    {
      "assertions": [
        {
          "message": "AcqFinished called after DEVICE_ERR",
          "passed": true
        },
        {
          "message": "No further InsertImage calls after error",
          "passed": true
        }
      ],
      "name": "seq-finished-on-error-finite",
      "passed": true
    },
    {
      "assertions": [
        {
          "message": "AcqFinished called after DEVICE_ERR",
          "passed": true
        },
        {
          "message": "No further InsertImage calls after error",
          "passed": true
        }
      ],
      "name": "seq-finished-on-error-continuous",
      "passed": true
    },
    {
      "assertions": [
        {
          "message": "AcqFinished called after DEVICE_BUFFER_OVERFLOW",
          "passed": true
        },
        {
          "message": "No further InsertImage calls after error",
          "passed": true
        }
      ],
      "name": "seq-finished-on-overflow-finite",
      "passed": true
    },
    {
      "assertions": [
        {
          "message": "AcqFinished called after DEVICE_BUFFER_OVERFLOW",
          "passed": true
        },
        {
          "message": "No further InsertImage calls after error",
          "passed": true
        }
      ],
      "name": "seq-finished-on-overflow-continuous",
      "passed": true
    }
  ],
  "timestamp": "2026-02-27T14:11:57Z",
  "version": 1
}