Who is using model-based testing with xstate in production systems?

[Silverwolf90]

At my work we've been experimenting with it for end-to-end tests (with @xstate/test + Cypress), and I'm starting to formulate a long-term testing strategy that integrates across product/dev/qa teams.

Probably the biggest obstacle in our experiments was maintaining a fast enough feedback loop. We found ourselves heavily compromising the fidelity of the test machine as a "user model" to avoid an explosion of test cases

• We avoided granular states
  • eg: A login-form only has idle,error instead of representing the email/password field states
• Had transitions that mapped to very complex UI interactions (ie: complexity is moved out of the state machine and into the events)
  • See example below
• Labeling different event cases explicitly with subtype to also trigger certain paths
  • See example below
• Included special "initializing" states to set flags to disable certain paths (via guards)
• Needed to write helpers to isolate specific plans/paths so that we could only run a single test at a time.

All of this to avoid "test explosions" even for relatively simple pages (eg: login form). Something doesn't feel right?

Additionally, the resulting test machines start to deviate pretty heavily from the model in our product manager's head. Ideally, the test machine when visualized should directly match the "user model" as defined by the product team. In fact, if the product team was using state machine diagrams, I don't see a reason why this couldn't be matched exactly?

That being said, I've opened up a PR here that should allow much greater flexibility in using a test model with @xstate/test by "manually" defining paths with a sequence of a events. I think it's the "missing link" to make model-based testing with xstate feasible at scale.

Curious to know if anyone else is using model-based testing with xstate in "real life"? And if so, what are your thoughts about it?

Example from above

const events = {
  FILL_LOGIN_FORM: {
    exec: (testContext, event) =>{
      const { user } = testContext;
      const { overrides, attempts } = event;

      const values = {
        ...pick(user, ['email', 'password']),
        ...overrides,
      };

      for (let i = 0; i < attempts; i += 1) {
        if (values.email) {
          cy.get('[name=email]').clear().type(values.email);
        }

        if (values.password) {
          cy.get('[name=password]').clear().type(values.password);
        }

        cy.get('[type=submit]').click();
      }
    },

    // To make sure we hit all possible states we want to provide an event case to hit
    // every transition.
    //
    // We use this `subtype` pattern as a mechanism to explicitly label the possible
    // cases which makes for simpler transition conditions
    cases: [
      // 5 failed attempts leads to a suspended user for 24hr
      { attempts: 5, subtype: 'invalid-password', overrides: { password: 'asdf1235' } },
      // 2 failed attempts will warn the user that they will be suspended after 5 failures
      { attempts: 2, subtype: 'invalid-password', overrides: { password: 'asdf1235' } },
      { attempts: 1, subtype: 'invalid-password', overrides: { password: 'asdf1235' } },
      { attempts: 1, subtype: 'invalid-email', overrides: { email: '[email protected]' } },
      { attempts: 1, subtype: 'missing-password', overrides: { password: '' } },
      { attempts: 1, subtype: 'missing-email', overrides: { email: '' } },
      { attempts: 1, subtype: 'valid-credentials', overrides: {} },
    ],
  },
};

[DanCouper]

Yep, so I had a load of issues with this relating to how I designed the first few iterations of the machine that runs our authentication, and it wasn't really feasible to use the model testing for quite a while. Now have it running and it's great, but some caveats. Don't know how applicable this is, but I can describe exactly how I have it set up

• machine backs an auth library that's then installed into our apps.
• apps are all React (native + web), and I'm just using @testing-library/react rather than Cypress/Selenium/whatever. Shouldn't make a practical difference, I could run as E2E tests, but I want speed. I can do this because:
• I have a test app that I use for development which is just the authentication, nothing else. It's a complete application. But it's also completely disconnected from the actual implementations (it has three roles: it's used for development, as an implementation example and for integration tests). Each stage of auth in the app includes a component that prints out the statuses (whether it's active, what the state ID is, whether there's an error etc) so that I can inspect that during the tests.
• each state in the machine === one stage of auth, except for two internal states which have conditional guards (specifically, one checks whether the machine has been configured for OTP or username/password auth and one checks whether PIN device security is on or off). The internal state IDs are just prefixed with "INTERNAL_", haven't done anything special there.
• machine does no async stuff, all API calls are done in a set of hooks, one for each state exposed to the users. The hooks accept the relevant API callback (for example Auth.signIn) and expose a method for the end-user (eg submitUsernameAndPassword).
• the method is just a useCallback (could be useEffect but whatever) which, when ran, executes the callback and sends the appropriate event to the machine depending on the response (eg { type: "USERNAME_AND_PASSWORD_VALID", userdata: resp.user }).
• the events as much as is possible avoid having any payload, the majority are just { type: "SOME_MESSAGE" }.
• loading states are handled by the hook, never in the machine to avoid a proliferation of either loading states or assigning/deassigning some loading flag to context. NOTE that these could be actors instead, but then I would have got into needing to marshal a shedload of actors in the main machine and frankly I didn't really want to do that.
• the events may have an error payload, that was unavoidable because the API we use (Cognito) can indicate different stuff depending on specific errors, which would then need to trigger a different path through the state machine.

So that gives me one machine, with a minimal context, minimal number of events, very few actions, with states that map directly to screens in the end apps.

Machine is initiated in a provider which exposes the result of useInterpret (via another hook); that exposed Interpreter is the object that's then pulled into each of the hooks to allow messages to be sent.

For the hook callbacks, I put all the implementations for my test app into a single file and import the relevant one into the relevant component that uses the relevant hook. For the tests I can then just mock that file out and replace all the implementations with spies. Need to do this because a couple of the callbacks don't take input (for example, checking sessions), so need adjustment during the test runs.

The model test can now be set up:

• import the callback implementations, and override with a manually mocked version of the module that returns known responses (VALID_USERNAME, INVALID_USERNAME etc).

1. create a test model that reflects user journeys through the app. NOTE that this is much more granular than the actual machine -- I do have error states here, and loading states and whatever else. Each state needs to reflect what the user can see at that particular point in time. eg:

   const machine = createMachine({
     id: "otpNoPin",
     initial: "CheckingSession",
     states: {
       CheckingSession: {
         on: {
           THERE_IS_A_SESSION: "Authenticated",
           THERE_IS_NO_SESSION: "SubmittingUsername",
         },
         meta: {
           test: async (screen: RenderResult) => {
             expect(await screen.findByText(`Current stage: ${AuthStateId.CheckingForSession}`).toBeDefined();
   

2. attach events, eg:

   const model = createModel<RenderResult>(machine).withEvents({
   THERE_IS_A_SESSION: async (screen: RenderResult) => {
   	userEvent.click(await screen.findByText("Check for a session"));
   	await waitFor(() => expect(MockCb.checkSessionCb).toHaveBeenCalled());
   	// expect(await screen.findByText("Loading: true")).toBeDefined();
   	await screen.findByText("Loading: false");
   },
   THERE_IS_NO_SESSION: async (screen: RenderResult) => {
   	MockCb.checkSessionCb.mockRejectedValueOnce(null);
   	userEvent.click(await screen.findByText("Check for a session"));
   	await waitFor(() => expect(MockCb.checkSessionCb).toHaveBeenCalled());
   	MockCb.checkSessionCb.mockReset();
   	// expect(await screen.findByText("Loading: true")).toBeDefined();
   	await screen.findByText("Loading: false");
   },
   

3. generate test plans, eg:

   const testPlans = otpNoPin.model.getSimplePathPlans();
   
   testPlans.forEach((plan) => {
     describe(`authentication test system ${plan.description}`, () => {
       plan.paths.forEach((path) => {
     it(path.description, async () => {
       const screen = render(<AuthProvider loginFlowType="OTP" deviceSecurityType="NONE"><TestApp /></AuthProvider>);
       await path.test(screen);
     });
   });
     });
   });
   
   it("should have full coverage", () => otpNoPin.model.testCoverage());
   

Once that's all set up, it has been. incredibly easy to add tests.

Notes, anyway:

• if I want to change to E2E, I basically just adjust the render to whatever the E2E test framework expects then search/replace on the methods used in the meta and the events.

• if I want to test the actual machine itself (I know this isn't recommended, but I am doing that to detach the XState code from the React code in preparation for making use of the machine in some non-React-based experiments), the same thing, though I then need to adjust the events a bit more to actually send relevant messages.

• Moving any async calls out of the machine and as close to the user as possible was a win: from a dev and test PoV, it made it a hundred times easier to debug. When they lived in the machine, I needed to build a facade for all the APIs, and pass that in. Then add the onError/onDone, often based on data sent in the response (I'm using TS, and typings were a huge pain point here even after recent improvements). Then store a load of stuff in context + provide all the actions needed to assign/deassign to context. And add conditionals to discriminate between responses in the onDone/onError blocks. Critically, this all made it extremely difficult to get at the requests or debug them.

• Moving transient states (eg loading) out was again a win; it's much easier from a test point of view, and seriously slimmed down context, events and actions.

• Not using hierarchical states or actors simplified things by a staggering amount -- found them incredibly useful, but at the cost of a huge jump in complexity.

---

All of this to avoid "test explosions" even for relatively simple pages (eg: login form). Something doesn't feel right?

So, once above was set up, the test files are pretty big. eg my state might be "SubmittingUsername". "USERNAME_VALID" event is sent, goes to "SubmittingOtp". The user has three goes, so "OTP_INVALID" sent, goes to "OtpInvalidRetry1". "OTP_INVALID" sent, goes to "OtpInvalidRetry2". "OTP_VALID" sent, goes to "SubmittingNewPin". etc. There are going to be a lot of states for anything complex (or even for simple stuff). But this doesn't really matter, because

1. the test model is not the real model, the test model just exercises the implementation of the real model
2. each state and each event is likely going to almost be a copy-paste job with minor adjustments
3. it's very easy to add, remove or adjust each state/event test/action
4. LoC count for the test model + events described may be high, but it's going to be very readable, you can be as verbose as you want with state/event names, and it's going to be

   "SomeUserState": { on: { DESCRIPTIVE_EVENT: "NewUserState" }, { meta: /* some basic test */ } }
   

   so what I think is ideal is lots of states and probably lots of events, but with the blocks of code that execute the tests as small in scope as possible.

Additionally, the resulting test machines start to deviate pretty heavily from the model in our product manager's head. Ideally, the test machine when visualized should directly match the "user model" as defined by the product team. In fact, if the product team was using state machine diagrams, I don't see a reason why this couldn't be matched exactly?

Deviating from the model is fine because the test should not be testing the underlying FSM model, it should be testing the implementation of that model.

That being said, I've opened up a PR here that should allow much greater flexibility in using a test model with @xstate/test by "manually" defining paths with a sequence of a events. I think it's the "missing link" to make model-based testing with xstate feasible at scale.

By any chance do you have a lot of guard clauses, and/or cycles? The point of the model-based test IMO is to exercise all possible paths through the graph, so manually specifying them would seem to indicate underlying design issues. Removing all guard clauses from my machines except for those that check values set at init time (ie no guards based on context) made it drastically easier to test, though I appreciate that may not be feasible.

All this might not be useful, sorry for wall of text!

[Silverwolf90]

Deviating from the model is fine because the test should not be testing the underlying FSM model, it should be testing the implementation of that model.

I'm not sure I understand what you mean by this. Isn't the implementation of the "user model" the UI itself? In other words, if the product manager says "our login page has an initial empty state, an error state, and a success state" then the machine that will drive my test model will have an empty, error and success states (and each should correspond to some UI). Of course in reality, the PM is suggesting something much more complex, we have warnings, different error types, partially filled fields, etc. So you really end up with lots of nested and parallel states if you want a high-fidelity representation of what your "user" sees. Why not have a machine that represents all these states? But when you generate paths it could go from error to different error to warning to another different error to success -- and all possible combinations.

[DanCouper]

Okay, as an example, we use an OTP. So once the user has submitted an email they get emailed a code, and say we're on that screen in the app.

So the user has a code input box and a submit button. If they type the code incorrectly once, they get a message saying they have two more attempts. If they type it incorrectly a second time, message says one more attempt. Type it incorrectly a third time, they get bumped out, but if I just focus on the OTP input: things that change for the user are error messages and loading indicators. There are six states:

1. an empty input, no error.
2. type code and press button results in loading.
3. same as 1 but with an error message.
4. same as 2, but with error from 3 still present.
5. same as 3, but slightly different error message.
6. same as 4, but with error from 5 still present.

But the machine, that has one state in my model. An event may come in and flip it to another state. But if there's an error event, pop the error message in context and don't do anything. The user couldn't care less how I store them or how I've set up the machine, the only thing that's important is that if there's been some error, tell the user at the right time.

And I could go and do some testing on what's currently in the context at any given time, but that doesn't actually reflect implementation directly, and that's what I need to test, that's what's important.

If I test what the actual interaction with an implemented machine looks like, then I'm testing actual usage. If I test the underlying machine, that's kinda redundant. The machine doesn't need to be a 1-to-1 diagram of the UI, that's far too complicated, the map is not the territory.

[Silverwolf90]

Ah, the machine I'm referencing in my post above is not the underlying implementation, it is the test model's machine. In other wods, it's a machine purely for driving the end-to-end tests. The underlying implementation of the UI doesn't even use xstate. The "test model" is literally representing the application from the user's point of view so that we can simulate a user -- I agree that testing the underlying implementation would be the wrong thing to do (and is a common confusion of how to test machines with xstate).

[UberMouse]

I use @xstate/test to power the E2E tests for the Electron app I'm building at our company. So far I've avoided having any overly specific state representations and I only test the happy paths through features, the test suite also isn't overly large (I think around 20-30 test cases generated) but it's been working out fairly well so far.

I was unsatisfied with how @xstate/test models were built so I built a wrapper around it that allows you to create nestable "test zones" and generate an @xstate/test model from those composed zones. The code for it is pretty hacked together but if there is interest I could see about tidying it up and packaging it into an open source repo. I've uploaded the README as a gist to give an example of it https://gist.github.com/UberMouse/2fcfc2d714e0eef68b2b2cc98d194600

[sukima]

I use it in my framework's (Ember.js) given test suite (QUnit). I have established a pattern for it and has helped immensely when unit testing complex (many conditionals) components.

function testWrapper(testFn) {
  return (testContext, state) => {
    testContext.assert.ok(
      true,
      `§ Testing state ${JSON.stringify(state.value)} ← event ${JSON.stringify(state.event)}`,
    );
    return testFn(testContext, state);
  };
}

module('Integration | Component | FooBar', function (hooks) {
  setupRenderingTest(hooks);

  const page = new FooBarPageObject('#test-subject');
  const testMachine = createMachine({
    id: 'fooBarTest',
    initial: 'firstRender',
    states: {
      firstRender: {
        on: { NEXT: 'secondStep' },
        meta: {
          test: testWrapper(function ({ assert }) {
            assert.dom(page.element).exists();
            assert.strictEqual(page.status, 'first');
          }),
        },
      },
      secondStep: {
        meta: {
          test: testWrapper(function ({ assert }) {
            assert.strictEqual(page.status, 'second');
          });
        },
      },
    },
  });
  const testModel = createModel(testMachine).withEvents({
    async NEXT() {
      await page.clickNextButton();
    },
  });
  const testPlans = testModel.getSimplePathPlans();

  testPlans.forEach((plan) => {
    module(plan.description, function () {
      plan.paths.forEach((path) => {
        test(path.description, async function (assert) {
          await render(hbs`<FooBar id="test-subject" />`);
          await path.test({ assert });
        });
      });
    });
  });
});

QUnit bundles assertions into a pile of steps inside the test output and it was difficult to tease out which state/events lead to a failed assertion. I use the testWrapper function to add sections into the output bundle so I know which state/events lead to a set of assertions in the output.