Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests for Tutorial #2702

Merged
merged 22 commits into from
Jan 19, 2025
Merged

Tests for Tutorial #2702

merged 22 commits into from
Jan 19, 2025

Conversation

snqb
Copy link
Collaborator

@snqb snqb commented Dec 15, 2024

closes #2313

A userEvents-only run through the Tutorial.
Assumes desktop, as relies on keyboard shortctuts.
Goes through most steps(excluding mostly hint steps), ensures those and faithfully replicates a user going through the instructions.
0 dispatches used.

{C0A862A8-D2FD-46F0-8491-5A8911420188}

@@ -47,6 +47,9 @@ const createTestApp = async () => {
})
}

/** Set up testing and mock document and window functions, and show tutorial. */
export const createTestAppWithTutorial = () => createTestApp({ tutorial: true })
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this, to preserve beforeEach(createTestApp) syntax

@snqb
Copy link
Collaborator Author

snqb commented Dec 16, 2024

in proccess of fixing various errors, not much ready for review yet
will be soon, though

@snqb snqb marked this pull request as ready for review December 16, 2024 21:07
@trevinhofmann trevinhofmann self-requested a review December 17, 2024 00:21
@trevinhofmann trevinhofmann self-assigned this Dec 17, 2024
Copy link
Collaborator

@trevinhofmann trevinhofmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good overall, @snqb. Thank you for the PR!

Just a few initial questions.

src/components/Tutorial/__tests__/Tutorial.ts Outdated Show resolved Hide resolved
src/components/Tutorial/__tests__/Tutorial.ts Outdated Show resolved Hide resolved
src/components/Tutorial/__tests__/Tutorial.ts Outdated Show resolved Hide resolved
@trevinhofmann trevinhofmann assigned snqb and unassigned trevinhofmann Dec 19, 2024
@snqb
Copy link
Collaborator Author

snqb commented Dec 21, 2024

Hey, I've added some showcase of keyboard issues I've got.
Apparently, it's kind of tricky to cleanly create thoughts via userEvent.
We get insane results like this tree(got it via exportContext) after user.keyboard and user.type manipulations.
{9C748D62-8535-4102-A643-0E23E38BA25B}

You can just run the tests via
yarn vitest src/components/Tutorial/ --project unit and tinker around with what's going on.

Below we have an almost-working test suite for the tutorial, where I'm stuck with this keyboard behavior, so you might want to have a look there, removing the .only from the first describe.

{1D445E4D-613B-4FDA-A8FF-96CB9EDE24BC}

@trevinhofmann
Copy link
Collaborator

@snqb -- I've started tinkering with this. It seems that regardless of whether we use user.keyboard or user.type for the enter key, it's creating two thoughts:

(before)
- __ROOT__

(after)
- __ROOT__
  - 
  -

I also gave fireEevent.keyPress a try, but it doesn't seem to create a new thought at all:

(before)
- __ROOT__

(after)
- __ROOT__

This has me a bit perplexed, so I will take a look at it tomorrow with a fresh perspective.

@trevinhofmann trevinhofmann self-assigned this Dec 24, 2024
@trevinhofmann
Copy link
Collaborator

Hi @snqb, I've been playing around with this issue without much success yet. Have you had a chance to look into it further?

@snqb
Copy link
Collaborator Author

snqb commented Dec 28, 2024

I am really doubtful I can do more than I did, because this thing almost got me into a mental breakdown

What I suggest is: maybe we just do redux-way for these kinds of cases, preserving the rest userEvents style, and make a new issue on investigating why we have such problems with creating thoughts. And also, it generally feels like something worth a separate issue.
Thus, I could just fix this and, I believe one more issue, and we're good to mark this complete.

@trevinhofmann trevinhofmann removed their assignment Jan 1, 2025
@snqb snqb force-pushed the tests-for-gestures branch from 344061e to a100cb5 Compare January 12, 2025 08:52
@@ -46,7 +46,7 @@ const Tutorial2StepContextViewOpen = () => {
{TUTORIAL_CONTEXT1_PARENT[tutorialChoice]}" or "{TUTORIAL_CONTEXT2_PARENT[tutorialChoice]}" to show it again.
</p>
) : contextViewClosed ? (
<p>Oops, somehow the context view was closed. Select "Relationships".</p>
<p>Oops, somehow the context view was closed. Select "{TUTORIAL_CONTEXT[tutorialChoice]}".</p>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was a bug I've found while testing

Comment on lines 11 to 13
"[typescript]": {
"editor.defaultFormatter": "esbenp.prettier-vscode"
},
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

files keep registering as typescript, not typescriptreact

is it fine we add typescript too?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I haven't run into any issues myself with the current configuration, but I don't think it can hurt.

@@ -14,7 +14,6 @@ const TutorialNavigationButton = React.forwardRef<
>(({ clickHandler, value, disabled = false, classes }, ref) => (
<a
className={cx(anchorButtonRecipe({ variableWidth: true, smallGapX: true }), classes)}
onClick={clickHandler}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's duplicated by fastClick, which caused a skip of step I've painfully experienced multiple times while testing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good catch. It doesn't make sense to have both.

@snqb
Copy link
Collaborator Author

snqb commented Jan 12, 2025

concerning the failing test:
it seems to be resolved if we manually await createTestApp, but I'm not really sure if we wanna do this

beforeEach(async () => {
  await createTestApp()
})

so the problem is with this case:

it('disable isLoading after initialize', async () => {
  expect(store.getState().isLoading).toBe(false)
})

isLoading keeps being undefined, while I haven't found a single use in our app where undefined is kinda possible via assignment, though it's possible judging from types in tutorial reducer.

/** Sets the Tutorial setting value. */
const tutorial = (state: State, { value }: { value?: boolean }) => {
  return ({
    ...settings(state, {
      key: 'Tutorial',
      value: value ? 'On' : 'Off',
    }),
    // disable isLoading when dismissing the tutorial, since we can assume this is a new thoughtspace or the thoughtspace has already been loaded
    isLoading: state.isLoading && value // <--- here `value` is `value?`, so it could leak 
    // it should be more of 
    isLoading: state.isLoading && value === true,
  })
}

but I haven't found at a glance a realistic way this could happen
so I judged, it's a create test thing

btw it really puzzles me that explicitly await-ing does lead to a different result
I assumed it kinda handles promises, as it works with all other cases.

beforeEach(async () => {
  await createTestApp()
})

@raineorshine
Copy link
Contributor

raineorshine commented Jan 12, 2025

Vitest passes a test context object to the beforeEach callback when written in point-free style: beforeEach(createTestApp). It was being ignored before, but now with the options object that was added, tutorial gets accidentally set to undefined. This gets passed down to isLoading through the tutorial action.

The point-free style is nice, so I would suggest just dispatching false explicitly when the tutorial should be closed.

Update: However, unless you emulate the user clicking "START TUTORIAL", you will have to explicitly set the tutorial to true.

      { type: 'tutorial', value: !!tutorial }

And have this reflected in the parameter type. Usually I declare options objects like this:

const createTestApp = async ({ tutorial } = { tutorial?: boolean } = {}) => {

@trevinhofmann
Copy link
Collaborator

@snqb -- Thank you again for the PR, as well as the latest updates. Please let me know when this is ready for review again, and I'll be sure to take a look right away.

@snqb
Copy link
Collaborator Author

snqb commented Jan 14, 2025

@trevinhofmann hey, I think you can have a look.
I'm pretty satisfied with the final code, if we set the goal to have a very straightforward user-events based walkthrough.

@trevinhofmann trevinhofmann assigned trevinhofmann and unassigned snqb Jan 15, 2025
Copy link
Collaborator

@trevinhofmann trevinhofmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @snqb. Nice work! The updated tests cover all of the steps for the full tutorial (Parts I & II) with direct user interaction rather than manipulating the store. I've walked through each of the tests manually with the tutorial in the live app, and I'm satisfied with the test coverage.

Copy link
Contributor

@raineorshine raineorshine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we have the right approach now!

src/components/Tutorial/__tests__/Tutorial.ts Outdated Show resolved Hide resolved
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime })

/** Get last created thought in the document, used mostly after `user.keyboard('{Enter}')`. */
const lastThought = () => Array.from(document.querySelectorAll('[contenteditable="true"]')).at(-1)!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use [aria-label="thought"] [contenteditable] or [data-editable="true"] to be a bit more specific. There happen to be no other contenteditable's on the screen, but it's a bit safer to not assume that.

Even better would be to specifically select the empty thought.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better would be to specifically select the empty thought.

I considered this during my previous review, but didn't mention it since the last thought and the empty thought were coincidentally always the same.

It would be nicer to explicitly select the thought to be edited (the empty thought), and easier to update in the future if we need to edit another thought during testing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I arrived at

/** Get last created empty thought in the document. Gets the last thought that is not empty. Mostly used after `user.keyboard('{Enter}')` to get the new thought. */
const lastThought = () =>
  Array.from(document.querySelectorAll('[data-editable="true"]'))
    .filter(it => !it.textContent)
    .at(-1)!

I think it works. Perhaps, a more refined solution could be out there, but it works :D
I'd prefer to modify this thing and move out to test-helpers at some point if we hit the wall, but not now.

Comment on lines 64 to 74
ensureStep(TUTORIAL_STEP_FIRSTTHOUGHT_ENTER)
expect(screen.getByText('Now type something. Anything will do.')).toBeInTheDocument()
await user.type(lastThought(), 'my first thought')
await user.keyboard('{Enter}')
await act(vi.runOnlyPendingTimersAsync)
})

it('step second thought - prompts to add another thought', async () => {
expect(screen.getByText('Well done!')).toBeInTheDocument()
expect(screen.getByText(/Try adding another thought/)).toBeInTheDocument()
ensureStep(TUTORIAL_STEP_SECONDTHOUGHT)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actions should generally be performed in the test that depends on them. So instead of typing and hitting enter in one test and asserting the result in the next, move the typing and hitting enter to the next test where it is asserted.

As a general rule, a test should end with an expect. Anything after that should probably belong to the next test.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this.

Some testing philosophies go further and recommend/require exactly one expect statement at the end of each test, but I believe that might be an excessively narrow rule for these tests.

Comment on lines 59 to 62
ensureStep(TUTORIAL_STEP_FIRSTTHOUGHT)
expect(screen.getByText('Hit the Enter key to create a new thought.')).toBeInTheDocument()
await user.keyboard('{Enter}')
expect(screen.getByText('You did it!')).toBeInTheDocument()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test actions look good and are very readable 👍

src/components/Tutorial/__tests__/Tutorial.ts Outdated Show resolved Hide resolved
- `,
)

// Now I am going to show you how to add a thought within another thought.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is redundant and can safely be removed.

Likewise with similar redundant comments below.

Comment on lines 103 to 104
expect(screen.getByText(/Hit the Delete key to delete the current blank thought/)).toBeInTheDocument()
expect(screen.getByText(/Then hold the Ctrl key and hit the Enter key/)).toBeInTheDocument()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's stick with a single screen.getByText for a given point in the tutorial. Testing the minimal unique text at the given step will give us effectively the same test coverage, but will be less maintenance if the tutorial text is changed in the future.

Likewise in similar occurrences below.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snqb Still needs attention

src/components/Tutorial/__tests__/Tutorial.ts Outdated Show resolved Hide resolved
@trevinhofmann trevinhofmann assigned snqb and unassigned raineorshine Jan 16, 2025
Comment on lines 220 to 223
// we created a new thought on 3rd level, so we shift-tab our way back to root
await user.keyboard('{Enter}')
await user.keyboard('{Shift>}{Tab}{/Shift}')
await user.keyboard('{Shift>}{Tab}{/Shift}')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind this, but it might be more natural to select a thought at the root and create the new thought from there with one enter (and not use shift+tab).

Thoughts, @raineorshine?

Copy link
Contributor

@raineorshine raineorshine Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree. Shift + Tab activates the Outdent command. It would be better to stick to simple navigation and New Thought commands for completing the tutorial. That way we can avoid coupling the tests to an unrelated command.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

totally makes sense, thanks
now it's

await user.click(screen.getByText('Home'))
await user.keyboard('{Enter}')

// skip tutorial
{ type: 'tutorial', value: false },
// there are cases where we want to show tutorial on test runs, whilst mostly we don't
{ type: 'tutorial', value: !!tutorial },
Copy link
Collaborator

@trevinhofmann trevinhofmann Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using !!tutorial to handle the undefined cases, it would be nicer to set a default value above so that the tutorial value is always a defined boolean.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI This is a small concession to allow point-free style in beforeEach.

See: #2702 (comment)

@snqb snqb requested a review from raineorshine January 17, 2025 13:19
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime })

/** Get last created empty thought in the document. Gets the last thought that is not empty. Mostly used after `user.keyboard('{Enter}')` to get the new thought. */
const lastThought = () =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call this lastEmptyThought to better reflect the new functionality.

// we should avoid using { delay: null }, and use jest.advanceTimersByTime instead
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime })

/** Get last created empty thought in the document. Gets the last thought that is not empty. Mostly used after `user.keyboard('{Enter}')` to get the new thought. */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gets the last thought that is not empty.

Maybe a typo on this part?

Comment on lines 54 to 56
await user.type(lastThought(), 'third thought')
await user.keyboard('{Enter}')
await act(vi.runOnlyPendingTimersAsync)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still a few places where tests do not end in expect.

See: #2702 (comment)

Comment on lines 103 to 104
expect(screen.getByText(/Hit the Delete key to delete the current blank thought/)).toBeInTheDocument()
expect(screen.getByText(/Then hold the Ctrl key and hit the Enter key/)).toBeInTheDocument()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snqb Still needs attention

@snqb
Copy link
Collaborator Author

snqb commented Jan 18, 2025

ok, I've rethought the thing and spent some time
but now almost everywhere we have one-expect-per suite
and it's kinda cleaner, pls have a look
I think it's all resolved

@snqb snqb requested a review from raineorshine January 18, 2025 23:09
Copy link
Contributor

@raineorshine raineorshine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I like the new structure. I think one expect at the end of each test is very clean.

.at(-1)!

describe('Tutorial 1', async () => {
beforeEach(() => createTestApp({ tutorial: true }))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm missing something, but if the tests are all run sequentially on a single app instance, wouldn't this need to be beforeAll instead of beforeEach?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really have not much idea how to make it work with one beforeAll, since if I don't create a new test app for each case, I'm left with empty-ish DOM
I think, the whole setup relies on re-creating an isolated testing env for each case, as it's done on every other test suite.

right now it's run with one db/store/state/whatever, but with cleaning up event listeners that keep attaching
but it's not one sequential run per se, though I'd argue it's a lot like reloading page at each step :D

I'm really not sure how to make it work with one beforeAll, really

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your reply. I'd like to dig a little deeper into this. The tests are quite readable right now, but it's not clear what state they share between test blocks.

I really have not much idea how to make it work with one beforeAll, since if I don't create a new test app for each case, I'm left with empty-ish DOM

What do you mean by "empty-ish DOM"?

I think, the whole setup relies on re-creating an isolated testing env for each case, as it's done on every other test suite.

Are you saying that the current implementation recreates an isolated testing environment for each case? Because my understanding is that these tests depend on the current test step being preserved between tests.

right now it's run with one db/store/state/whatever, but with cleaning up event listeners that keep attaching

The event listeners only keep attaching because you remove them and re-attach them each time.

but it's not one sequential run per se, though I'd argue it's a lot like reloading page at each step :D

Is it a sequential run or not? Or some hybrid? I think for clarity we need to stick with one or the other, or more carefully document the unique combination of shared state between test runs.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by "empty-ish DOM"?

html
<body
  data-browser="safari"
  data-color-mode="dark"
  data-device="desktop"
  data-drag-in-progress="false"
  data-native="false"
  data-platform="other"
/>

Are you saying that the current implementation recreates an isolated testing environment for each case? Because my understanding is that these tests depend on the current test step being preserved between tests.

we have everything in db, store etc., but dom is empty if we don't call the createTestApp on each test case
so the current approach, and me attaching and detaching events, is:

  1. we createTestApp, get correct dom state, built from store/db.
  2. we run tests
  3. we remove event listeners created

Just runing a createTestApp once does not work, and it's a problem of our general testing environment.
It's easy to demonstrate if we try to do this:

  1. we createTestApp
  2. we run one test, where we create a thought "first"
  3. we should have "first" thought present in our dom

but it does not work like this at all.
the test suite below gets 2/2 green, but it should not.

describe.only('T_T', () => {
  beforeAll(createTestApp)

  it('we create "first" thought', async () => {
    await user.keyboard('{Enter}')
    await user.type(lastEmptyThought(), 'first')
    await user.keyboard('{Enter}')

    expect(exportContext(store.getState(), HOME_TOKEN)).toEqual(`<ul>
  <li>__ROOT__  
    <ul>
      <li>first</li>
      <li></li>
    </ul>
  </li>
</ul>`)
    expect(screen.getByText('first')).toBeInTheDocument()
  })

  it('we expect to see "first" thought', () => {
    console.log(exportContext(store.getState(), HOME_TOKEN))

    screen.debug(document.querySelector('.content')!) /* <-- prints this
    <body
      data-browser="safari"
      data-color-mode="dark"
      data-device="desktop"
      data-drag-in-progress="false"
      data-native="false"
      data-platform="other"
    />
    */

    expect(() => screen.getByText('first')).toThrow()
  })
})

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should "fix" this, but will it break other tests?

i generally feel like this pr now achieves what we want it to, given the current testing environment
maybe we should create another issue for it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I'd love to figure out why the DOM is empty. Without knowing the answer to that, it's hard to know what the right call is here.

However, since all the tests pass and they are quite readable, I'm okay with merging the current implementation so that we have a good checkpoint in the commit history. We can investigate further in a separate issue.

If you could address the small stylistic changes below, then I'd be happy to merge. Thanks!

Comment on lines +114 to +116
it('we can continue to advanced tutorial', async () => {
expect(screen.getByText('Learn more')).toBeInTheDocument()
})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a contradiction here: If the user just exited the tutorial, then how could the "Learn more" button be visible?

This seems to rely on the tutorial inadvertently being re-launched on beforeEach, which does not reflect real-world usage. We should make this more explicit, as it's not super obvious what's happening here, and closer to end-user behavior. I recommend one of the following options:

  1. Don't click "Play on my own" at all. Then the test can click "Learn more" just as the user would. (less test coverage)
  2. Click "Play on my own". Then navigate to the Help and start the second Tutorial as the user would. (more work, but more test coverage)

Copy link
Collaborator Author

@snqb snqb Jan 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since I can do it, given the circumstances, I left it as is
at least it tests something that can break easily
and if we'll change our setup to have correct dom each time, this case will fail right away

(test of a test? xd)

beforeEach(() => createTestApp({ tutorial: true }))
afterEach(cleanupTestEventHandlers)
afterAll(cleanupTestApp)
it('we learn about context indicators', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more accurate to say "contexts" or "context view" than "context indicators". Let's update this to be more consistent with em terminology.

Likewise below.

expect(screen.getByText(/Add a thought with the text "To Do"/)).toBeInTheDocument()
})

it('we add a Home->To Do subthought', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use Home/To Do or Home • To Do to indicate subthought relationships. That will be a bit more consistent with our internal documentation, since -> is not used.

Likewise below.

@snqb
Copy link
Collaborator Author

snqb commented Jan 19, 2025

I've updated the stylistics

@raineorshine raineorshine merged commit 8126692 into cybersemics:main Jan 19, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tutorial: Test completion of tutorial
3 participants