-
Notifications
You must be signed in to change notification settings - Fork 124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tests for Tutorial #2702
Tests for Tutorial #2702
Conversation
src/test-helpers/createTestApp.tsx
Outdated
@@ -47,6 +47,9 @@ const createTestApp = async () => { | |||
}) | |||
} | |||
|
|||
/** Set up testing and mock document and window functions, and show tutorial. */ | |||
export const createTestAppWithTutorial = () => createTestApp({ tutorial: true }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this, to preserve beforeEach(createTestApp)
syntax
in proccess of fixing various errors, not much ready for review yet |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good overall, @snqb. Thank you for the PR!
Just a few initial questions.
@snqb -- I've started tinkering with this. It seems that regardless of whether we use
I also gave
This has me a bit perplexed, so I will take a look at it tomorrow with a fresh perspective. |
Hi @snqb, I've been playing around with this issue without much success yet. Have you had a chance to look into it further? |
I am really doubtful I can do more than I did, because this thing almost got me into a mental breakdown What I suggest is: maybe we just do redux-way for these kinds of cases, preserving the rest |
344061e
to
a100cb5
Compare
@@ -46,7 +46,7 @@ const Tutorial2StepContextViewOpen = () => { | |||
{TUTORIAL_CONTEXT1_PARENT[tutorialChoice]}" or "{TUTORIAL_CONTEXT2_PARENT[tutorialChoice]}" to show it again. | |||
</p> | |||
) : contextViewClosed ? ( | |||
<p>Oops, somehow the context view was closed. Select "Relationships".</p> | |||
<p>Oops, somehow the context view was closed. Select "{TUTORIAL_CONTEXT[tutorialChoice]}".</p> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it was a bug I've found while testing
.vscode/settings.json
Outdated
"[typescript]": { | ||
"editor.defaultFormatter": "esbenp.prettier-vscode" | ||
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
files keep registering as typescript
, not typescriptreact
is it fine we add typescript
too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. I haven't run into any issues myself with the current configuration, but I don't think it can hurt.
@@ -14,7 +14,6 @@ const TutorialNavigationButton = React.forwardRef< | |||
>(({ clickHandler, value, disabled = false, classes }, ref) => ( | |||
<a | |||
className={cx(anchorButtonRecipe({ variableWidth: true, smallGapX: true }), classes)} | |||
onClick={clickHandler} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's duplicated by fastClick
, which caused a skip of step I've painfully experienced multiple times while testing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, good catch. It doesn't make sense to have both.
concerning the failing test: beforeEach(async () => {
await createTestApp()
}) so the problem is with this case: it('disable isLoading after initialize', async () => {
expect(store.getState().isLoading).toBe(false)
})
/** Sets the Tutorial setting value. */
const tutorial = (state: State, { value }: { value?: boolean }) => {
return ({
...settings(state, {
key: 'Tutorial',
value: value ? 'On' : 'Off',
}),
// disable isLoading when dismissing the tutorial, since we can assume this is a new thoughtspace or the thoughtspace has already been loaded
isLoading: state.isLoading && value // <--- here `value` is `value?`, so it could leak
// it should be more of
isLoading: state.isLoading && value === true,
})
} but I haven't found at a glance a realistic way this could happen btw it really puzzles me that explicitly beforeEach(async () => {
await createTestApp()
}) |
Vitest passes a test context object to the The point-free style is nice, so I would suggest just dispatching Update: However, unless you emulate the user clicking "START TUTORIAL", you will have to explicitly set the tutorial to { type: 'tutorial', value: !!tutorial } And have this reflected in the parameter type. Usually I declare options objects like this: const createTestApp = async ({ tutorial } = { tutorial?: boolean } = {}) => { |
@snqb -- Thank you again for the PR, as well as the latest updates. Please let me know when this is ready for review again, and I'll be sure to take a look right away. |
@trevinhofmann hey, I think you can have a look. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, @snqb. Nice work! The updated tests cover all of the steps for the full tutorial (Parts I & II) with direct user interaction rather than manipulating the store. I've walked through each of the tests manually with the tutorial in the live app, and I'm satisfied with the test coverage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I think we have the right approach now!
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime }) | ||
|
||
/** Get last created thought in the document, used mostly after `user.keyboard('{Enter}')`. */ | ||
const lastThought = () => Array.from(document.querySelectorAll('[contenteditable="true"]')).at(-1)! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use [aria-label="thought"] [contenteditable]
or [data-editable="true"]
to be a bit more specific. There happen to be no other contenteditable's on the screen, but it's a bit safer to not assume that.
Even better would be to specifically select the empty thought.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even better would be to specifically select the empty thought.
I considered this during my previous review, but didn't mention it since the last thought and the empty thought were coincidentally always the same.
It would be nicer to explicitly select the thought to be edited (the empty thought), and easier to update in the future if we need to edit another thought during testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, I arrived at
/** Get last created empty thought in the document. Gets the last thought that is not empty. Mostly used after `user.keyboard('{Enter}')` to get the new thought. */
const lastThought = () =>
Array.from(document.querySelectorAll('[data-editable="true"]'))
.filter(it => !it.textContent)
.at(-1)!
I think it works. Perhaps, a more refined solution could be out there, but it works :D
I'd prefer to modify this thing and move out to test-helpers
at some point if we hit the wall, but not now.
ensureStep(TUTORIAL_STEP_FIRSTTHOUGHT_ENTER) | ||
expect(screen.getByText('Now type something. Anything will do.')).toBeInTheDocument() | ||
await user.type(lastThought(), 'my first thought') | ||
await user.keyboard('{Enter}') | ||
await act(vi.runOnlyPendingTimersAsync) | ||
}) | ||
|
||
it('step second thought - prompts to add another thought', async () => { | ||
expect(screen.getByText('Well done!')).toBeInTheDocument() | ||
expect(screen.getByText(/Try adding another thought/)).toBeInTheDocument() | ||
ensureStep(TUTORIAL_STEP_SECONDTHOUGHT) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actions should generally be performed in the test that depends on them. So instead of typing and hitting enter in one test and asserting the result in the next, move the typing and hitting enter to the next test where it is asserted.
As a general rule, a test should end with an expect
. Anything after that should probably belong to the next test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with this.
Some testing philosophies go further and recommend/require exactly one expect
statement at the end of each test, but I believe that might be an excessively narrow rule for these tests.
ensureStep(TUTORIAL_STEP_FIRSTTHOUGHT) | ||
expect(screen.getByText('Hit the Enter key to create a new thought.')).toBeInTheDocument() | ||
await user.keyboard('{Enter}') | ||
expect(screen.getByText('You did it!')).toBeInTheDocument() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Test actions look good and are very readable 👍
- `, | ||
) | ||
|
||
// Now I am going to show you how to add a thought within another thought. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is redundant and can safely be removed.
Likewise with similar redundant comments below.
expect(screen.getByText(/Hit the Delete key to delete the current blank thought/)).toBeInTheDocument() | ||
expect(screen.getByText(/Then hold the Ctrl key and hit the Enter key/)).toBeInTheDocument() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's stick with a single screen.getByText
for a given point in the tutorial. Testing the minimal unique text at the given step will give us effectively the same test coverage, but will be less maintenance if the tutorial text is changed in the future.
Likewise in similar occurrences below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@snqb Still needs attention
// we created a new thought on 3rd level, so we shift-tab our way back to root | ||
await user.keyboard('{Enter}') | ||
await user.keyboard('{Shift>}{Tab}{/Shift}') | ||
await user.keyboard('{Shift>}{Tab}{/Shift}') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't mind this, but it might be more natural to select a thought at the root and create the new thought from there with one enter
(and not use shift+tab
).
Thoughts, @raineorshine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I agree. Shift + Tab activates the Outdent command. It would be better to stick to simple navigation and New Thought commands for completing the tutorial. That way we can avoid coupling the tests to an unrelated command.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
totally makes sense, thanks
now it's
await user.click(screen.getByText('Home'))
await user.keyboard('{Enter}')
// skip tutorial | ||
{ type: 'tutorial', value: false }, | ||
// there are cases where we want to show tutorial on test runs, whilst mostly we don't | ||
{ type: 'tutorial', value: !!tutorial }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of using !!tutorial
to handle the undefined
cases, it would be nicer to set a default value above so that the tutorial
value is always a defined boolean.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI This is a small concession to allow point-free style in beforeEach
.
See: #2702 (comment)
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime }) | ||
|
||
/** Get last created empty thought in the document. Gets the last thought that is not empty. Mostly used after `user.keyboard('{Enter}')` to get the new thought. */ | ||
const lastThought = () => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's call this lastEmptyThought
to better reflect the new functionality.
// we should avoid using { delay: null }, and use jest.advanceTimersByTime instead | ||
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime }) | ||
|
||
/** Get last created empty thought in the document. Gets the last thought that is not empty. Mostly used after `user.keyboard('{Enter}')` to get the new thought. */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gets the last thought that is not empty.
Maybe a typo on this part?
await user.type(lastThought(), 'third thought') | ||
await user.keyboard('{Enter}') | ||
await act(vi.runOnlyPendingTimersAsync) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are still a few places where tests do not end in expect
.
See: #2702 (comment)
expect(screen.getByText(/Hit the Delete key to delete the current blank thought/)).toBeInTheDocument() | ||
expect(screen.getByText(/Then hold the Ctrl key and hit the Enter key/)).toBeInTheDocument() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@snqb Still needs attention
ok, I've rethought the thing and spent some time |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! I like the new structure. I think one expect
at the end of each test is very clean.
.at(-1)! | ||
|
||
describe('Tutorial 1', async () => { | ||
beforeEach(() => createTestApp({ tutorial: true })) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe I'm missing something, but if the tests are all run sequentially on a single app instance, wouldn't this need to be beforeAll
instead of beforeEach
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really have not much idea how to make it work with one beforeAll
, since if I don't create a new test app for each case, I'm left with empty-ish DOM
I think, the whole setup relies on re-creating an isolated testing env for each case, as it's done on every other test suite.
right now it's run with one db/store/state/whatever, but with cleaning up event listeners that keep attaching
but it's not one sequential run per se, though I'd argue it's a lot like reloading page at each step :D
I'm really not sure how to make it work with one beforeAll
, really
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your reply. I'd like to dig a little deeper into this. The tests are quite readable right now, but it's not clear what state they share between test blocks.
I really have not much idea how to make it work with one
beforeAll
, since if I don't create a new test app for each case, I'm left with empty-ish DOM
What do you mean by "empty-ish DOM"?
I think, the whole setup relies on re-creating an isolated testing env for each case, as it's done on every other test suite.
Are you saying that the current implementation recreates an isolated testing environment for each case? Because my understanding is that these tests depend on the current test step being preserved between tests.
right now it's run with one db/store/state/whatever, but with cleaning up event listeners that keep attaching
The event listeners only keep attaching because you remove them and re-attach them each time.
but it's not one sequential run per se, though I'd argue it's a lot like reloading page at each step :D
Is it a sequential run or not? Or some hybrid? I think for clarity we need to stick with one or the other, or more carefully document the unique combination of shared state between test runs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you mean by "empty-ish DOM"?
html
<body
data-browser="safari"
data-color-mode="dark"
data-device="desktop"
data-drag-in-progress="false"
data-native="false"
data-platform="other"
/>
Are you saying that the current implementation recreates an isolated testing environment for each case? Because my understanding is that these tests depend on the current test step being preserved between tests.
we have everything in db, store etc., but dom is empty if we don't call the createTestApp
on each test case
so the current approach, and me attaching and detaching events, is:
- we
createTestApp
, get correct dom state, built from store/db. - we run tests
- we remove event listeners created
Just runing a createTestApp
once does not work, and it's a problem of our general testing environment.
It's easy to demonstrate if we try to do this:
- we
createTestApp
- we run one test, where we create a thought "first"
- we should have "first" thought present in our dom
but it does not work like this at all.
the test suite below gets 2/2 green, but it should not.
describe.only('T_T', () => {
beforeAll(createTestApp)
it('we create "first" thought', async () => {
await user.keyboard('{Enter}')
await user.type(lastEmptyThought(), 'first')
await user.keyboard('{Enter}')
expect(exportContext(store.getState(), HOME_TOKEN)).toEqual(`<ul>
<li>__ROOT__
<ul>
<li>first</li>
<li></li>
</ul>
</li>
</ul>`)
expect(screen.getByText('first')).toBeInTheDocument()
})
it('we expect to see "first" thought', () => {
console.log(exportContext(store.getState(), HOME_TOKEN))
screen.debug(document.querySelector('.content')!) /* <-- prints this
<body
data-browser="safari"
data-color-mode="dark"
data-device="desktop"
data-drag-in-progress="false"
data-native="false"
data-platform="other"
/>
*/
expect(() => screen.getByText('first')).toThrow()
})
})
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe we should "fix" this, but will it break other tests?
i generally feel like this pr now achieves what we want it to, given the current testing environment
maybe we should create another issue for it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. I'd love to figure out why the DOM is empty. Without knowing the answer to that, it's hard to know what the right call is here.
However, since all the tests pass and they are quite readable, I'm okay with merging the current implementation so that we have a good checkpoint in the commit history. We can investigate further in a separate issue.
If you could address the small stylistic changes below, then I'd be happy to merge. Thanks!
it('we can continue to advanced tutorial', async () => { | ||
expect(screen.getByText('Learn more')).toBeInTheDocument() | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a contradiction here: If the user just exited the tutorial, then how could the "Learn more" button be visible?
This seems to rely on the tutorial inadvertently being re-launched on beforeEach
, which does not reflect real-world usage. We should make this more explicit, as it's not super obvious what's happening here, and closer to end-user behavior. I recommend one of the following options:
- Don't click "Play on my own" at all. Then the test can click "Learn more" just as the user would. (less test coverage)
- Click "Play on my own". Then navigate to the Help and start the second Tutorial as the user would. (more work, but more test coverage)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since I can do it, given the circumstances, I left it as is
at least it tests something that can break easily
and if we'll change our setup to have correct dom each time, this case will fail right away
(test of a test? xd)
beforeEach(() => createTestApp({ tutorial: true })) | ||
afterEach(cleanupTestEventHandlers) | ||
afterAll(cleanupTestApp) | ||
it('we learn about context indicators', async () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's more accurate to say "contexts" or "context view" than "context indicators". Let's update this to be more consistent with em terminology.
Likewise below.
expect(screen.getByText(/Add a thought with the text "To Do"/)).toBeInTheDocument() | ||
}) | ||
|
||
it('we add a Home->To Do subthought', async () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use Home/To Do
or Home • To Do
to indicate subthought relationships. That will be a bit more consistent with our internal documentation, since ->
is not used.
Likewise below.
I've updated the stylistics |
closes #2313
A
userEvents
-only run through the Tutorial.Assumes desktop, as relies on keyboard shortctuts.
Goes through most steps(excluding mostly hint steps), ensures those and faithfully replicates a user going through the instructions.
0 dispatches used.