Testcafe is taking screenshot on catched expect Errors

[ferencbeutel4711]

What is your Test Scenario?

We sometimes have to catch the error thrown by testcafe.expect, for example when we want to verify a process involving an asynchronous backend task. In that case we try to validate the existance of an element on the page and if not found, catch the Error, wait some time, reload the page and assert again. This works regarding the test result, so at the end if the element is found eventually, even though some of the testcafe.expects did not succeed, the test is green, which is also our expectation.
We have also activated the screenshot-option "takeOnFails". To our surprise, enabling this option leads to screenshots being taken, even if the exception thrown by the testcafe.expect call is catched.
If our approach is not optimal regarding catching the Exception, please guide us to a way that is more appropriate.

What is the Current behavior?

a catched testcafe.expect Exception leads to a screenshot being taken if the option "takeOnFails" is true

What is the Expected behavior?

screenshots are only taken for failed tests

Your Environment details:

• testcafe version: 1.8.2
• node.js version: 10.17.0
• browser name and version: tested in chrome latest version
• platform and version: happened under windows, linux and osx

[ferencbeutel4711]

Also, please note that the screenshot behavior is documented as:
Specifies if screenshots should be taken automatically when a test fails.
So a failed(and caught) expecation does not lead to a failed test, therefore the documentation also does not match the current implementation

[miherlosev]

Hi @ferencbeutel4711

Thank you for your input. We need additional time to research your case and find the most suitable solution. We will update this issue when we have any news.

[Shreky]

+1
This enforce me to continue using older testcafe version (1.8).
@miherlosev can you share any estimation for this to be fixed?

[miherlosev]

I'm unable to provide you with any time frame as any personal estimate may be misleading. Once we get any results, we will update this thread.

[cattermo]

This is super annoying! Makes it hard to find the real screenshots since so many fake (from non failing tests) ones are created.

[AndreyBelym]

I apologize for the inconvenience caused by this bug. I increased the bug level, so we will give it a higher priority when planning the next development sprint.

However, if the issue is caused by a failed assertion that checks for element existence, you can use the Selector.exists property instead of suppressing the assertion error. @cattermo, @Shreky could you please share your error suppressing cases as well?

For the team:
The issue can be reproduced with the following test:

import fs from 'fs';

fixture `Fixture`;

test('Test', async t => {
    if(!fs.existsSync('screenshots'))
        fs.mkdirSync('screenshots');
    
    try {
        await t.expect(false).ok();
    }
    catch (e) {
    }

    await t.expect(fs.readdirSync('screenshots').length).eql(0);
});

It is caused by the explicit screenshot capture call, which didn't exist in previous versions:
https://github.com/DevExpress/testcafe/blame/5ee0c0d9b620d542746d2fbc752fd9736b46a17e/src/test-run/index.js

[cattermo]

I apologize for the inconvenience caused by this bug. I increased the bug level, so we will give it a higher priority when planning the next development sprint.

However, if the issue is caused by a failed assertion that checks for element existence, you can use the Selector.exists property instead of suppressing the assertion error. @cattermo, @Shreky could you please share your error suppressing cases as well?

For the team:
The issue can be reproduced with the following test:

import fs from 'fs';

fixture `Fixture`;

test('Test', async t => {
    if(!fs.existsSync('screenshots'))
        fs.mkdirSync('screenshots');
    
    try {
        await t.expect(false).ok();
    }
    catch (e) {
    }

    await t.expect(fs.readdirSync('screenshots').length).eql(0);
});

It is caused by the explicit screenshot capture call, which didn't exist in previous versions:
https://github.com/DevExpress/testcafe/blame/5ee0c0d9b620d542746d2fbc752fd9736b46a17e/src/test-run/index.js

I ended up rebuilding the test so no error will be thrown but I don't know if the new solution will be stable. I couldn't get it to wait in the same way.

My use case is a button that might be disabled but might also loose the disabled attribute if we wait a little. If it keeps the disabled attribute it should fail unless some other condition is true (which is why we caught the error previously). Now I check for the disabled attribute instead but the hasAttribute will not wait so it's not the same as previously.

[AlexKamaev]

@cattermo
Thank you for the additional information. Your scenario is clear. So, as @AndreyBelym mentioned, we increased the bug's priority level. This means that we'll prioritize the issue in our next development sprints.

[Makavelic]

FYI, this is also an issue when skipjserrors is set to true. If a test passes without any errors, a screenshot still gets generated if there are browser console errors. Additionally, when a test does fail, sometimes it only includes the screenshot taken when a js error occurs, not when the assertion fails. This makes it very difficult to troubleshoot test failures.

[AlexKamaev]

@Makavelic
I was not able to reproduce the issue. When the --skip-js-errors option is enabled, TestCafe does not make screenshots in my example. Here is my HTML and test code:
HTML:

<!DOCTYPE html>
<html>
<head>
</head>
<body>
<button>click me</button>

<script>
    document.querySelector('button').addEventListener('click', () => {
        throw new Error('error');
    });
</script>
</body>
</html>

Test code:

fixture('screehshots and js errors')
    .page `..\\pages\\index.html`;

test('Log in to Paypal', async t => {
    await t.click('button');
});

I use the following command to run TestCafe: testcafe chrome test.js --skip-js-errors -s takeOnFails=true
Could you please modify my example to reproduce the issue?

Additionally, when a test does fail, sometimes it only includes the screenshot taken when a js error occurs, not when the assertion fails

This behavior seems correct. If the page has JS error and there is no --skip-js-error option, the test will fail at this stage since the page state is already incorrect.

[Makavelic]

@AlexKamaev
The issue only occurs when you navigate away from the original page. For example:
test1.html

<!DOCTYPE html>
<html>
<head>
</head>
<body>
<button>click me</button>
<a href="./test2.html">test2</a>
<script>
    document.addEventListener("DOMContentLoaded", function() {
	throw new Error("error loading");
});
</script>
</body>
</html>

test2.html

<!DOCTYPE html>
<html>
<head>
</head>
<body>
<button>new page</button>

<script>
    document.addEventListener("DOMContentLoaded", function() {
	throw new Error("error loading");
});
</script>
</body>
</html>

test

import { t } from "testcafe";


fixture("errorFixture")
	.page("test1.html")
	

test("testError", async () => {
    await t.click("a");
    await t.wait(3000);
});

Note: You'll need to update the paths, I was just using hard coded paths which I removed here.

[AlexKamaev]

@Makavelic
I was still unable to reproduce the issue following your instructions. If I enable the --skip-js-errors option, the screenshots are not taken. If I do not use the --skip-js-errors option, the screenshot of the first page is taken because the error occurs inside the 'DOMContentLoaded' event handler of the first page.

I am using the following command to run tests: testcafe chrome test.js --skip-js-errors -s takeOnFails=true

My TestCafe version is 1.9.4.

Could you please specify which command you are using to run tests? Please also specify the browser, OS, and the TestCafe version.
If you have some additional instructions on how to reproduce the issue, please share them with us.

[Makavelic]

@AlexKamaev, I'm using a custom runner, here are the relevant parts:

.screenshots({ 
                takeOnFails: true,
                path: paramMap.get(ParamNames.PathToScreenshots),
                pathPattern: '${TEST}/${DATE}_${TIME}.png'})
            .run({skipJsErrors: true});
            

Testcafe: 1.9.4
Chrome: 86

Screenshots are taken on all of our tests even when they do not fail. This did not occur in earlier versions of testcafe.

[AlexKamaev]

@Makavelic
I created a sample based on your runner and your test files. However, I was not able to reproduce the issue. Here is my project: https://gist.github.com/AlexKamaev/24bd42cc8a844d7f7c3f3acfb0f6c1ea
Please specify how I should modify it to reproduce the issue.

[Makavelic]

@Makavelic
I created a sample based on your runner and your test files. However, I was not able to reproduce the issue. Here is my project: https://gist.github.com/AlexKamaev/24bd42cc8a844d7f7c3f3acfb0f6c1ea
Please specify how I should modify it to reproduce the issue.

I can no longer reproduce using my test pages. It's still consistent when testing my companies actual product though. It happens even when the dev console doesn't show any errors (only warnings). Now that that we improved our logging we're also seeing logs indicating that screenshots are being overwritten (so multiple screenshots are generated during a failed test).

The file at already exists. It has just been rewritten with a recent screenshot. This situation can possibly cause issues. To avoid them, make sure that each screenshot has a unique path. If a test runs in multiple browsers, consider including the user agent in the screenshot path or generate a unique identifier in another way.

This occurs even though we're using pathPattern: '${TEST}/${DATE}_${TIME}.png'}) and not running tests concurrently.

[wentwrong]

Unfortunately, there's not much we can say here without a reproducible example. As for screenshots that are being overwritten, you can try to add the ${FILE_INDEX} placeholder to your "pathPattern" option and see if it helps.

[Makavelic]

Unfortunately, there's not much we can say here without a reproducible example. As for screenshots that are being overwritten, you can try to add the ${FILE_INDEX} placeholder to your "pathPattern" option and see if it helps.

Is there a way to get more logging from testcafe to hopefully gather more info that would be useful? For whatever reason, testcafe takes a screenshot after logging into the website its testing.

[wentwrong]

There is a guide in our docs, which describes how to debug TestCafe tests: Debug. Manual step-by-step execution should help to identify a particular test action or assertion after which the screenshot is taken.

[miherlosev]

Hi folks,

The problem occurs because you use an assertion for checking element visibility or existence.
It's incorrect. Instead of that you need to use the Selector.visible, Selector.exists, or something else depending on your scenario.
Also do not catch assertion errors during test execution. It's not a recommended practice. If the test logic depends on some Selector values, retrieve them in the test and use them later.
Also, we will add a note about capturing screenshots for caught assertion errors.

If you have some problems getting rid of caught assertion errors, please describe them in separate issues. We will investigate them separately.

[testcafe-HK]

Hi

issue is closed but I am
Getting screenshot while test is passed . I am using Testcafe 2.5 version . Every time after login it takes screen shot and test passed . When same test failed due to selector got changed it never takes the screenshot of new page .it always showing same screenshot.

{
takeOnFails: true,
path: paramMap.get(ParamNames.PathToScreenshots),
pathPattern: '${TEST}/${DATE}_${TIME}.png'})
.run({skipJsErrors: true});

Any help will
Save me

[Artem-Babich]

Hi @testcafe-HK ,
It is difficult to determine the cause of the issue you encountered without being able to reproduce it locally. If I understand correctly, it is not related to the current topic. Please create a separate GitHub issue (Bug Report) and share a simple working project with exact steps to reproduce the issue.