Question about: Some test suites likely terminated without passing on results

[aram-bigid]

Hi,

First off, thanks for the repo and work :)

So i want to use this package in our tests, but am seeing the following error randomly on my test runs:
Some test suites likely terminated without passing on results

I know that it's a validation, but do we know why this is happening?

Thanks,
-arnon

[mahanthi-arun-kumar]

I am also facing a similar kind of issue,
Found test suites does not match results.
Test suites found: 1036
Test suite results: 44
Some test suites likely terminated without passing on results, exit with error

This is the error I am getting:
npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! ui@0.1.0 cypress:parallel: cypress-parallel -s cypress:run -t 2 -d './src' -a '"--config baseUrl=http://localhost:3000 integrationFolder=src testFiles=**/integration-test/*cypress.js" '`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the ui@0.1.0 cypress:parallel script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/maarunkumar/.npm/_logs/2022-03-26T05_14_05_325Z-debug.log`

[ronit15]

I am also facing same issue any updates?

[Oryhara]

Mahanthi-arun-kumar, I believe your problem is in the command line for cypress-parallel.
"-d './src'" tells cypress-parallel to look in your src directory for cypress test suite files. cypress-parallel treats every file it finds as a test suite file. Hence "Test suites found: 1036" that is every code file in your src directory. in your -a arguments passed to cypress itself, you declare testFiles=**/integration-test/*cypress.js" which tells cypress where to find test suite files. further, cypress can distinguish between files that contain tests and files which do not. This accounts for Test suite results: 44 You likely only have 44 test suites.
But cypress-parallel thinks you have 1036 test suites, and so expects 1036 test suite results. Upon getting less than 1036 results, it throws an error.
If you only pass directories which contain only cypress test suite files to cypress-parallel, that should fix your problem. I ran into this same issue and that fixed it for me.

[olga-govor]

I passed correct tests directory with correct regexp and in 95% tests pass without any issues. But sometimes randomly I've got results like
Found test suites does not match results. Test suites found: 71 Test suite results: 61 Some test suites likely terminated without passing on results, exit with error

It mentions specific suites that were not found, but I see results for these suites. Any ideas why it could happen?

[Densf2]

Also tests failed with the same message in console ^^ - if launch by specific tag.

[chojnicki]

@tnicola is this package abandoned? Maybe someone has working fork?

This script should using glob/pattern for spec files, there is no reason to do this differently than Cypress itself. I'm using Cypress for component testing, and like many other people I have spec files together with component files. All my specs are passing, but I'm still getting false error from cypress-parallel :/

[SaniaSid]

Any update on this one ? Facing this issue and there is no information on why this is happening

[prashantpundhir]

I am also facing similar issue, but for me, results shows only 0 even though tests are running successfully.
Any updates on this ticket?

[ezunigaf90]

Has anyone managed to solve/bypass this issue?

Still getting this issue on version 0.13.0

[olga-govor]

For us it was an issue with different threads that used the same results folder. After small update to put results in different folders for different threads (unfortunately it's hardcoded in source lib) we didn't meet these problem. For now 2nd month without this issue

[ezunigaf90]

Since we don't get answers here I've been digging through the code because I think this plugin is really nice and I think I found what the issue was for us...

First of all, the error I was getting is:

Found test suites does not match results. 
Test suites found: 68 
Test suite results: 0 
Some test suites likely terminated without passing on results, exit with error
The following tests suites are missing results: { test1, test2, ... all 68 files }

So, it was false failing even if all the tests passed and the statement triggering the error is this:

cypress-parallel/lib/cli.js

Line 99 in 710dd19

if (settings.strictMode && timeMap.size !== testSuitePaths.length) {

To make shorter the story, after debugging it I realized that for "some" unknown reason none of the tests was generating the .json file needed and created automatically by this plugin in order to output the results, it also creates a runner-results folder for that but basically it was empty after any execution and that's why Test suite results: 0 (timeMap.size). I continued debugging and found where this issue started to happen:

We have mochawesome, mocha-junit-reporter and cypress-multi-reporters, so our code looked like this:

cypress v10.11
mochawesome v6.2.2
cypress-multi-reporters v1.6.1
mocha-junit-reporter v2.0.2
cypress-parallel v0.13.0

Piece of our cm-desktop.config.js file
module.exports = defineConfig({
	e2e: {
	        specPattern: 'cypress/user/*.test.js',
		reporter: 'cypress-multi-reporters',
		reporterOptions: {
			configFile: 'cypress/config/user/reporter-config.json',
		}
	}
});

Our reporter-config.json file
{
  "reporterEnabled":"mochawesome, mocha-junit-reporter",
  "mochaJunitReporterReporterOptions":{
    "mochaFile":"cypress/results/user/dsk/user-[hash].xml"
  },
  "mochawesomeReporterOptions":{
    "reportDir":"cypress/results/user/dsk",
    "overwrite":false,
    "html":false,
    "json":true
  }
}

Our package.json scripts
"cm:dsk:user": "cypress run --config-file ./cypress/config/user/cm-desktop.config.js --browser chrome",
"cm:run:dsk:user:parallel": "cypress-parallel -s cm:dsk:user -t 4 -d ./cypress/user",

^ That way, I was getting the error... So, after almost a full day of trial and error, I found that when I was executing this way then the multi-reporter-config.json file generated by default by the plugin was writing this lines:

{
  "reporterEnabled": "cypress-parallel/json-stream.reporter.js, cypress-parallel/simple-spec.reporter.js"
}

So, it immediately made me think that my reporter and reporterOptions flag was not being read... why? I don't know the answer, I thought it should have worked but looks like it's not working in that way, maybe a bug?? anyway... this statement confirmed my theory:

cypress-parallel/lib/thread.js

Line 36 in 710dd19

if (settings.reporter) {

effectively it was not recognizing my reporting flags.

Note: I tried passing -r and -o and a lot of ways/combinations but can't make it work.

The solution? ... I made it work by doing this:

1. Removed the reporter and reporterOptions flags from the cm-desktop.config.js file (yes, deleted those lines from there)
2. Our reporter-config.json file now looks like this:

{
  "reporterEnabled":"mochawesome, cypress-parallel/json-stream.reporter.js, cypress-parallel/simple-spec.reporter.js, mocha-junit-reporter",
    "mochawesomeReporterOptions":{
      "reportDir":"cypress/results/user/dsk",
      "overwrite":false,
      "html":false,
      "json":true
    },
  "cypressParallelSimpleSpecReporterJsReporterOptions": {
    "reportDir": "runner-results"
  },
  "mochaJunitReporterReporterOptions":{
    "mochaFile":"cypress/results/user/dsk/user-[hash].xml"
  }
}

3. Our package.json now look like this:

"cm:dsk:user": "cypress run --config-file ./cypress/config/user/cm-desktop.config.js --browser chrome",
"cm:run:dsk:user:parallel": "cypress-parallel -s cm:dsk:user -t 4 -d ./cypress/user -p 'cypress/config/user/reporter-config.json'",

I found that -p flag in the settings file:

cypress-parallel/lib/settings.js

Line 50 in 710dd19

.option('reporterOptionsPath', {

which is not documented in README but it was the correct for us, now I'm not passing a reporter flag and letting the reporter-config.json to drive the enabled reporters.

Hope it helps anyone to solve the issue, I don't know if it's a bug or a miss by me while following instructions but that's what it is.

cc @tnicola

[guychienll]

hi @ezunigaf90
May I know after your change above , has this issue happened again ?

[guychienll]

hi @olga-govor
May I know you said hardcoded part is to change runner-results/ folder path ? thanks !