Cypress: Vanilla or Cucumber?

[varl]

Based on the discussion in the meeting #134 we need more time to properly evaluate if using Cucumber together with Cypress is something we want to heavily invest in.

This topic is to share experiences, discuss pros and cons, and see if we can figure out a way forward from here.

First off, let's state some questions we would like to answer. Post them below and I will add them to the list here.

To start off I will note a few from the frontend meeting.

Questions

• What do we want to use the feature files for?
• Is it possible to use Cucumber for some things but Vanilla for others in the same suite?
• What is the actual overhead of Cucumber?
• How can we report our test results automatically to Zephyr in JIRA?
• Have we had experiences where the feature files themselves have been really valuable?

Current usages

These are the usages I am currently aware of, let me know if an app or library is missing and I will amend the list.

Team Platform

Cucumber

• UI
• Import/Export
• SMS Configuration
• WHO Data Quality
• Cache Cleaner

Team Analytics

Vanilla

• Data Visualizer
• Maps

Cucumber

• Dashboards

Team Tracker

Cucumber

• Capture

[Mohammer5]

What do we want to use the feature files for?

• Replace writing zephyr test steps, include a link to a feature file (/w lines to the relevant scenarios if applicable) instead
• Automated detailed test failure report to QA in JIRA

Is it possible to use Cucumber for some things but Vanilla for others in the same suite?

• We could use feature files for DHIS2-XXXX jira issues and use vanilla cypress for all TECH-, LIB-, and other XXXX issues
  • DHIS2-XXXX feature files could use the same tagging mechanism that QA is already using to tag which scenario/feature belongs to which issue

What is the actual overhead of Cucumber?

• Parsing time of feature files is a biggie!
• Onboarding: Cognitive overhead when new to feature files
  • It takes a bit of experience to figure out how to structure feature files so that the tests actually work
  • Feature files could be proposed by non-techies, but they'd never be in their final form (see point above)
• For some people the mental overhead of having to think in different structures instead of just writing plain vanilla cypress tests causes an increase in time to write these tests
  • (For me it's the other way, it takes a bit more time to write the feature files first, but then I have a very clear structure and just need to implement the individual steps without having to think about the "bigger picture" (the full scneario))
• We have a higher maintenance burden when it comes to support libraries (the cli utils tool) as we have to maintain code related to cypress AND cucumber

How can we report our test results automatically to Zephyr in JIRA?

• Regardless of whether we want to use feature files or not, cypress has a mature reporting support. We could leverage on existing reporters or write a custom one that sends the data to the appropriate apis of our Jira

The README of cypress-cucumber-preprocessor library states the following:

The cypress-cucumber-preprocessor can generate a cucumber.json file output as it runs the features files. This is separate from, and in addition to, any Mocha reporter configured in Cypress.

These files are intended to be used with one of the many available Cucumber report generator packages. Seems to work fine with both https://github.com/jenkinsci/cucumber-reports-plugin and https://github.com/wswebcreation/multiple-cucumber-html-reporter

Have we had experiences where the feature files themselves have been really valuable?

• I didn't have to write zephyr test steps
  • This is of course a personal experience, but everytime I'm writing a manual zephyr test, it's really frustrating. The UI is just bad and it eventually contains exactly the same stuff that I would've put into a feature file anyways, just without being connected to actual test code
• I find writing tests this way a lot easier.
  • It definitely took some time to get used to it. But my workflow now is quite nice:
    • I don't group steps by scenario mentally anymore, I look at each step independently when looking at the test code.
    • When writing feature files, I "just" have to think about what the app should be capable of and I write that down.
      I don't have to think about test code whatsoever while thinking about how the app works.
      When it comes to writing the test code after I finished the feature files, I first copy all the individual steps and create the empty execution functions. I then just go from top to bottom, from step to step and implement that step, regardless of its context
    • => To me it feels it actually speeds up writing tests
      • Test code is chunked into logical parts which are easy to find
      • The title of the steps is included in the cypress GUI, I know exactly how to quickly find the responsible code
• It's very easy to understand what a test should do - when not having worked on a feature yet - without having to read through actual code

[awgaan]

Have we had experiences where the feature files themselves have been really valuable?

I rewrote the import-export app, and I found the feature files very helpful to read when I started looking at the app and tried to determine what it was/(should be) capable of.

When I added some new features (like tracked entity instance import/export) I wrote feature files to help me understand the exact requirements.

Background

No Cucumber or Cypress experience before working on the import-export app. For me, the Cucumber feature files were not a cognitive overhead, but rather made it easier to see the whole picture.

What is the actual overhead of Cucumber?

@Mohammer5 mentions that the parsing time of feature files is significant. I don't get why. A parser cannot possibly be that slow. It's just matching our humanly readable Cucumber syntax into a list of already defined functions. Or am I missing something?

Stubbed Cypress responses

I think we should be careful trusting passed Cypress results where the tests use stubbed responses (at least for apps). I worked on something else for a while before the 2.35 freeze and came back to a ton of broken functionality in the import-export app that I thought worked because the Cypress tests passed, but forgot to consider that the 2.35 API responses did not match the 2.34 stubbed responses.

[Mohammer5]

@Mohammer5 mentions that the parsing time of feature files is significant. I don't get why. A parser cannot possibly be that slow.

@awgaan (copied from slack:) I assume it's the additional IO processes of reading files files. In my experience, that's always the slowest part when having to deal with parsing

• The files have to be read
• The plugin has to
  • parse the individual steps, taking scenarios with multiple examples into account
  • create adhoc test files with describe and it, re-arranging the scenarios
  • Has to find relevant test code (files in the feature's folder, common code), which requires checking some configs

And the above is happening for every single feature file. It doesn't matter too much with a couple of feature files. But with 100+ feature files and 500+ scenarios, this can take a while (a minute or two).

I think, while this is a nuisance while waiting for CI to run the tests, I don't mind the extra time here too much, I prefer having the feature files, at least when just comparing this trade-off vs feature files in isolation.

[ghost]

Ideally, I think our test suite should resemble the distribution mentioned here: https://kentcdodds.com/blog/unit-vs-integration-vs-e2e-tests. That means that the more realistic tests should make up the smallest part of our test suite (like cypress tests), because they're expensive. You also don't paint a wall with a tiny brush. It might be accurate, but it's not practical.

Currently, repos like UI have a ton of cypress tests, and some unit tests. That should be the other way around. That is my core consideration for all our discussions about tests. So I think we should test most functionality in jest, and a (much) smaller portion with cypress. That is the practical, common approach. Whether we use feature files or not. If we want to guard all our features in cypress it's not going to be a practical test suite, but a slow, costly to maintain one.

It might be precise to paint an entire wall with a tiny brush, but it's going to lead to fatigue, extra cost, and noone is going to want to paint anymore in the end. So realistically we can't guard all our features with cypress.

So if we can't guard all our features with our cypress tests, then what we have left is a slightly more human readable syntax for a couple of integration tests. To me, that isn't worth it.

[Mohammer5]

@ismay what about apps? Would you also integrate most of the tests that currently leverage on cypress with jest instead?

In regards to UI I 100% agree that we overshot with writing all the tests with cypress. But testing apps is a completely different scenario, which includes forms, "complex" interaction between user/app and app/backend, etc. This seems to warrant using browser-based tests.

In that scenario I'm currently in favor of using feature files. The experience @awgaan had is exactly what I was hoping for as well (someone new to the app has little to no problems understanding how the app is supposed to function)

[varl]

I think the workflow that has emerged from using Cucumber is quite good, but we don't have to use Cucumber itself to use the workflow:

1. Write documentation for how the feature should work, and get feedback from stakeholders.
2. Write tests for how the feature should work.
3. Implement the feature.
4. Run tests to verify the implementation.

The end result is that we end up with documentation that our users can read.

What we end up with Cucumber/Gherkin is documentation that a computer can read.

---

In practice, I concede that I would write the feature first, then the tests. But I would use the documentation as guide for how the feature is implemented.

[ghost]

@ismay what about apps? Would you also integrate most of the tests that currently leverage on cypress with jest instead?

In regards to UI I 100% agree that we overshot with writing all the tests with cypress. But testing apps is a completely different scenario, which includes forms, "complex" interaction between user/app and app/backend, etc. This seems to warrant using browser-based tests.

Check out the article I linked. It's specifically about writing tests for frontend apps. The guy who wrote it is a former lead at Paypal, so he has experience with setting up decent test suites for frontend applications. I trust his judgement and have found following that advice to result in maintainable, reliable test suites.

I don't see why we would diverge from that.

[Mohammer5]

@ismay so you'd prefer if we tried to write the tests that we currently have in some of our apps (import export, sms config) to be written with a non-browser-based tool (e. g. jest) instead?

What would you use e2e for in our case (or do you think we shouldn't do that at all)?

[ghost]

@ismay so you'd prefer if we tried to write the tests that we currently have in some of our apps (import export, sms config) to be written with a non-browser-based tool (e. g. jest) instead?

What would you use e2e for in our case (or do you think we shouldn't do that at all)?

Maybe we're miscommunicating a bit. And this is diverging a bit from the core topic, but to clarify my position: basically I'm suggesting we follow best practice here, i.e. what's outlined in the article. That's what I'm basing my position on. Do you disagree with any of the statements made in the article, or do you feel the same?

[martinkrulltott]

I think the workflow that has emerged from using Cucumber is quite good, but we don't have to use Cucumber itself to use the workflow...

I very much agree with @varl 's statement above. The workflow is good, but can be achieved without Cucumber itself.

As long as there are no strong benefits of using Cucumber I don't see why we should force everyone to use it. We also have to take in to consideration that we have a lot of different apps which have different levels of complexity, linearity, amount of user paths, legacy code, external dependencies etc which all affect what needs to be tested and how. Looking at Data Visualizer, which has high levels of all above (very complex, not linear, tons of user paths, lots of legacy code and external deps), and comparing that to a smaller greenfield linear app and forcing a technical solution on to both would not be a good idea.

The workflow (as mentioned by @varl ) could still be applied though, but I don't see why a technical solution like Cucumber is neither necessary nor a good way to achieve this.

[Mohammer5]

Maybe we're miscommunicating a bit. And this is diverging a bit from the core topic, but to clarify my position: basically I'm suggesting we follow best practice here, i.e. what's outlined in the article. That's what I'm basing my position on. Do you disagree with any of the statements made in the article, or do you feel the same?

The article made me thing about a few thing. What I'm trying to get to is:
How many tests do you think we'd have that we do with cypress. 10 or 50 per app? I guess that you could argue that most of the tests I'm writing now with cypress for the sms configuration app could be written with jest as well.

If we end up with more or less 10 cypress tests per app, then there wouldn't be a real need for feature files, so I agree with what you're saying @ismay IF we really will have a low cypress test amount.

---

When talking about the usefulness of feature file, I think we have to zoom out a bit. So far we (as an org) don't have much experience with feature files except for writing them. To me it feels like the real gains show up later in the process, not right away.

I think there are different things that feature files will do for us:

1. Zephyr test steps
We always have to write a zephyr test issue when we're done working on a DHIS2-XXXX ticket. The content of these will be more or less exactly the same as what we'd put into feature file, just in a very non-user-friendly way in Jira

2. We'd have human readable documentation
I think the experience @awgaan had is quite good. Of course this could be achieved with regular tests as well, but if we have to write zephyr test feature like descriptions anyways, we could at least do that in the repository.

• It doesn't really matter if the behavior gets out of date in jira or the repository when not connected to an automated test, it'd happen anyways. In that case I'd still prefer to have feature files in the repository and just link to the relevant parts in the zephyr test issues, but that's a slightly different topic

For me, the Cucumber feature files were not a cognitive overhead, but rather made it easier to see the whole picture.
(by @awgaan, see above)

3. Workflow?
This is very subjective of course, but I find working with feature files way more comfortable to work with. I think there are some who had the same experience (@jenniferarnesen?)

When I added some new features (like tracked entity instance import/export) I wrote feature files to help me understand the exact requirements.

I have a very similar experience. Writing feature files first helps me structuring my thoughts and the information gathering process as it's so easy to just write text. This could be done with describe/it as well, but it's more not as nice (because one scenario in feature files would be one it block in normal tests, the description string contains fewer information that a couple of steps)

But I guess this is highly personal and others do not share the experience.

4. CI test failure reporting
(I think you, @ismay, weren't there when we had the meeting about cypress & cucumber/vanilla)
QA has told us that the best we could do is to provide detailed information when sending the error reports, which would contain

• which test failed
• at what step the test failed
• the reason why the test failed

With feature files, I imagine this to be very straight forward to provide that information

Summary
So from my POV, we have to write feature-content anyways (zephyr), it could add value later on (documentation), when getting used to it it could improve the workflow and it could help with error reporting cross team (dev / QA). To me these are good reasons to keep using feature files. BUT this is of course only true if there are enough tests that we'd implement that way

---

forcing a technical solution on to both would not be a good idea (by @martinkrulltott)

I don't think we're talking about forcing a technical solution. If I remember correctly, we've said that having both vanilla and cucumber is an option and could depend on the use-case and/or context (e. g. DHIS2-XXXX vs TECH-XXXX, etc)

---

I think this article also makes some fair points:

Taxing test coverage

Within BDD, scenarios are only created for key examples of behaviour and are not used to document every single edge case as tests. However, when teams look at or consider their test coverage, they often get concerned about the low coverage they have achieved using Cucumber.

The typical response to the poor test coverage is to write more scenarios in their existing BDD Cucumber framework to increase the test coverage. This knee-jerk reaction is caused by the investment already made into their Cucumber framework, but the regex tax is overlooked within the equation. A team going down this route will quickly start paying the overhead of making the tests human-friendly, without getting any benefits for it.

I guess this ties in with what @ismay refers to in the article (the amount of tests in the various categories). The workflow @varl has laid out is a BDD workflow (describe behavior/feature first, then write code). The important part of the quote above is: "key examples of behaviour and are not used to document every single edge case as tests".

The conclusion of the author of the article is:

Cucumber is a powerful tool but its use isn’t always the most suitable one to use, particularly when you are pursuing high test coverage or a broad test automation strategy. Teams would do better to select the most appropriate tool for each individual job and should remember that more than one automation tool can be used within each testing stage

If our primary goal is to get a "high test coverage" or "broad test automation strategy" and we don't really want to focus on key features, then I guess we could even completely ignore cypress for now and switch to cli based tools

[martinkrulltott]

2. We'd have human readable documentation

Feature files shouldn't replace proper documentation and I think this could be a whole discussion on its own.

[Mohammer5]

Feature files shouldn't replace proper documentation and I think this could be a whole discussion on its own.

There are different types of documentation. I'm not trying to suggest to replace all our documentation with feature files

[martinkrulltott]

Thanks for the unit-vs-integration-vs-e2e-tests link @ismay . My take from the article is that you have to consider why you're writing tests and what you're trying to achieve. For the smaller greenfield apps one might want to achieve full test coverage for future use and make it easier to understand how the app works for a future developer (as also pointed out by @Mohammer5 's #2 point). However in the Analytics apps (most prominently in DV, but also in Maps and Dashboard) AFAIK the main goal at the moment is to enhance the regression testing and make up for the fact that we have very large apps and relatively scarce test resources.

Using the articles painting analogy I'm trying to achieve a wide coverage by using a wall roller in random pattern across a large wall (like how a robot lawnmower / vacuum cleaner works) rather than systematically painting each square cm by moving in straight lines upwards down (like an inkjet printer).

You can indeed write feature files for linear faeture-driven "inkjet testing", but how do you write feature files for the robot lawnmower? And will those feature files actually benefit any of the points @Mohammer5 made above?

Obviously (if still going for Cucumber) the hybrid model would be the only viable solution here. Using vanilla for wide coverage legacy regression and using Cucumber for upcoming features in the future.

[varl]

Throwing this in here too. Deep integration of feature files into JIRA.

https://marketplace.atlassian.com/apps/1221264/cucumber-for-jira-bdd-natively-in-jira?hosting=cloud&tab=overview

🧰Synchronize Feature Files from Git

• Seamlessly integrate with the 3 Git providers: GitHub, GitLab and Bitbucket
• Visualize feature files (from multiple Git branches) in Jira

🚀Unleash Full BDD Capabilities in Jira

• Support Gherkin syntax and collaborate using Jira
• Create Jira issues directly in Cucumber for Jira
• Link BDD features to Jira issues
• Download feature content from a Jira issue

🔌Flexible Integration with Automation & Test Management

• Push tests results from any CI/CD tool
• Integrate with Zephyr for Jira to combine test management with BDD
• Upload test results directly from Cucumber for Jira

---

You can indeed write feature files for linear faeture-driven "inkjet testing", but how do you write feature files for the robot lawnmower? And will those feature files actually benefit any of the points @Mohammer5 made above?

I don't want to derail the discussion, but it is important to note that feature files are defining behavior of an application, and they are strictly not a part of the test code implementation that tests the behavior.

As long as you can describe the behavior of a feature, you should be able to implement e.g. property based or randomization test strategies in the test code itself to verify the behavior.

[Mohammer5]

I'm trying to achieve a wide coverage by using a wall roller in random pattern across a large wall

@martinkrulltott I'd like to schedule a meeting and would ask you to demonstrate the part that you're testing. Maybe there's a way to test this without randomness and complicated mechanisms.

I can imagine that we have different testing options. I also think that this is exactly what the article, that @ismay has linked to, includes a potentially valid point in this case:

if you try to use an E2E test to check that typing in a certain field and clicking the submit button for an edge case [...], you're doing a lot of setup work by running the entire application (backend included)

An E2E test has more points of failure making it often harder to track down what code caused the breakage, but it also means that your test is giving you more confidence. This is especially useful if you don't have as much time to write tests. I'd rather have the confidence and be faced with tracking down why it's failing, than not having caught the problem via a test in the first place.

Maybe a more generalized e2e test would suit this better rather than covering all possible combinations and integration tests would suit testing the combinations better

[varl]

The problem now is that I agree with everyone:

• Yes, we need to consider the balance of our test pyramid.
• Yes, we need to specify the intended behavior of an application.
• Yes, we need to have a way to do characterization tests.
• Yes, we need a way to fulfill the interface with the DHIS2 Agile process.
• Yes, we need user facing documentation.
• Yes, we need to utilize different test strategies for different purposes.
• Yes, we need to have acceptance criteria for features to understand them better.
• Yes, we need to have E2E tests for application features.

Nothing in that list exclusively requires Cucumber. My concern is that we are inventing a new system to avoid working with JIRA.

This isn't entirely intentional, but we are definitely heading down a path here.

Let's look at what we are doing:

• Descriptions for features in JIRA are notably poorly defined, so instead of contributing to improve those we write the descriptions in Feature files.

• Writing Zephyr test cases is a terrible experience, so instead we write our test cases based on our own Feature files.

• Instead of colocating our feature specs with bugs/improvements/tasks, we colocate our Feature files with code in repos.

• We have very poor documentation and find it hard to write, so we use Feature files to have some form of human readable documentation.

• JIRA has a shit editor, so we want to use our own editor.

The effect of this is that we are fragmenting information that should be in JIRA because it is the tool, for better or worse, we use to manage the entire development process over time for DHIS2.

By positioning Cucumber between ourselves and JIRA we are ultimately not improving the problems we have at their root. For Cucumber to be effective for DHIS2 as a whole and not cause further fragmentation, we would need DHIS2 to embrace BDD.
Using Cucumber should come as an extension to the organization using BDD.

---

edit: updated with clarifications.

[Mohammer5]

I agree with what you're saying @varl, I'm not certain what your actual conclusions are.
My interpretation is:

1. we need to figure out a way to do BDD with jira that works
2. then we can figure out how to incorporate feature files into the workflow (if we event want to do that after finishing with step 1)
   • This could replace/extend parts of the workflow defined in the previous step