Performance testing framework

[BobdenOs]

Goal

With these changes it should be possible to define comprehensive performance tests for any part of the CAP library functionalities. By defining scenarios based upon the SFlight example application.

Mechanism

The basis of the performance test is cds w with a custom server.js file which exposes an /config endpoint which can be used to change the .cdsrc.json file. Which triggers cds w to restart the SFlight application with the newly defined configurations. Which is used by the jest tests to configure the application before starting the test scenarios. All requests send to the application are tracked and can consume the trace middleware results for more detailed performance analysis.

Adapters

To be able to run the exact same scenario with different adapters (odata, rest, graphql). The tests use CQN to create the corresponding end point calls. So when a new adapter is supported it would be possible to run the same scenarios with that adapter after adding the exec function implementation to the performance test suite.

Databases

For the possibility of testing all the different database implementation it is required to ensure that the dependencies are installed before calling cds w. This is especially important when deploying the application as only the defined dependencies will be installed. Additionally it is required to configure all service bindings for the database implementations that are going to be tested. As changing the configurations will only restart the application and not re deploy the application with new bindings. This is an intentional behavior as deploying the application takes more time and it might be deployed to different physical hardware which might impact the measured results.

Scenarios

Currently a very simple scenario is included where users slowly scroll down the landing page as this is a request that every user needs to make before doing anything else with the SFlight application. To add more scenarios it will be required to define more actions that users could perform with the SFlight application. This is done by manually going through the steps in the application and taking the requests send by the application and translating them to the CQN format. Which will allow the tests to use these actions and create more complex scenarios. It is important that the scenarios are reproducible to reduce variance between executions.

Result

Here is an example of how the measurement results look like:

{scenario}

{load} ({nr} users)

	{database}	...
{adapter}	{min} ms (avg: {average} ms)	...
...	...	...

To Do

• Automate deployment
• Add performance script in /package.json
• Improve reporting

[BobdenOs]

Local performance run with all available protocol adapters and most database service.

---

landingPageList

baseline (1 users)

	sqlite (new)	sqlite disk (new)	sqlite (old)	sqlite disk (old)	hana hdb (new)	hana hana-client (new)	postgres
okra	5 ms (avg: 7 ms)	6 ms (avg: 7 ms)	14 ms (avg: 16 ms)	14 ms (avg: 16 ms)	87 ms (avg: 91 ms)	85 ms (avg: 93 ms)	14 ms (avg: 19 ms)
odata	5 ms (avg: 7 ms)	5 ms (avg: 7 ms)	14 ms (avg: 15 ms)	14 ms (avg: 15 ms)	87 ms (avg: 92 ms)	85 ms (avg: 90 ms)	15 ms (avg: 19 ms)
rest	5 ms (avg: 6 ms)	5 ms (avg: 6 ms)	13 ms (avg: 15 ms)	13 ms (avg: 15 ms)	86 ms (avg: 91 ms)	83 ms (avg: 90 ms)	13 ms (avg: 18 ms)
graphql	6 ms (avg: 7 ms)	5 ms (avg: 7 ms)	10 ms (avg: 12 ms)	10 ms (avg: 12 ms)	84 ms (avg: 89 ms)	84 ms (avg: 88 ms)	13 ms (avg: 18 ms)
hcql	7 ms (avg: 8 ms)	7 ms (avg: 8 ms)	15 ms (avg: 16 ms)	14 ms (avg: 16 ms)	62 ms (avg: 68 ms)	60 ms (avg: 65 ms)	17 ms (avg: 21 ms)

m (128 users)

	sqlite (new)	sqlite disk (new)	sqlite (old)	sqlite disk (old)	hana hdb (new)	hana hana-client (new)	postgres
okra	2 ms (avg: 4 ms)	2 ms (avg: 4 ms)	10 ms (avg: 13 ms)	9 ms (avg: 13 ms)	12 ms (avg: 76 ms)	11 ms (avg: 76 ms)	8 ms (avg: 15 ms)
odata	2 ms (avg: 4 ms)	2 ms (avg: 4 ms)	10 ms (avg: 13 ms)	10 ms (avg: 13 ms)	11 ms (avg: 75 ms)	12 ms (avg: 77 ms)	9 ms (avg: 15 ms)
rest	3 ms (avg: 5 ms)	3 ms (avg: 4 ms)	10 ms (avg: 14 ms)	10 ms (avg: 14 ms)	12 ms (avg: 95 ms)	12 ms (avg: 99 ms)	9 ms (avg: 19 ms)
graphql	3 ms (avg: 5 ms)	3 ms (avg: 5 ms)	6 ms (avg: 10 ms)	6 ms (avg: 10 ms)	12 ms (avg: 959 ms)	13 ms (avg: 80 ms)	7 ms (avg: 14 ms)
hcql	4 ms (avg: 6 ms)	4 ms (avg: 6 ms)	11 ms (avg: 15 ms)	11 ms (avg: 14 ms)	20 ms (avg: 593 ms)	20 ms (avg: 557 ms)	10 ms (avg: 29 ms)