Reliable benchmarks without throttling side-effects

[roookeee]

Most mappers (Orika, BULL etc.) in this repository had new releases since the last benchmark run from february - let's get some up to date data and update the readme's table + image.

[roookeee]

Anything you need here? I think you should be the one to run the benchmarks and create a new graph as you have the control over the environment then :)

[arey]

You're right. I will do soon the update.

[arey]

Last version of BULL has been compiled for Java 11:

/Users/arey/Dev/GitHub/java-object-mapper-benchmark/src/main/java/com/javaetmoi/benchmark/mapping/mapper/bull/BullMapper.java:[3,24] cannot access com.hotels.beans.BeanUtils
[ERROR]   bad class file: /Users/arey/.m2/repository/com/hotels/beans/bull-bean-transformer/1.5.1/bull-bean-transformer-1.5.1.jar(com/hotels/beans/BeanUtils.class)

Any idea @fborriello ?

[fborriello]

@arey yes it is, if you need the java 8 build, set the bull-version to: 1.1.24.
The only difference between the two versions is that, the first one is compiled with java 11 and the second one with java 8

[arey]

Done. MapStruct and Selma are new before the manual version ...

[roookeee]

Hi @arey - that simply should not be the case - look at my results on an I7 6700k:

MapperBenchmark.mapper       Manual  thrpt    5  43199115,871 ± 2958841,336  ops/s
MapperBenchmark.mapper    MapStruct  thrpt    5  41722179,500 ±  216444,962  ops/s
MapperBenchmark.mapper        Selma  thrpt    5  41194963,858 ±  860577,996  ops/s
MapperBenchmark.mapper      JMapper  thrpt    5  32720748,374 ±  737046,085  ops/s
MapperBenchmark.mapper        datus  thrpt    5   8457751,577 ± 1303780,678  ops/s
MapperBenchmark.mapper        Orika  thrpt    5   4082450,031 ±  116644,367  ops/s
MapperBenchmark.mapper  ModelMapper  thrpt    5    178733,642 ±    7162,971  ops/s
MapperBenchmark.mapper         BULL  thrpt    5    154111,877 ±    2714,244  ops/s

As you use an Intel CPU please keep in mind that it throttles excessively not only based on temperature but even based on time (e.g. the maximum frequency is only held for at max 10 seconds, then you get throttling no matter the temp) on newer generation consumer intel cpus (especially in notebooks). For stable results please use a desktop pc + cpu with a fixed clock speed or something like this may arise again - it makes the results seem untrustworthy.

Kind regards,
roookeee

[filiphr]

Does it make sense to run the benchmarks on a CI? For example on Azure Pipelines or Travis CI?

[roookeee]

That would surely make sense as those should be properly cooled server CPUs which are not configured with short-term max boosting in mind. But you would lose control on what exact hardware the tests were run (which RAM and CPU) ? (correct me if Azure or Travis allows for this)

[arey]

Thanks @roookeee for this point of attention.
My Macbook Pro is a bit older (a mid-2014 model with an Intel Core i7) and I run the benchmark with the power adapter.
I don't have any desktop (just at work but it it will be replaced by a virtual machine)
On Windows 7, do you know if we could fix the clock speed?

[roookeee]

Can't help you with setting a fixed clock speed on windows - sorry. Although imperfect maybe using VirtualBox or some other virtualization and starting a Linux distro would work as limiting the cpu frequency in Linux is quite easy to achieve