Poor performance compared to baseline

[drewboardman]

I have recently been having some issues with performance, so I decided to make a minimal application and compare it to some benchmark.

TL;DR - I'm seeing ~10%-12% efficiency when using fs2-kafka (wrt benchmark).

Sample application

https://github.com/drewboardman/fs2-kafka-perf

Instructions

Kafka provides a command line interface to performance test against your topic. Please see the sample application for how to install and use provided scripts to set up docker and the benchmark tests.

Results

Using the test-application linked in the github repo, here are the results I
got:

perf-test.sh benchmarks

Partitions	Replica Factor	Record Size (Bytes)	volume	rec/sec average
5	1	1024	3_000_000	113528
5	1	556	1_000_000	138985
5	1	556	1_000_000	174703

fs2-kafka application performance

Parallelism	Linger (sec)	Batch Size	Chunking	Partitions	Replica Factor	Record Size (Bytes)	volume	rec/sec average	Efficiency WRT benchmark average @ 1_000_000 events
default	default	default	No	5	1	~560	1_000_000	5787	3.69%
default	default	default	1000	5	1	~560	1_000_000	18518	11.81%
default	default	1_000_000	1000	5	1	~560	1_000_000	19915	12.70%
30_000	default	default	1000	5	1	~560	1_000_000	19149	12.21%
200	default	default	1000	5	1	~560	1_000_000	18711	11.93%
default	2	default	1000	5	1	~560	1_000_000	17562	11.20%

As you can see, the performance is quite throttled with respect to the kafka
performance test. Chunking was, by far, the largest increase in performance - however I'm still not close to the benchmark values.

Circe as a contributing factor

It's possible that the serialization step is contributing to the performance
issues. In this application I'm using circe to do my serialization and JSON
and as protocol. Here are averages over 4x1_000_000 runs using only circe (no
kafka at all).

volume	total encoding time (sec)	average chunked total time with kafka (sec)	percentage of total time
1_000_000	13.9	53.2737	26.09%
1_000_000	14.875	53.2737	27.92%
1_000_000	13.9	53.2737	26.09%
1_000_000	13.5	53.2737	25.34%

It looks like about 75% of the time is still contributed by fs2-kafka.

[bplommer]

Thanks for taking the time to do this! We haven't really focused much on performance so far but I think it's something we should look to do in the future. Adding benchmarks to this repo would be a good start - how would you feel about opening a PR?

[drewboardman]

The benchmarks are coming from a kafkactl performance script. Can you elaborate on what you mean by adding them to the repo?

[bplommer]

I’ll need to look in more detail to give a specific answer to that - I’ll try to get to it soon.

[sholokhov]

Just wanted to post a small update here.

We recently had an issue with fs2-kafka producer performance and I stumbled upon this issue. Quite some time has passed since last activity here, so I decided to re-run benchmarks against the latest versions of fs2-kafka and kafka brokers. I also saw a couple of issues in the original benchmark (e.g. Scala app was producing records into one partition instead of using all of them).

Env: Scala 2.13.8, fs2-kafka 3.0.0-M9, Apple M1 Max, 32Gb RAM, single Kafka broker running in docker.

Some results that I've managed to achieve:

# kafka-producer-perf-test
222710 records sent, 44533,1 records/sec (21,24 MB/sec), 1113,7 ms avg latency, 1412,0 ms max latency.
268960 records sent, 53770,5 records/sec (25,64 MB/sec), 1225,8 ms avg latency, 1290,0 ms max latency.
273280 records sent, 54634,1 records/sec (26,05 MB/sec), 1194,1 ms avg latency, 1244,0 ms max latency.
1000000 records sent, 52075,196584 records/sec (24,83 MB/sec), 1181,42 ms avg latency, 1412,00 ms max latency, 1203 ms 50th, 1352 ms 95th, 1397 ms 99th, 1405 ms 99.9th.

# fs2-kafka results
complete (1000000 records), stats: 20832 millis (50000 recs/sec, 23 Mb/sec)
complete (1000000 records), stats: 19629 millis (52631 recs/sec, 25 Mb/sec)
complete (1000000 records), stats: 19299 millis (52631 recs/sec, 25 Mb/sec)

So in general it looks like current fs2-kafka producer performance looks pretty close to the baseline (kafka perf test script). Here is a gist with Scala benchmark app code - https://gist.github.com/sholokhov/d4b5a2da866c5a2117b5a335896a22a1