Intermittent errors producing to Redpanda using Sarama library

[gary-lloyd]

Version & Environment

Redpanda version: (use rpk version): v21.10.1 (rev 8df2069)

Please also give versions of other components:

• macOS 12.0 (21A344) arm64 Apple Silicon
• sarama
• go1.17.3 darwin/arm64

What went wrong?

I created a local Redpanda instance using rpm container start
Then ran the code below to produce two messages into a new topic. This code sometimes works and sometimes fails. When it fails there is an error in the Redpanda log that says:

ERROR 2021-11-24 12:22:28,725 [shard 0] kafka - legacy_message.h:121 - Cannot validate legacy batch (magic=1) CRC. Expected 2323769037. Computed 670664417

I've tried essentially the same code with Franz-go and it seems to work just fine. Unfortunately, I can't easily switch to this as I'm trying to use Venom to write integration tests and the Kafka executor uses Sarama.

What should have happened instead?

The messages should always be produced to the Kafka topic without error

How to reproduce the issue?

To reproduce

1. start a local Redpanda instance with rpm container start
2. Execute code below having updated address.

package main

import (
	"fmt"
	"github.com/Shopify/sarama"
	"time"
)

const (
	defaultProducerMaxRetries     = 10
	defaultDialTimeout            = 10 * time.Second
)

func main() {
	kafkaAddress := []string{"127.0.0.1:51419"}

	messageValue1 := []byte("The first message")
	messageValue2 := []byte("The second message")
	messageValues := [][]byte{messageValue1, messageValue2}
	produceSarama(kafkaAddress, "mytopic", messageValues)
}

func produceSarama(kafkaAddress []string, topic string, messageValues [][]byte) {
	config := createSaramaKafkaConfig()

	sp, err := sarama.NewSyncProducer(kafkaAddress, config)
	if err != nil {
		fmt.Println("Can't open Redpanda connection")
	}
	defer sp.Close()

	saramaMessages := []*sarama.ProducerMessage{}

	for i := range messageValues {
		messageValue := messageValues[i]
		saramaMessages = append(saramaMessages, &sarama.ProducerMessage{
			Topic: topic,
			Value: sarama.ByteEncoder(messageValue),
		})
	}

	err = sp.SendMessages(saramaMessages)
	if err != nil {
		fmt.Println(err)
	}
}

func createSaramaKafkaConfig() (*sarama.Config) {
	config := sarama.NewConfig()
	config.Producer.Return.Successes = true
	config.Version = sarama.V0_10_2_0

	return config
}

[dotnwat]

Thanks for the detailed report @gary-lloyd. Even though you aren't able to switch away from Sarama, would you be able to bump up the version config.Version = sarama.V0_10_2_0 to something newer? Presumably that would then use a newer version of the Produce API. In either case, we should be able to fix this pretty quickly based on this reproducer. Legacy support is in Redpanda, but isn't nearly as well tested as newer version of the API.

[gary-lloyd]

Thanks @dotnwat that was exactly the issue. Venom supports specifying the version in test configuration and that has got my tests passing. I'll look at submitting a PR to them to bump the default version as that one is a bit old!

[emaxerrno]

@gary-lloyd - moving this into a discussion.

[dotnwat]

why is this not an issue?

[emaxerrno]

seemed like a discussion. happy to promote to issue again.

[emaxerrno]

but in retrospect it looks like an issue. oops. sry @gary-lloyd

[gary-lloyd]

No worries @senior7515 I got the answer to my problem so I don't mind where you guys want to track it!

[dotnwat]

fyi @senior7515 it looks like there isn't a clean way to convert the discussion back into an issue. there is a button, but it doesn't seem to preserve the discussion.

[emaxerrno]

sorry :'( my bad.