How to set some configs with Redpanda's docker image

[iamkucuk]

Hello guys, I'm trying to set the following configs at the startup of docker container:

• log_cleanup_policy
• log_compression_type
• retention_bytes
• transaction_coordinator_cleanup_policy

I've tried many things like setting config with a config file. However, that gave me the following error:

Error: rename /etc/redpanda/redpanda-20221020185230.yaml /etc/redpanda/redpanda.yaml: device or resource busy, temp file removed from disk

My best guess is; redpanda is trying to create a temporary config file out of my original config file, but for some reason, that procedure gives me the error above. So, I tried to give those configs while executing redpanda start command with:

        command:
            - redpanda
            - start
            - --smp 1
            - --overprovisioned
            - --node-id 0
            - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
            - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://100.83.206.91:9092
            - --set log_cleanup_policy=compact
            - --set log_compression_type=zstd
            - --set retention_bytes=64424509440

That was not sufficient. I couldn't find the convention, nor a specification about how to pass those arguments to the redpanda start command.

Here, I leave the relevant docker-compose part for the container creation. You can see the things I've tried by observing the commented out parts.

    redpanda:
        # cpus: 1.5
        mem_reservation: "3G"
        mem_limit: "4G"
        command:
            - redpanda
            - start
            # - --config /var/lib/redpanda/redpanda.yaml
            - --smp 1
            - --overprovisioned
            - --node-id 0
            - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
            - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://100.83.206.91:9092
            # - --set log_cleanup_policy=compact
            # - --set log_compression_type=zstd
            # - --set retention_bytes=64424509440
            # - --set transaction_coordinator_cleanup_policy=compact
            # - rpk cluster config --config /var/lib/redpanda/redpanda.yaml
        image: vectorized/redpanda:v22.2.6
        container_name: redpanda
        ipc: host
        # volumes:
        #     - /home/furkan/configs/redpanda.yaml:/etc/redpanda/redpanda.yaml
        hostname: redpanda
        ports:
            - "9092:9092"
            - "29092:29092"
        

Waiting for your help. Thanks in advance.

[emaxerrno]

@twmb is this via --set redpanda.log_cleanup_policy=compact style so the redpanda prefix is missing?

[iamkucuk]

@emaxerrno Can you point me to a docs page for further information?

[vuldin]

Here's a link https://docs.redpanda.com/docs/reference/node-configuration-sample/

[iamkucuk]

As @emaxerrno suggests, adding redpanda prefix solved my problem. However, this setting resulted in the following topic info message:

INFO 2022-10-20 20:45:56,018 [shard 0] cluster - topics_frontend.cc:82 - Create topics {{configuration: { topic: {ns: {kafka}, topic: {ticker-FTX-APE-USD-PERP}}, partition_count: 1, replication_factor: 1, properties: { compression: {nullopt}, cleanup_policy_bitflags: {nullopt}, compaction_strategy: {nullopt}, retention_bytes: {}, retention_duration_ms: {}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {nullopt}, read_replica: {nullopt}, read_replica_bucket: {nullopt} remote_topic_properties: {nullopt}}}, custom_assignments: {}}}

This message tells me that what I do is wrong. Is it? How can I enable (and apply) specified configs to all topics that ever created within the cluster? (like they are default settings)

[emaxerrno]

looks like the cluster is not yet up and running. or use rpk topic describe ticker... --brokers <ip:port,ip:port,ip:port>

[iamkucuk]

Unfortunately, the container is up for 21 hours, still no topic.

[vuldin]

Could you post the file /home/furkan/configs/redpanda.yaml?

[vuldin]

Here is a working version of your script (without the redpanda.yaml file being mounted):

version: "3.9"
services:
  redpanda:
    container_name: redpanda
    image: vectorized/redpanda:v22.2.6
    command:
    - redpanda start
    - --smp 1
    - --memory 512M
    - --reserve-memory 0M
    - --overprovisioned
    - --node-id 0
    - --check=false
    - --kafka-addr INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
    - --advertise-kafka-addr INSIDE://redpanda0:29092,OUTSIDE://localhost:9092
    - --rpc-addr 0.0.0.0:33145
    - --set redpanda.log_cleanup_policy=compact
    - --set redpanda.log_compression_type=zstd
    - --set redpanda.retention_bytes=64424509440
    - --set redpanda.transaction_coordinator_cleanup_policy=compact
    hostname: redpanda
    ports:
    - 9092:9092
    - 9644:9644

[iamkucuk]

Here is a working version of your script (without the redpanda.yaml file being mounted):

version: "3.9"
services:
  redpanda:
    container_name: redpanda
    image: vectorized/redpanda:v22.2.6
    command:
    - redpanda start
    - --smp 1
    - --memory 512M
    - --reserve-memory 0M
    - --overprovisioned
    - --node-id 0
    - --check=false
    - --kafka-addr INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
    - --advertise-kafka-addr INSIDE://redpanda0:29092,OUTSIDE://localhost:9092
    - --rpc-addr 0.0.0.0:33145
    - --set redpanda.log_cleanup_policy=compact
    - --set redpanda.log_compression_type=zstd
    - --set redpanda.retention_bytes=64424509440
    - --set redpanda.transaction_coordinator_cleanup_policy=compact
    hostname: redpanda
    ports:
    - 9092:9092
    - 9644:9644

With a few adjustments this one works perfectly. Don't know what was wrong earlier, probably messed up ip specifications. Thanks for the help.

[freef4ll]

The transaction_coordinator_cleanup_policy is a cluster setting value setting, so setting this in yaml doesn't seem to adjust the value it self:

$ grep transaction_coordinator_cleanup_policy /etc/redpanda/redpanda.yaml 
    transaction_coordinator_cleanup_policy: compact

Cluster config value still remains:

rpk cluster config get transaction_coordinator_cleanup_policy
delete

[vuldin]

Apologies for the confusion, cluster settings need to be set in one of two ways:

1. as a flag given to the redpanda start command (as shown in the docker compose examples above) or in `/etc/redpanda/redpanda.yaml for initial start only. On initial start there is a conversion process that looks for cluster-specific settings in the node config and applies them as needed
2. via rpk, ie. rpk cluster config set transaction_coordinator_cleanup_policy compact --api-urls 127.0.0.1:9644

You can modify transaction_coordinator_cleanup_polcy without restarting redpanda, more details on this and other similar properties here: https://docs.redpanda.com/docs/reference/cluster-properties/#transactions