[FLI-1067] Kafka Audit Sink

[markphelps]

Problem

Flipt currently supports audit event sinks to 3 places:

• log file soon to also be stdout, see [FLI-1026] Support audit logs to stdout #3101
• webhooks
• flipt cloud

I think we could get support for several other sinks by adding a message broker/queue such as Kafka, especially with projects like Redpanda Connect (previously Benthos RIP 💀 )

Ideal Solution

Add a new sink type kafka, which would send audit events to configured Apache Kafka message brokers

configuration may look like:

audit:
  sinks:
    kafka:
      enabled:
      topic: foo
      bootstrap_servers:
        - my-cluster-kafka-bootstrap.kafka:9092

It seems bootstrap_servers are the new broker_lists ? https://stackoverflow.com/questions/50312579/what-is-difference-between-broker-list-and-bootstrap-servers

I haven't used Kafka in years though so I could be wrong

Search

• I searched for other open and closed issues before opening this

Additional Context

Would love to know if any existing users would find this functionality useful

_FLI-1067

[thepabloaguilar]

@markphelps I'd like to suggest this library here: franz-go

It's feature complete (compression, transaction, TLS, SASL Auth) and has support not only for vanilla Kafka but for Redpanda, Microsoft Event Hubs, and MSK too

[markphelps]

Great suggestion @thepabloaguilar !!!