CDC Listener

---

A service helps for tracking MongoDB data changes asynchronously

---

Table of Contents

• Introduction
• Features
• Requirements
• Quick Start
• Contributing

Introduction

A service helps for tracking MongoDB data changes asynchronously. Change Data Capture (CDC) is a design pattern that allows you to track and capture change streams from a source data system so that downstream systems can efficiently process these changes. It's useful for "Event-Driven architecture" and implementation patterns like "Outbox" - just insert or updates data in database collections and subscribe to topic(s). Service publish event to topic (see below):

{
  "id": "ObjectID string",
  "db": "db name",
  "collection": "collection name",
  "action": "insert/update/delete operation types",
  "data": {},
  "data_old": {},
  "event_time": "ISO time"
}

Messages are published to the broker at least once!

Filter configuration example

filter:
- db: "test"
  collections:
    test:
      - insert
      - update

This filter means that we only process events occurring with the test database and test collection, and in particular insert and update data.

Topic mapping

By default, using topic provided to publisher.topic option, but if you want to send all update in one topic you should be configured the topic map:

topicsMap:
  test-test: "notifier" # [database name]-[collection name]: [topic name]

Available metrics

name	description	fields	status
published_events_total	the total number of published events	subject, table	* WIP *
filter_skipped_events_total	the total number of skipped events	table	* WIP *

Features

• Handle database(s) collection(s) changes;
• Publish using RabbitMQ type=rabbitmq;
• [TODO] Support other publishers;
• [TODO] Old changes included with new document in event;
• [TODO] Resume token logic;
• [TODO] Metrics;
• [TODO] Health checks.

Requirements

• Git
• Go >= 1.22
• Docker
• MongoDB

Quick start

1. MongoDB must be replicated;
2. Create config.yml

# Application level configs
app:
  name: "cdc-listener"
  port: 8080
  mode: "develop"
logger:
  level: info
  format: json
# MongoDB listener config
listener:
  filter:
    - db: "test"
      collections:
        test:
          - insert
          - update
  topicsMap:
    test-test: "notifier"
# MongoDB config
database:
  uri: "mongodb://127.0.0.1:27017,127.0.0.1:27018,127.0.0.1:27019/?replicaSet=rs0"
  debug: false
# Publisher config
publisher:
  type: rabbitmq
  uri: "amqp://guest:guest@localhost:5672/"
  topic: "notifier" # used as default topic
  topicPrefix: ""
# Prometheus config
monitoring:
  promAddr: ":2112"

3. Run using air or docker

docker-compose up -d build listener-dev

Contributing

Please, use git cz for commit messages!

git clone https://github.com/WildEgor/cdc-listener
cd cdc-listener
git checkout -b feature-or-fix-branch
git add .
git cz
git push --set-upstream-to origin/feature-or-fix-branch