Skip to content

A script to automatically sync vcons from Redis into Mongo

License

Notifications You must be signed in to change notification settings

vcon-dev/mongo-redis-sync

Repository files navigation

Below is an example README.md file in Markdown format. Feel free to tailor it to your specific project needs.


Redis to Mongo Sync

A Python script that synchronizes Redis keys (prefixed with vcon:) into a MongoDB collection. It can optionally listen to Redis keyspace events to update MongoDB in real time when the keys are changed.

Table of Contents


Features

  • Initial Key Scan: Scans Redis keys (vcon:*) and inserts/updates them in MongoDB.
  • Real-time Updates: Listens to Redis keyspace notifications to stay in sync whenever Redis keys are created or updated.
  • Configurable: Uses environment variables (via python-dotenv) for easy setup.
  • Logging: Leveraging Python’s logging module for structured logs and error reporting.

Prerequisites

  1. Redis 7.x or later (with Keyspace Notifications enabled)
  2. MongoDB 4.x or later
  3. Python 3.9+ (for local runs)
  4. Docker (optional, if you wish to run via container)

Make sure Redis is configured to broadcast keyspace notifications, e.g.:

redis-cli config set notify-keyspace-events ExA

Installation

  1. Clone or download the repository containing this script.

  2. Install Python dependencies:

    pip install -r requirements.txt

    The default requirements.txt should include:

    redis
    pymongo
    python-dotenv
    
  3. Create a .env file in the same directory (if one does not already exist). For example:

    cp .env.example .env

    Then update the contents as needed.


Configuration

The script reads environment variables from the .env file:

# .env
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0

MONGO_URI=mongodb://localhost:27017
MONGO_DB=mydatabase
MONGO_COLLECTION=vcon

Adjust these values to match your local or production environment. You can also pass them via the command line or other environment injection methods, but .env is the simplest for local development.


Usage

  1. Make sure that both Redis and MongoDB are running and properly configured.

  2. Run the script from your terminal:

    python main.py

    This will:

    • Perform a one-time scan of the Redis keyspace for vcon:* keys and upsert them into MongoDB.
    • Begin listening for keyspace notifications to keep MongoDB in sync with new or updated vcon: keys in Redis.
  3. Check the logs for confirmation messages:

    • Keys found and upserted into MongoDB.
    • Any real-time updates when vcon: keys are created or modified.

Use CTRL+C or another termination signal to stop the script.


Running via Docker

You can run this script in a Docker container if you prefer:

  1. Create a Dockerfile in your project (an example is below):

    # Use a lightweight Python base image
    FROM python:3.9-slim
    
    WORKDIR /app
    
    # Copy requirements to leverage Docker cache
    COPY requirements.txt /app/requirements.txt
    RUN pip install --no-cache-dir -r requirements.txt
    
    # Copy your code into the container
    COPY . /app
    
    # If needed, expose a port (not necessary for a background worker)
    # EXPOSE 8000
    
    CMD ["python", "main.py"]
  2. Build the Docker image:

    docker build -t redis-mongo-sync .
  3. Run the container:

    docker run --name sync -d \
      -e REDIS_HOST=your-redis-host \
      -e MONGO_URI=mongodb://your-mongodb:27017 \
      redis-mongo-sync
    • If you rely on the .env file inside your container, ensure it’s copied into the image (as in the Dockerfile example) or mount it at runtime.

Check docker logs sync to view the script logs.


Contributing

  1. Fork the repository.
  2. Create a feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -m 'Add some feature'
  4. Push to your branch: git push origin my-new-feature
  5. Open a pull request.

Feedback, bug reports, and feature requests are welcome in the issue tracker.


Enjoy your Redis ↔ MongoDB syncing!

About

A script to automatically sync vcons from Redis into Mongo

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published