Mini MDS

Lightweight, open source, locally-hosted Modern Data Stack

• Extract and Load: Polars and dlt
• Data Quality: Pandera
• Storage: DuckDB
• Transformation: dbt
• Orchestration: Prefect
• Visualization: Dash

Installation

Prerequisites: Install git and uv.

Clone repository and change directory:

git clone https://github.com/esadek/mini-mds.git
cd mini-mds

Usage

Extract, validate, load and transform data:

uv run prefect/elt.py

Visualize data:

uv run dash/app.py

Architecture

flowchart LR
    A(CSV) --> B[Polars]
    subgraph Prefect
        B --> C[Pandera]
        C --> D[dlt]
        E[dbt Core]
    end
    D --> F[(DuckDB)]
    E <--> F
    F --> G[Dash]

Loading

Project Structure

mini-mds
├── .github/                # GitHub workflows
├── dash/                   # Dash application
├── dbt/                    # dbt project
├── duckdb/                 # DuckDB warehouse
├── prefect/                # Prefect workflows
├── .editorconfig           # Editor configuration
├── .gitignore              # Untracked files to ignore
├── .python-version         # Default Python version
├── LICENSE                 # MIT license
├── pyproject.toml          # Project metadata
├── README.md               # Documentation
└── uv.lock                 # Dependency lockfile