Gopher - Distributed Task Queue for Go

A robust, distributed task queue in Go with Redis backend, designed for reliable asynchronous job execution

Supports prioritization, scheduling, retries, dead-letter queues, monitoring, and CLI management for production-grade workloads

---

✨ Features

🎯 Core Functionality

• ⚡ HTTP API for job submission and management
• 🔝 Priority queues: high, normal, and low
• ⏰ Scheduled & recurring jobs
• 🔄 Automatic retries with configurable limits
• 💀 Dead Letter Queue for failed jobs
• 🐌 Rate limiting per job type
• 📊 Monitoring: Prometheus metrics & health endpoints
• 🔗 Distributed tracing via OpenTelemetry
• 🛑 Graceful shutdown ensures no jobs are lost
• 🛠️ CLI tool for queue management

🛠️ Developer Experience

• 🐳 Docker Compose support for easy setup
• ⚙️ Configurable concurrency and worker settings
• 📦 Modular architecture for easy extension
• 📘 Example job handlers for email, math, and image processing

---

📄 Workflow & Architecture Documentation

For a detailed step-by-step explanation of Gopher's workflow, job lifecycle, and real-world use cases, see ARCHITECTURE.md.

---

📦 Installation

Clone & Build

git clone https://github.com/aneeshsunganahalli/Gopher.git
cd Gopher
make build

Using Docker Compose

docker-compose up -d

---

🚀 Quick Start

Start Redis Backend

docker run --rm -p 6379:6379 --name redis-job-queue redis:7-alpine

Start Gopher Server

# Using Go
go run ./cmd/server/main.go

# Or built binary
./bin/server

Start Worker Process

go run ./cmd/worker/main.go
# Or binary
./bin/worker

Use CLI Tool

# Submit a job
go run ./cmd/cli/cli.go submit -t email -p '{"to":"[email protected]","subject":"Hello","body":"This is a test"}'

# Check queue stats
go run ./cmd/cli/cli.go stats

# Retry failed jobs
go run ./cmd/cli/cli.go retry-all

---

⚙️ Configuration

Gopher uses environment variables for server, Redis, and worker settings:

# Server
SERVER_PORT=8080
SERVER_HOST=localhost
SERVER_READ_TIMEOUT=10s
SERVER_WRITE_TIMEOUT=10s

# Redis
REDIS_URL=redis://localhost:6379
REDIS_PASSWORD=
REDIS_DB=0
REDIS_TIMEOUT=5s

# Worker
WORKER_CONCURRENCY=5
WORKER_POLL_INTERVAL=1s
WORKER_MAX_RETRIES=3
WORKER_SHUTDOWN_TIMEOUT=30s

# Logging
LOG_LEVEL=info
LOG_FORMAT=console

---

📁 Project Structure

Gopher/
├── cmd/
│   ├── server/
│   ├── worker/
│   └── cli/
├── pkg/
│   ├── handlers/
│   ├── types/
│   └── queue/
├── configs/
│   └── config.example.env
├── Dockerfile
├── docker-compose.yml
├── Makefile
└── README.md

---

📬 Example Job Submission

# Email job
curl -X POST http://localhost:8080/api/v1/jobs \
-H "Content-Type: application/json" \
-d '{
  "type": "email",
  "payload": {"to":"[email protected]","subject":"Hello","body":"Test email"},
  "priority": "high",
  "max_retries": 3
}'

# Scheduled job
curl -X POST http://localhost:8080/api/v1/jobs \
-H "Content-Type: application/json" \
-d '{
  "type": "report",
  "payload": {"report_type":"daily_summary"},
  "execute_at":"2025-10-01T10:00:00Z"
}'

# Recurring job
curl -X POST http://localhost:8080/api/v1/jobs \
-H "Content-Type: application/json" \
-d '{
  "type": "cleanup",
  "payload": {},
  "recurring": {"cron_expression":"0 0 * * *"}
}'

---

💡 Best Practices

• 🏗️ Idempotent jobs to prevent duplicate processing
• 📦 Keep payloads small; use external storage for large files
• ⏱️ Timeout handling in job handlers
• 🛑 Graceful shutdown of workers
• ⚠️ Error classification: transient vs permanent
• 📊 Monitor queues and setup alerts
• 🐌 Rate limiting to avoid overloading services

---

⚡ Requirements

---

📄 License

This project is licensed under the MIT License.