A production-grade, crash-consistent Write-Ahead Log engine written in Go. Designed for use as a durability primitive in embedded storage engines, databases, and distributed systems that need an ordered, durable append log.
A Write-Ahead Log ensures that every mutation is recorded on durable storage before it is acknowledged to the caller. On crash or restart, the engine replays the log to reconstruct state. This gives you:
- Crash consistency — no acknowledged write is ever lost
- Ordered replay — mutations are re-applied in the exact order they were originally written
- Atomic checkpoints — periodic snapshots that bound replay cost
This WAL is not a database. It is the durability layer that a database (or any stateful service) plugs into.
| Feature | Details |
|---|---|
| Binary wire format | 22-byte header, big-endian, magic sentinel 0xDEADBA1F |
| Integrity checking | CRC32C (Castagnoli) over header fields + payload |
| Record types | PUT, DELETE, CHECKPOINT, NOOP |
| Durability modes | Sync (fsync per write), Batch (fsync per batch), Async (no fsync) |
| Group commit | Single writer goroutine drains a bounded channel; no per-write mutex |
| Segment lifecycle | OPEN → ACTIVE → ROTATING → READONLY → COMPACTED → DELETED |
| LSN assignment | Atomic uint64, assigned before enqueue — callers know their LSN immediately |
| Recovery modes | Strict, Best-effort, Verify-only |
| Checkpoints | Atomic write (tmp → fsync → rename) to checkpoints/ subdirectory |
| MANIFEST | JSON metadata for fast startup; segments are always authoritative |
| CLI | Full command-line tool: append, inspect, recover, verify, compact, benchmark, stats, start |
| Fake and faulty FS | In-process storage backends for testing and chaos injection |
| Metrics | Structured counters exposed via stats command and HTTP endpoint |
go install github.com/Mohith1612/wal/cmd/wal@latestgit clone https://github.com/Mohith1612/wal
cd wal
go build ./...
go test ./...import (
"context"
"github.com/Mohith1612/wal/pkg/wal"
"go.uber.org/zap"
)
cfg := wal.DefaultConfig("/var/lib/myapp/wal")
cfg.SyncPolicy.Mode = wal.DurabilityBatch
engine, err := wal.Open(cfg, zap.NewExample())
if err != nil {
log.Fatal(err)
}
defer engine.Close()
lsn, err := engine.Append(context.Background(), wal.RecordTypePUT, []byte("key"), []byte("value"))
if err != nil {
log.Fatal(err)
}
fmt.Printf("written at LSN %d\n", lsn)All commands accept --dir to specify the WAL directory (default: ./wal-data).
Write a single record.
wal append --dir /data/wal --key mykey --value myvalue [--type put|delete]
Decode and print records from a segment file.
wal inspect --dir /data/wal [--hex]
--hex prints the raw bytes of each record alongside the decoded fields.
Replay all segments and print the recovered state.
wal recover --dir /data/wal [--verbose] [--mode strict|best-effort|verify-only]
--verbose prints each replayed record. Exits non-zero if recovery fails in strict mode.
Check all segments for corruption. Suitable for use in cron jobs or health checks.
wal verify --dir /data/wal
Exit codes: 0 = clean, 1 = corruption detected, 2 = I/O error.
Run compaction: recover state, write a checkpoint, and prune superseded segments.
wal compact --dir /data/wal
Run a write throughput benchmark and report results.
wal benchmark --dir /data/wal [--json]
--json emits a machine-readable JSON result for CI pipelines.
Print runtime metrics: segment count, LSN range, queue depth, bytes written.
wal stats --dir /data/wal
Start an HTTP server that exposes the WAL over a REST-like API.
wal start --dir /data/wal [--addr :8080]
| Mode | fsync behavior | Throughput (laptop) | Data loss on crash |
|---|---|---|---|
sync |
fsync after every single write | ~260 ops/sec | None — every ACK is durable |
batch |
fsync after each batch flush | ~100K ops/sec | Writes in last unflushed batch |
async |
No fsync | ~100K ops/sec | Writes since last OS writeback |
The sync throughput number (~260 ops/sec) reflects real fsync(2) latency on a laptop with an NVMe SSD. On a server with a battery-backed RAID controller, sync mode can reach 10K–50K ops/sec. Batch and async modes are I/O-bound only by the OS page cache.
Measured on a commodity laptop (NVMe SSD, Linux 6.8, Go 1.22), 64-byte values:
BenchmarkAppendSync ~260 ops/sec (real fsync per write)
BenchmarkAppendBatch ~100,000 ops/sec (fsync per batch)
BenchmarkAppendAsync ~100,000 ops/sec (page-cache write only)
BenchmarkRecovery1K replay 1,000 records
BenchmarkRecovery100K replay 100,000 records
Run benchmarks locally:
go test -bench=. -benchtime=10s ./benchmarks/ Caller goroutines (N)
|
| Append(ctx, type, key, value)
| 1. atomic LSN increment
| 2. Submit to bounded channel queue
| 3. block on req.Done channel
v
┌─────────────────────────────────────┐
│ buffer.Buffer │
│ │
│ chan *WriteRequest (bounded depth) │
│ | │
│ single writer goroutine │
│ - dequeues requests │
│ - drains queue non-blocking │
│ - builds WriteBatch │
│ - calls segmentFlusher.Flush │
│ - closes req.Done on ACK │
└─────────────────────────────────────┘
|
| WriteBatch
v
┌─────────────────────────────────────┐
│ segmentFlusher.Flush │
│ │
│ 1. EncodeRecord() each entry │
│ 2. active segment Write(encoded) │
│ 3. check size → Rotate(nextLSN)? │
│ 4. active.Sync() per SyncPolicy │
│ 5. update lastLSN atomic │
└─────────────────────────────────────┘
|
v
┌─────────────────────────────────────┐
│ segment.FileSegment │
│ │
│ wal-000001-0000000000000001.log │
│ wal-000002-0000000000010001.log │
│ ... │
└─────────────────────────────────────┘
|
| (on crash / restart)
v
┌─────────────────────────────────────┐
│ recovery.Recoverer │
│ │
│ 1. scan & sort segments by ID │
│ 2. SegmentReader.Next() per record │
│ 3. verify magic + CRC32C │
│ 4. apply PUT/DELETE to state map │
│ 5. return RecoveryResult + LastLSN │
└─────────────────────────────────────┘
github.com/Mohith1612/wal/
├── cmd/wal/ CLI entry point and subcommands
├── pkg/wal/ Public API (thin wrapper over internal/wal)
├── internal/
│ ├── types/ Shared types: LSN, RecordType, sentinel errors
│ ├── checksum/ CRC32C (Castagnoli) computation
│ ├── storage/ FS abstraction: OSFS, FakeFS, FaultyFS
│ ├── segment/ Wire format, encoder, reader, manager, manifest
│ ├── buffer/ Bounded channel queue + single writer goroutine
│ ├── wal/ Engine: wires buffer + segment manager
│ ├── recovery/ Replay engine: strict / best-effort / verify-only
│ ├── compaction/ Checkpoint write + segment pruning
│ ├── metrics/ Counters and gauges
│ ├── chaos/ Fault injection for tests
│ └── benchmark/ Benchmark harness
└── benchmarks/ Go benchmark suite
# Run all tests
go test ./...
# Run tests with race detector
go test -race ./...
# Run benchmarks (10 seconds each)
go test -bench=. -benchtime=10s ./benchmarks/
# Build the CLI
go build -o bin/wal ./cmd/wal- Architecture — write path, recovery path, compaction, LSN assignment, segment lifecycle
- Record Format — byte-level field table, CRC32C coverage, hex dumps
- Disk Layout — directory structure, file roles
- Durability — write visibility lifecycle, mode comparison
- Tradeoffs — engineering decisions and their rationale
- Limitations — known constraints and sharp edges
- Future Extensions — replication, compression, parallel replay