Comprehensive benchmark suite for LLM, embedding, and reranking models across Ollama, GGUF, ModelScope, and HuggingFace backends
- 🧠 LLM Benchmarks: MMLU, MMLU-Pro, HellaSwag, ARC-Easy/Challenge, GSM8K, TruthfulQA (MC1/MC2), C-Eval, CMMLU, HumanEval, BBH
- 📊 Embedding Benchmarks: MTEB Classification, Clustering, Retrieval, STS; C-MTEB (Chinese)
- 🔄 Reranking Benchmarks: LLM Pointwise Reranking, Embedding Reranking, LLM Listwise Reranking
- 🔌 Multi-Backend: Ollama, GGUF (llama-cpp-python), ModelScope, HuggingFace
- ⚡ Interactive Selection: Choose which models and benchmarks to run
- 📋 Structured Output: JSON output, rich tables, and ToolResult API pattern
- 🔧 Agent Integration: OpenAI function-calling tools for AI agent use
- 🎮 GPU VRAM-Aware: Auto-detect GPU, filter models that fit in VRAM
- Python 3.10+
- For Ollama backend: Ollama running locally (default: http://localhost:11434)
- For GGUF backend:
llama-cpp-pythonwith CUDA support - For ModelScope backend:
modelscope+llama-cpp-python - For HuggingFace backend:
huggingface_hub+llama-cpp-python - Internet connection (for dataset download, first run only)
pip install -e .With specific backends:
pip install -e ".[ollama]" # Ollama backend
pip install -e ".[gguf]" # GGUF backend
pip install -e ".[modelscope]" # ModelScope backend
pip install -e ".[huggingface]" # HuggingFace backend
pip install -e ".[all]" # All backends
pip install -e ".[dev]" # Dev dependencies# One-command auto benchmark: detect GPU, pick models, run all applicable tests
biwu auto
# Auto benchmark without interactive prompt (GPU-fitable models only)
biwu auto --no-confirm
# Override VRAM for auto benchmark
biwu auto --vram 24 --no-confirm
# List available benchmarks
biwu list benchmarks
# List available models on Ollama
biwu list models
# List models that fit in GPU VRAM
biwu list models --gpu-only
# Show GPU info and model VRAM analysis
biwu gpu
# Run all LLM benchmarks on a model (Ollama)
biwu run -m llama3 --category llm
# Run benchmarks on a GGUF model
biwu run -m /path/to/model.gguf --backend gguf --category llm
# Run benchmarks on a ModelScope model
biwu run -m Qwen/Qwen2-7B-Instruct-GGUF --backend modelscope --category llm
# Run benchmarks on a HuggingFace model
biwu run -m TheBloke/Llama-2-7B-GGUF --backend huggingface --category llm
# Run specific benchmarks
biwu run -m llama3 -b mmlu hellaswag gsm8k
# Auto-select GPU-fitable models and run benchmarks
biwu run --gpu-only --category llm
# Run full suite (all categories)
biwu suite -m llama3
# Run suite only on models that fit in VRAM
biwu suite --gpu-only
# Run with limited samples for quick testing
biwu run -m llama3 -b mmlu -n 100
# Output results to JSON file
biwu run -m llama3 --category llm --json -o results.jsonbiwu list benchmarks
biwu list benchmarks --category llm
biwu list benchmarks --category embedding
biwu list benchmarks --category reranking# List models on Ollama (default backend)
biwu list models
# List models that fit in GPU VRAM
biwu list models --gpu-only
# List models for a specific backend
biwu list models --backend gguf
biwu list models --backend modelscopebiwu gpu
biwu gpu --vram 24
biwu gpu --json# Interactive mode
biwu auto
# Non-interactive: auto-select all GPU-fitable models
biwu auto --no-confirm
# Override VRAM
biwu auto --vram 16 --no-confirm
# Use a specific backend
biwu auto --backend gguf --no-confirm
biwu auto --backend modelscope --no-confirmThe auto command:
- Detects your GPU VRAM via
nvidia-smi - Lists models with VRAM estimates
- Presents an interactive multi-select menu (GPU-fitable models pre-selected)
- Matches each model to its applicable benchmarks (LLM/embedding/reranking)
- Runs all benchmarks and outputs results
Interactive selection: 1,3,5 or 1-3, all, gpu (default), q
# Ollama (default)
biwu run -m llama3 -b mmlu hellaswag
# GGUF
biwu run -m /path/to/model.gguf --backend gguf --category llm
# ModelScope
biwu run -m Qwen/Qwen2-7B-Instruct-GGUF --backend modelscope --category llm
# HuggingFace
biwu run -m TheBloke/Llama-2-7B-GGUF --backend huggingface --category llm
# Multiple models
biwu run -m llama3 mistral qwen2 --category llm
# GPU-only models
biwu run --gpu-only --category llm
# Custom Ollama host
biwu run -m llama3 -b mmlu --host http://192.168.1.100:11434biwu suite -m llama3
biwu suite --gpu-only
biwu suite -m nomic-embed-text --category embedding
biwu suite -m jina-reranker-v2-small --category reranking
biwu suite -m model.gguf --backend gguf --category full| Flag | Description |
|---|---|
-V, --version |
Show version |
-v, --verbose |
Verbose output (debug logging) |
-o, --output |
Output to file path (JSON) |
--json |
JSON output format |
-q, --quiet |
Suppress non-essential output |
from biwu import ToolResult, list_benchmarks, run_benchmark, run_suite, discover_models, gpu_info, auto_run
# List benchmarks
result = list_benchmarks(category="llm")
print(result.success) # True
print(result.data) # List of benchmark info dicts
# Discover models (Ollama by default)
models = discover_models()
print(models.data) # List of model info dicts
# Discover models on specific backend
models = discover_models(backend="gguf")
models = discover_models(backend="modelscope")
models = discover_models(backend="huggingface")
# GPU-only models
models = discover_models(gpu_only=True)
# GPU information
info = gpu_info()
print(info.data["gpus"])
print(info.data["gpu_fitable_models"])
print(info.data["offload_required_models"])
# Auto-detect hardware, select GPU-fitable models, run all benchmarks
result = auto_run(gpu_only=True)
# Run a benchmark (Ollama)
result = run_benchmark(benchmark_name="mmlu", model="llama3")
# Run a benchmark (GGUF)
result = run_benchmark(benchmark_name="mmlu", model="/path/to/model.gguf", backend="gguf")
# Run a benchmark (ModelScope)
result = run_benchmark(benchmark_name="mmlu", model="Qwen/Qwen2-7B-Instruct-GGUF", backend="modelscope")
# Run a suite
result = run_suite(models=["llama3", "mistral"], category="llm")from biwu.tools import TOOLS, dispatch
# Use TOOLS in your OpenAI function-calling setup
result = dispatch("biwu_run_benchmark", {
"benchmark_name": "mmlu",
"model": "llama3"
})
# Auto benchmark
result = dispatch("biwu_auto_run", {"gpu_only": True})
# GPU info
result = dispatch("biwu_gpu_info", {})
# Discover models (with backend)
result = dispatch("biwu_discover_models", {"backend": "ollama", "gpu_only": True})
# Discover GGUF models
result = dispatch("biwu_discover_models", {"backend": "gguf"})| Benchmark | Reference | Metric | Description |
|---|---|---|---|
| MMLU | Hendrycks et al., 2020 | Accuracy | 57 subjects, multiple-choice |
| MMLU-Pro | Wang et al., 2024 | Accuracy | Harder MMLU with 10 choices |
| HellaSwag | Zellers et al., 2019 | Accuracy | Commonsense NLI sentence completion |
| ARC-Easy | Clark et al., 2018 | Accuracy | Grade-school science (easy) |
| ARC-Challenge | Clark et al., 2018 | Accuracy | Grade-school science (hard) |
| GSM8K | Cobbe et al., 2021 | Accuracy | Math word problems |
| TruthfulQA MC1 | Lin et al., 2021 | Accuracy | Single-true truthfulness |
| TruthfulQA MC2 | Lin et al., 2021 | Accuracy | Multiple-true truthfulness |
| C-Eval | Huang et al., 2023 | Accuracy | Chinese multi-discipline |
| CMMLU | Li et al., 2023 | Accuracy | Chinese massive multitask |
| HumanEval | Chen et al., 2021 | pass@1 | Code generation |
| BBH | Suzgun et al., 2022 | Accuracy | Hard reasoning tasks |
| Benchmark | Reference | Metric | Description |
|---|---|---|---|
| embed_classification | Muennighoff et al., 2022 | Accuracy | k-NN classification |
| embed_clustering | Muennighoff et al., 2022 | V-measure | k-Means clustering |
| embed_retrieval | Muennighoff et al., 2022 | NDCG@10 | Cosine-similarity retrieval |
| embed_sts | Muennighoff et al., 2022 | Spearman | Semantic textual similarity |
| cmteb | Xiao et al., 2023 | Accuracy | Chinese MTEB |
| Benchmark | Reference | Metric | Description |
|---|---|---|---|
| llm_reranking | Askari et al., 2023 | NDCG@5 | LLM pointwise reranking |
| embed_reranking | Muennighoff et al., 2022 | NDCG@5 | Embedding cosine reranking |
| llm_listwise_reranking | Askari et al., 2023 | NDCG@5 | LLM listwise reranking |
All benchmark papers are available in the pdf/ directory:
| Paper | arXiv ID |
|---|---|
| MMLU | 2009.03300 |
| HellaSwag | 1905.07830 |
| ARC | 1803.05457 |
| GSM8K | 2110.14168 |
| TruthfulQA | 2109.07958 |
| CMMLU | 2306.09212 |
| C-Eval | 2305.08322 |
| HumanEval | 2107.03374 |
| MMLU-Pro | 2406.01564 |
| BBH | 2206.04615 |
| HELM | 2211.09110 |
| MTEB | 2210.07316 |
| C-MTEB | 2307.09371 |
| BEIR | 2104.08663 |
| RankLLM | 2310.18548 |
| Jina ColBERT | 2402.14759 |
pip install -e ".[dev]"
pytest tests/ -v
ruff format . && ruff check .
mypy biwu/GPL-3.0-or-later