diff --git a/README.md b/README.md index 9286a8d8a..60d877be3 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@

semble logo
Fast and Accurate Code Search for Agents
- Uses ~98% fewer tokens than grep+read + Uses ~99% fewer tokens than grep+read

@@ -24,7 +24,7 @@
-Semble is a code search library built for agents. It returns the exact code snippets they need instantly, using ~98% fewer tokens than grep+read. Indexing and searching a full codebase end-to-end takes under a second, with ~200x faster indexing and ~10x faster queries than a code-specialized transformer, at 99% of its retrieval quality (see [benchmarks](#benchmarks)). Everything runs on CPU with no API keys, GPU, or external services. Use it as an MCP server, a CLI tool via AGENTS.md, or a dedicated sub-agent, and any coding agent (Claude Code, Cursor, Codex, OpenCode, etc.) gets instant access to any repo. +Semble is a code search library built for agents. It returns the exact code snippets they need instantly, using ~99% fewer tokens than grep+read. Indexing and searching a full codebase end-to-end takes under a second, matching the retrieval quality of a code-specialized transformer while indexing ~220x faster and querying ~17x faster (see [benchmarks](#benchmarks)). Everything runs on CPU with no API keys, GPU, or external services. Use it as an MCP server, a CLI tool via AGENTS.md, or a dedicated sub-agent, and any coding agent (Claude Code, Cursor, Codex, OpenCode, etc.) gets instant access to any repo. ## Quickstart @@ -72,9 +72,9 @@ semble install --agent claude --type mcp subagent --yes ## Main Features -- **Fast**: indexes an average repo in ~250 ms and answers queries in ~1.5 ms, all on CPU. +- **Fast**: indexes an average repo in ~500 ms and answers queries in ~1 ms, all on CPU. - **Accurate**: NDCG@10 of 0.854 on our [benchmarks](#benchmarks), on par with code-specialized transformer models, at a fraction of the size and cost. -- **Token-efficient**: returns only the relevant chunks, using [~98% fewer tokens than grep+read](#benchmarks). +- **Token-efficient**: returns only the relevant chunks, using [~99% fewer tokens than grep+read](#benchmarks). - **Zero setup**: runs on CPU with no API keys, GPU, or external services required. - **MCP server**: works with Claude Code, Cursor, Codex, OpenCode, VS Code, and any other MCP-compatible agent. - **Local and remote**: pass a local path or a git URL. @@ -235,7 +235,7 @@ We benchmark quality and speed across ~1,250 queries over 63 repositories in 19 -The quality benchmark (left) scores retrieval quality (NDCG@10) against total latency; semble achieves 99% of the quality of the 137M-parameter [CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) Hybrid while indexing 218x faster. The token efficiency benchmark (right) measures how many tokens each method needs to reach a given recall level; semble uses 98% fewer tokens on average and hits 94% recall at only 2k tokens, while grep+read needs a full 100k context window to reach 85%. See [benchmarks](benchmarks/README.md) for per-language results, ablations, and full methodology. +The quality benchmark (left) scores retrieval quality (NDCG@10) against total latency; semble matches the quality of the 137M-parameter [CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) while indexing 220x faster. The token efficiency benchmark (right) measures how many tokens each method needs to reach a given recall level; semble uses 99% fewer tokens on average and hits 97% recall at only 2k tokens, while grep+read needs a full 100k context window to reach 85%. See [benchmarks](benchmarks/README.md) for per-language results, ablations, and full methodology. ## How it works diff --git a/assets/images/speed_vs_ndcg_cold.png b/assets/images/speed_vs_ndcg_cold.png index 2056b9681..5aebed3b2 100644 Binary files a/assets/images/speed_vs_ndcg_cold.png and b/assets/images/speed_vs_ndcg_cold.png differ diff --git a/assets/images/speed_vs_ndcg_warm.png b/assets/images/speed_vs_ndcg_warm.png index 4d97d2322..333273181 100644 Binary files a/assets/images/speed_vs_ndcg_warm.png and b/assets/images/speed_vs_ndcg_warm.png differ diff --git a/assets/images/token_efficiency.png b/assets/images/token_efficiency.png index df33e8ea1..6d663511f 100644 Binary files a/assets/images/token_efficiency.png and b/assets/images/token_efficiency.png differ diff --git a/benchmarks/README.md b/benchmarks/README.md index dd1335d62..72d365595 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -15,22 +15,24 @@ Quality and speed benchmarks for `semble`. Quality and speed across all methods. -| Method | NDCG@10 | Index | Query p50 | -|---|---:|---:|---:| -| CodeRankEmbed Hybrid | 0.862 | 57 s | 16 ms | -| **semble** | **0.854** | **263 ms** | **1.5 ms** | -| CodeRankEmbed | 0.765 | 57 s | 16 ms | -| ColGREP | 0.693 | 5.8 s | 124 ms | -| BM25 | 0.673 | 263 ms | 0.02 ms | -| grepai | 0.561 | 35 s | 48 ms | -| probe | 0.387 | — | 207 ms | -| ripgrep | 0.126 | — | 12 ms | +| Method | NDCG@10 | Index | Query p50 | +| -------------------- | --------: | ---------: | ----------: | +| **semble** | **0.854** | **518 ms** | **0.91 ms** | +| CodeRankEmbed | 0.839 | 116 s | 16 ms | +| ColGREP | 0.693 | 5.4 s | 122 ms | +| BM25 | 0.673 | 47 ms | 0.17 ms | +| ck | 0.642 | 96 s | 187 ms | +| codebase-memory-mcp | 0.630 | 454 ms | 46 ms | +| grepai | 0.561 | 35 s | 48 ms | +| probe | 0.387 | — | 207 ms | +| cs | 0.200 | — | 22 ms | +| ripgrep | 0.126 | — | 14 ms | | ![Speed vs quality (cold)](../assets/images/speed_vs_ndcg_cold.png) | ![Speed vs quality (warm)](../assets/images/speed_vs_ndcg_warm.png) | -|:--:|:--:| -| *Time to first result (index + query) vs NDCG@10* | *Query latency on a warm index vs NDCG@10* | +| :-----------------------------------------------------------------: | :-----------------------------------------------------------------: | +| _Time to first result (index + query) vs NDCG@10_ | _Query latency on a warm index vs NDCG@10_ | -The 137M-param CodeRankEmbed Hybrid wins NDCG@10 by 0.008. semble wins index time by 218x and query latency by 11x. +semble matches the NDCG@10 of the 137M-param CodeRankEmbed while winning index time by ~220x and query latency by ~17x. NDCG@10 is averaged across all queries. Speed numbers use one repo per language, CPU only: cold-start index time and warm query p50 (median across 5 consecutive runs). @@ -44,19 +46,19 @@ Coding agents (Claude Code, OpenCode, etc.) typically find code by running `grep For each query: tokens consumed at first relevant hit, or 32k if the method never finds anything. Averaged across all 1251 queries. -| Method | Expected tokens | Savings | -|---|---:|---:| -| ripgrep + read file | 45,692 | baseline | -| **semble** | **566** | **98% fewer** | +| Method | Expected tokens | Savings | +| ------------------- | --------------: | ------------: | +| ripgrep + read file | 45,587 | baseline | +| **semble** | **348** | **99% fewer** | ### Recall at fixed token budgets A relevant file is "covered" once any retrieved unit comes from it. -| Method | 500 | 1k | 2k | 4k | 8k | 16k | 32k | -|---|---:|---:|---:|---:|---:|---:|---:| -| **semble** | **0.685** | **0.849** | **0.938** | **0.976** | **0.991** | **0.996** | **0.996** | -| ripgrep + read file | 0.001 | 0.008 | 0.037 | 0.088 | 0.212 | 0.379 | 0.583 | +| Method | 500 | 1k | 2k | 4k | 8k | 16k | 32k | +| ------------------- | --------: | --------: | --------: | --------: | --------: | --------: | --------: | +| **semble** | **0.842** | **0.923** | **0.967** | **0.988** | **0.995** | **0.995** | **0.995** | +| ripgrep + read file | 0.001 | 0.008 | 0.037 | 0.086 | 0.207 | 0.374 | 0.583 |
Methodology @@ -67,51 +69,53 @@ semble returns the top-50 ranked chunks. `ripgrep+read` splits the query into ke ## By language -NDCG@10 per language, sorted by CodeRankEmbed Hybrid (CRE in the table). Best score per row is bolded. - -| Language | semble | CRE Hybrid | CRE | ColGREP | grepai | probe | ripgrep | -|---|---:|---:|---:|---:|---:|---:|---:| -| scala | 0.909 | **0.922** | 0.845 | 0.765 | 0.330 | 0.392 | 0.180 | -| cpp | **0.915** | 0.913 | 0.846 | 0.626 | 0.731 | 0.375 | 0.126 | -| ruby | **0.909** | **0.909** | 0.769 | 0.708 | 0.643 | 0.382 | 0.230 | -| elixir | 0.894 | **0.905** | 0.869 | 0.808 | 0.669 | 0.412 | 0.134 | -| javascript | 0.917 | 0.903 | **0.920** | 0.823 | 0.675 | 0.588 | 0.176 | -| zig | **0.913** | 0.901 | 0.807 | 0.474 | 0.755 | 0.369 | 0.000 | -| csharp | 0.885 | **0.889** | 0.743 | 0.614 | 0.277 | 0.392 | 0.117 | -| go | **0.895** | 0.884 | 0.676 | 0.785 | 0.722 | 0.410 | 0.133 | -| python | 0.867 | **0.880** | 0.794 | 0.777 | 0.634 | 0.488 | 0.202 | -| php | 0.858 | **0.874** | 0.758 | 0.663 | 0.402 | 0.340 | 0.123 | -| swift | 0.860 | **0.873** | 0.721 | 0.710 | 0.429 | 0.280 | 0.160 | -| bash | 0.825 | 0.852 | **0.892** | 0.706 | 0.723 | 0.226 | 0.000 | -| lua | 0.823 | **0.847** | 0.803 | 0.798 | 0.699 | 0.336 | 0.000 | -| java | **0.849** | 0.841 | 0.706 | 0.641 | 0.386 | 0.536 | 0.198 | -| kotlin | 0.821 | **0.830** | 0.670 | 0.637 | 0.478 | 0.335 | 0.166 | -| rust | **0.856** | 0.827 | 0.627 | 0.662 | 0.519 | 0.242 | 0.162 | -| c | 0.741 | **0.806** | 0.706 | 0.676 | 0.555 | 0.384 | 0.000 | -| haskell | 0.765 | 0.771 | **0.776** | 0.683 | 0.483 | 0.313 | 0.000 | -| typescript | 0.706 | **0.708** | 0.545 | 0.430 | 0.394 | 0.354 | 0.128 | -| **overall** | **0.854** | **0.862** | **0.765** | **0.693** | **0.561** | **0.387** | **0.126** | +NDCG@10 per language, sorted by CodeRankEmbed (CRE in the table). Best score per row is bolded. + +| Language | semble | CRE | ColGREP | ck | cbm | grepai | probe | cs | ripgrep | +| ----------- | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | +| javascript | 0.917 | **0.925** | 0.823 | 0.772 | 0.770 | 0.675 | 0.588 | 0.171 | 0.176 | +| scala | 0.909 | **0.925** | 0.765 | 0.717 | 0.704 | 0.330 | 0.392 | 0.111 | 0.180 | +| zig | **0.913** | 0.911 | 0.474 | 0.511 | 0.766 | 0.755 | 0.369 | 0.121 | 0.000 | +| ruby | **0.909** | 0.905 | 0.708 | 0.738 | 0.689 | 0.643 | 0.382 | 0.255 | 0.230 | +| cpp | **0.915** | 0.897 | 0.626 | 0.687 | 0.630 | 0.731 | 0.375 | 0.262 | 0.126 | +| elixir | **0.894** | 0.893 | 0.808 | 0.786 | 0.506 | 0.669 | 0.412 | 0.397 | 0.134 | +| python | 0.867 | **0.878** | 0.777 | 0.721 | 0.643 | 0.634 | 0.488 | 0.305 | 0.202 | +| csharp | **0.885** | 0.848 | 0.614 | 0.548 | 0.775 | 0.277 | 0.392 | 0.248 | 0.117 | +| php | **0.858** | 0.847 | 0.663 | 0.615 | 0.608 | 0.402 | 0.340 | 0.180 | 0.123 | +| swift | **0.860** | 0.845 | 0.710 | 0.672 | 0.630 | 0.429 | 0.280 | 0.151 | 0.160 | +| bash | 0.825 | **0.834** | 0.706 | 0.677 | 0.768 | 0.723 | 0.226 | 0.170 | 0.000 | +| lua | 0.823 | **0.829** | 0.798 | 0.738 | 0.591 | 0.699 | 0.336 | 0.050 | 0.000 | +| kotlin | 0.821 | **0.823** | 0.637 | 0.587 | 0.611 | 0.478 | 0.335 | 0.170 | 0.166 | +| haskell | 0.765 | **0.811** | 0.683 | 0.733 | 0.624 | 0.483 | 0.313 | 0.160 | 0.000 | +| java | **0.849** | 0.790 | 0.641 | 0.606 | 0.554 | 0.386 | 0.536 | 0.136 | 0.198 | +| c | 0.741 | **0.771** | 0.676 | 0.606 | 0.655 | 0.555 | 0.384 | 0.175 | 0.000 | +| rust | **0.856** | 0.754 | 0.662 | 0.419 | 0.454 | 0.519 | 0.242 | 0.193 | 0.162 | +| go | **0.895** | 0.713 | 0.785 | 0.458 | 0.506 | 0.722 | 0.410 | 0.183 | 0.133 | +| typescript | **0.706** | 0.671 | 0.430 | 0.456 | 0.455 | 0.394 | 0.354 | 0.145 | 0.128 | +| **overall** | **0.854** | 0.839 | **0.693** | **0.634** | **0.630** | **0.561** | **0.387** | **0.200** | **0.126** | + +cbm = [codebase-memory-mcp](#methods). ## Ablations `raw` returns retrieval scores directly; `+ ranking` feeds them through semble's hybrid ranker. -| Retrieval | Raw | + ranking | -|---|---:|---:| -| BM25 | 0.675 | 0.834 | -| potion-code-16M | 0.650 | 0.821 | -| BM25 + potion-code-16M | — | **0.854** | +| Retrieval | Raw | + ranking | +| ---------------------- | ----: | --------: | +| BM25 | 0.675 | 0.834 | +| potion-code-16M | 0.650 | 0.821 | +| BM25 + potion-code-16M | — | **0.854** |
By query category -| Mode | Architecture | Semantic | Symbol | -|---|---:|---:|---:| -| BM25 raw | 0.628 | 0.676 | 0.719 | -| potion-code-16M raw | 0.626 | 0.666 | 0.629 | -| semble BM25 (+ ranking) | 0.770 | 0.819 | 0.957 | -| semble potion-code-16M (+ ranking) | 0.757 | 0.808 | 0.943 | -| **semble hybrid** | **0.802** | **0.846** | **0.958** | +| Mode | Architecture | Semantic | Symbol | +| ---------------------------------- | -----------: | --------: | --------: | +| BM25 raw | 0.628 | 0.676 | 0.719 | +| potion-code-16M raw | 0.626 | 0.666 | 0.629 | +| semble BM25 (+ ranking) | 0.770 | 0.819 | 0.957 | +| semble potion-code-16M (+ ranking) | 0.757 | 0.808 | 0.943 | +| **semble hybrid** | **0.802** | **0.846** | **0.958** |
@@ -119,11 +123,11 @@ NDCG@10 per language, sorted by CodeRankEmbed Hybrid (CRE in the table). Best sc ~1,250 queries over 63 repositories in 19 languages, grouped into three categories: -| Category | Queries | What it tests | -|---|---:|---| -| semantic | 711 | Code that implements a specific behavior or concept | -| architecture | 343 | Design decisions, module boundaries, structural patterns | -| symbol | 204 | Named entity lookup (function, class, type, variable) | +| Category | Queries | What it tests | +| ------------ | ------: | -------------------------------------------------------- | +| semantic | 711 | Code that implements a specific behavior or concept | +| architecture | 343 | Design decisions, module boundaries, structural patterns | +| symbol | 204 | Named entity lookup (function, class, type, variable) |
Notes @@ -137,18 +141,23 @@ NDCG@10 per language, sorted by CodeRankEmbed Hybrid (CRE in the table). Best sc ## Methods - **[ripgrep](https://github.com/BurntSushi/ripgrep)**: fast regex search over files, included as a raw keyword-match baseline. -- **[probe](https://github.com/buger/probe)**: BM25 keyword ranking backed by tree-sitter parse trees. No persistent index; scans on the fly. +- **[probe](https://github.com/buger/probe)**: BM25 keyword ranking backed by tree-sitter parse trees. We use its free `probe search` mode; its `probe agent` mode (which does query rewriting) needs a paid LLM API key and isn't tested here. +- **[cs (Code Spelunker)](https://github.com/boyter/cs)**: structural BM25 ranker which ranks matches differently in code/comments/strings). - **[ColGREP](https://github.com/lightonai/next-plaid/tree/main/colgrep)**: late-interaction code retrieval built on next-plaid with the [LateOn-Code-edge](https://huggingface.co/lightonai/LateOn-Code-edge) model. - **[grepai](https://github.com/nicholasgasior/grepai)**: semantic search using [nomic-embed-text](https://huggingface.co/nomic-ai/nomic-embed-text-v1) (137M params) via a local Ollama daemon. -- **[CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed)**: 137M-param transformer embedding model for code retrieval. *CodeRankEmbed Hybrid* fuses its dense scores with BM25. +- **[codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)**: code intelligence engine that indexes a repo into a SQLite/graph store. We benchmark its `search_graph` tool in `fast` mode, which does BM25 full-text search with structural boosting. +- **[ck](https://github.com/BeaconBay/ck)**: hybrid regex + semantic search using [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5). +- **[CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed)**: 137M-param transformer embedding model for code retrieval, used for semantic-only dense search. - **[semble](https://github.com/your-repo/semble)**: this library. [potion-code-16M](https://huggingface.co/minishlab/potion-code-16M) static embeddings + BM25 + the semble reranking stack. ## Excluded methods -Two tools were considered but not included in the benchmark: +The following tools were considered but not included in the main comparison: -- **[codanna](https://codanna.io)**: symbol-level semantic search with fastembed. Excluded because it does not support Haskell, Bash, Zig, Scala, Elixir, or Ruby (6 of the 19 benchmark languages, covering 20 of 63 repos (~38% of tasks)). +- **[codanna](https://codanna.io)**: symbol-level semantic search with fastembed. Excluded because it does not support Haskell, Bash, Zig, Scala, Elixir, or Ruby (6 of the 19 benchmark languages). - **[claude-context](https://github.com/zilliztech/claude-context)**: retrieval-augmented code search using OpenAI embeddings and a vector database. Excluded because it requires a paid OpenAI API key and a running vector-DB service. +- **[GitNexus](https://github.com/abhigyanpatwari/GitNexus)**: knowledge-graph code search (BM25 + local embeddings + RRF). Excluded because it does not support Bash, Elixir, Haskell, Lua, Scala, or Zig (6 of the 19 benchmark languages). +- **[codegraph](https://github.com/colbymchenry/codegraph)**: SQLite FTS5 symbol search + graph traversal. Excluded because it does not support Haskell, Bash, Elixir, or Zig (4 of the 19 benchmark languages). ## Running the benchmarks @@ -232,6 +241,44 @@ The `--output` flag enables resume mode: already-completed repos are skipped on
+
+cs (Code Spelunker) + +Needs `cs` on `$PATH` (`go install github.com/boyter/cs/v3@latest`). + +```bash +uv run python -m benchmarks.baselines.cs +uv run python -m benchmarks.baselines.cs --repo fastapi --repo axios +``` + +
+ +
+codebase-memory-mcp + +Needs `codebase-memory-mcp` on `$PATH` (`uv tool install codebase-memory-mcp`, or `pip`/`npm`/`brew`). First run downloads a small platform binary. + +```bash +uv run python -m benchmarks.baselines.codebase_memory +uv run python -m benchmarks.baselines.codebase_memory --repo fastapi --repo axios +``` + +Each repo is indexed under a `semble-bench-` project name and deleted again after evaluation, so it doesn't collide with any projects you have indexed for real use. + +
+ +
+ck + +Needs `ck` on `$PATH` (`cargo install ck-search` or `npm install -g @beaconbay/ck-search`). Requires outbound HTTPS to Hugging Face to download its `BAAI/bge-small-en-v1.5` embedding model on first run. + +```bash +uv run python -m benchmarks.baselines.ck +uv run python -m benchmarks.baselines.ck --repo fastapi --repo axios +``` + +
+
ripgrep @@ -265,7 +312,7 @@ Requires the `benchmark` extra (`uv sync --extra benchmark`). ```bash uv run python -m benchmarks.baselines.coderankembed -uv run python -m benchmarks.baselines.coderankembed --mode semantic +uv run python -m benchmarks.baselines.coderankembed --repo fastapi --repo axios ```
diff --git a/benchmarks/baselines/ck.py b/benchmarks/baselines/ck.py new file mode 100644 index 000000000..2c8b68285 --- /dev/null +++ b/benchmarks/baselines/ck.py @@ -0,0 +1,212 @@ +import argparse +import json +import shutil +import subprocess +import sys +import time +from dataclasses import dataclass +from pathlib import Path + +from benchmarks.data import ( + RepoSpec, + Task, + add_filter_args, + grouped_tasks, + load_filtered_tasks, + save_results, +) +from benchmarks.metrics import file_rank, ndcg_at_k + +_CK = "ck" +_TOP_K = 10 +_LATENCY_RUNS = 3 +_INDEX_TIMEOUT = 1800 +_SEARCH_TIMEOUT = 60 + + +@dataclass(frozen=True) +class RepoResult: + """Per-repo benchmark result.""" + + repo: str + language: str + ndcg10: float + p50_ms: float + index_ms: float + + +def _cleanup_index(benchmark_dir: Path) -> None: + shutil.rmtree(benchmark_dir / ".ck", ignore_errors=True) + (benchmark_dir / ".ckignore").unlink(missing_ok=True) + + +def _build_index(benchmark_dir: Path) -> tuple[bool, float]: + """Build a ck hybrid (BM25 + embedding) index for a repo; return (success, elapsed_ms).""" + _cleanup_index(benchmark_dir) + started = time.perf_counter() + try: + proc = subprocess.run( + [_CK, "--index", "--quiet", str(benchmark_dir)], + capture_output=True, + text=True, + timeout=_INDEX_TIMEOUT, + ) + except subprocess.TimeoutExpired: + print(f" WARNING: ck --index timed out after {_INDEX_TIMEOUT}s", file=sys.stderr) + return False, (time.perf_counter() - started) * 1000 + elapsed_ms = (time.perf_counter() - started) * 1000 + if proc.returncode != 0: + print(f" WARNING: ck --index failed: {proc.stderr.strip()[:300]}", file=sys.stderr) + return False, elapsed_ms + return True, elapsed_ms + + +def _run_search(query: str, benchmark_dir: Path, *, top_k: int) -> list[str]: + """Return absolute file paths from ck's hybrid (regex + semantic) JSON output.""" + cmd = [_CK, "--hybrid", "--json", "--quiet", "--topk", str(top_k), query, str(benchmark_dir)] + try: + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=_SEARCH_TIMEOUT) + except subprocess.TimeoutExpired: + return [] + # ck exits 1 with "No matches found" on stderr (empty stdout) rather than an empty JSON array. + # --json with --quiet is actually JSONL (one object per line), not a wrapped JSON array. + if not proc.stdout.strip(): + return [] + items: list[dict] = [] + for line in proc.stdout.splitlines(): + line = line.strip() + if not line: + continue + try: + items.append(json.loads(line)) + except json.JSONDecodeError: + continue + seen: dict[str, None] = {} + for item in items: + rel = item.get("file", "") + if rel: + abs_path = str((benchmark_dir / rel).resolve()) + seen[abs_path] = None + return list(seen)[:top_k] + + +def _evaluate_repo( + tasks: list[Task], + benchmark_dir: Path, + *, + verbose: bool = False, +) -> tuple[float, float]: + """Return (mean ndcg@10, p50 latency ms) for a list of tasks.""" + ndcg10_sum = 0.0 + latencies: list[float] = [] + + for task in tasks: + query_latencies: list[float] = [] + file_paths: list[str] = [] + for _ in range(_LATENCY_RUNS): + started = time.perf_counter() + file_paths = _run_search(task.query, benchmark_dir, top_k=_TOP_K) + query_latencies.append((time.perf_counter() - started) * 1000) + latencies.append(sorted(query_latencies)[_LATENCY_RUNS // 2]) + + relevant_ranks = [rank for t in task.all_relevant if (rank := file_rank(file_paths, t.path)) is not None] + q_ndcg10 = ndcg_at_k(relevant_ranks, len(task.all_relevant), _TOP_K) + ndcg10_sum += q_ndcg10 + + if verbose: + print( + f" ndcg@10={q_ndcg10:.3f} ranks={relevant_ranks} n_rel={len(task.all_relevant)} q={task.query!r}", + file=sys.stderr, + ) + print(f" targets: {', '.join(t.path for t in task.all_relevant)}", file=sys.stderr) + print(f" top-5: {[Path(fp).name for fp in file_paths[:5]]}", file=sys.stderr) + + latencies.sort() + return ndcg10_sum / len(tasks), latencies[len(latencies) // 2] + + +def _run_repo(spec: RepoSpec, tasks: list[Task], *, verbose: bool) -> RepoResult | None: + """Index, evaluate, and clean up a single repo.""" + benchmark_dir = spec.benchmark_dir + ok, index_ms = _build_index(benchmark_dir) + if not ok: + print(f" SKIP: {spec.name} — ck indexing failed", file=sys.stderr) + _cleanup_index(benchmark_dir) + return None + + try: + ndcg10, p50_ms = _evaluate_repo(tasks, benchmark_dir, verbose=verbose) + finally: + _cleanup_index(benchmark_dir) + + return RepoResult(repo=spec.name, language=spec.language, ndcg10=ndcg10, p50_ms=p50_ms, index_ms=index_ms) + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Benchmark ck on the semble benchmark suite.") + add_filter_args(parser, verbose=True) + return parser.parse_args() + + +def main() -> None: + """Run the ck baseline benchmark.""" + args = _parse_args() + repo_specs, tasks = load_filtered_tasks(args.repo or None, args.language or None) + + print("ck (hybrid: regex + bge-small-en-v1.5 semantic, RRF fusion)", file=sys.stderr) + print(f"{'Repo':<22} {'Language':<12} {'Index':>9} {'NDCG@10':>8} {'p50':>8}", file=sys.stderr) + print(f"{'-' * 22} {'-' * 12} {'-' * 9} {'-' * 8} {'-' * 8}", file=sys.stderr) + + results: list[RepoResult] = [] + for repo, repo_task_list in sorted(grouped_tasks(tasks).items()): + spec = repo_specs[repo] + if args.verbose: + print(f"\n--- {repo} ---", file=sys.stderr) + result = _run_repo(spec, repo_task_list, verbose=args.verbose) + if result is None: + continue + results.append(result) + print( + f"{repo:<22} {spec.language:<12} {result.index_ms:>8.0f}ms {result.ndcg10:>8.3f} {result.p50_ms:>7.1f}ms", + file=sys.stderr, + ) + + if not results: + return + + avg_ndcg10 = sum(r.ndcg10 for r in results) / len(results) + avg_p50 = sum(r.p50_ms for r in results) / len(results) + avg_index = sum(r.index_ms for r in results) / len(results) + print(f"{'-' * 22} {'-' * 12} {'-' * 9} {'-' * 8} {'-' * 8}", file=sys.stderr) + avg_label = f"Average ({len(results)})" + print( + f"{avg_label:<22} {'':<12} {avg_index:>8.0f}ms {avg_ndcg10:>8.3f} {avg_p50:>7.1f}ms", + file=sys.stderr, + ) + + summary = { + "tool": "ck", + "note": "hybrid regex + BAAI/bge-small-en-v1.5 (33M params) semantic search, RRF fusion", + "repos": [ + { + "repo": r.repo, + "language": r.language, + "ndcg10": round(r.ndcg10, 4), + "p50_ms": round(r.p50_ms, 1), + "index_ms": round(r.index_ms, 0), + } + for r in results + ], + "avg_ndcg10": round(avg_ndcg10, 4), + "avg_p50_ms": round(avg_p50, 1), + "avg_index_ms": round(avg_index, 0), + } + print(json.dumps(summary, indent=2)) + + if not args.repo and not args.language: + out = save_results("ck", summary) + print(f"\nResults saved to {out}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/baselines/codebase_memory.py b/benchmarks/baselines/codebase_memory.py new file mode 100644 index 000000000..1a7955b33 --- /dev/null +++ b/benchmarks/baselines/codebase_memory.py @@ -0,0 +1,219 @@ +import argparse +import json +import re +import subprocess +import sys +import time +from dataclasses import dataclass +from pathlib import Path + +from benchmarks.data import ( + RepoSpec, + Task, + add_filter_args, + grouped_tasks, + load_filtered_tasks, + save_results, +) +from benchmarks.metrics import file_rank, ndcg_at_k + +_BIN = "codebase-memory-mcp" +_TOP_K = 10 +_LATENCY_RUNS = 3 +_INDEX_TIMEOUT = 600 +_SEARCH_TIMEOUT = 60 + +# codebase-memory-mcp normalizes non-ASCII/unsafe characters into the project name; +# benchmark repo names are already ASCII-safe slugs, but strip anything it wouldn't accept. +_PROJECT_NAME_RE = re.compile(r"[^a-zA-Z0-9_-]") + + +@dataclass(frozen=True) +class RepoResult: + """Per-repo benchmark result.""" + + repo: str + language: str + ndcg10: float + p50_ms: float + index_ms: float + + +def _project_name(repo: str) -> str: + return f"semble-bench-{_PROJECT_NAME_RE.sub('-', repo)}" + + +def _run_cli(args: list[str], *, timeout: int) -> dict | None: + """Run `codebase-memory-mcp cli ...` and return the parsed structuredContent, or None on failure.""" + cmd = [_BIN, "cli", *args, "--json"] + try: + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + except subprocess.TimeoutExpired: + return None + if proc.returncode != 0: + return None + try: + data = json.loads(proc.stdout) + except json.JSONDecodeError: + return None + if data.get("isError"): + return None + return data.get("structuredContent") + + +def _cleanup_index(project: str) -> None: + _run_cli(["delete_project", "--project", project], timeout=30) + + +def _build_index(benchmark_dir: Path, project: str) -> tuple[bool, float]: + """Index a repo into a fresh graph project; return (success, elapsed_ms).""" + _cleanup_index(project) + started = time.perf_counter() + result = _run_cli( + ["index_repository", "--repo-path", str(benchmark_dir), "--name", project, "--mode", "fast"], + timeout=_INDEX_TIMEOUT, + ) + elapsed_ms = (time.perf_counter() - started) * 1000 + return result is not None, elapsed_ms + + +def _run_search(query: str, benchmark_dir: Path, project: str, *, top_k: int) -> list[str]: + """Return absolute file paths from search_graph's BM25-ranked results.""" + result = _run_cli( + ["search_graph", "--project", project, "--query", query, "--limit", str(top_k)], + timeout=_SEARCH_TIMEOUT, + ) + if result is None: + return [] + seen: dict[str, None] = {} + for item in result.get("results", []): + rel = item.get("file_path", "") + if rel: + abs_path = str((benchmark_dir / rel).resolve()) + seen[abs_path] = None + return list(seen)[:top_k] + + +def _evaluate_repo( + tasks: list[Task], + benchmark_dir: Path, + project: str, + *, + verbose: bool = False, +) -> tuple[float, float]: + """Return (mean ndcg@10, p50 latency ms) for a list of tasks.""" + ndcg10_sum = 0.0 + latencies: list[float] = [] + + for task in tasks: + query_latencies: list[float] = [] + file_paths: list[str] = [] + for _ in range(_LATENCY_RUNS): + started = time.perf_counter() + file_paths = _run_search(task.query, benchmark_dir, project, top_k=_TOP_K) + query_latencies.append((time.perf_counter() - started) * 1000) + latencies.append(sorted(query_latencies)[_LATENCY_RUNS // 2]) + + relevant_ranks = [rank for t in task.all_relevant if (rank := file_rank(file_paths, t.path)) is not None] + q_ndcg10 = ndcg_at_k(relevant_ranks, len(task.all_relevant), _TOP_K) + ndcg10_sum += q_ndcg10 + + if verbose: + print( + f" ndcg@10={q_ndcg10:.3f} ranks={relevant_ranks} n_rel={len(task.all_relevant)} q={task.query!r}", + file=sys.stderr, + ) + print(f" targets: {', '.join(t.path for t in task.all_relevant)}", file=sys.stderr) + print(f" top-5: {[Path(fp).name for fp in file_paths[:5]]}", file=sys.stderr) + + latencies.sort() + return ndcg10_sum / len(tasks), latencies[len(latencies) // 2] + + +def _run_repo(spec: RepoSpec, tasks: list[Task], *, verbose: bool) -> RepoResult | None: + """Index, evaluate, and clean up a single repo.""" + project = _project_name(spec.name) + benchmark_dir = spec.benchmark_dir + ok, index_ms = _build_index(benchmark_dir, project) + if not ok: + print(f" SKIP: {spec.name} — codebase-memory-mcp indexing failed", file=sys.stderr) + _cleanup_index(project) + return None + + try: + ndcg10, p50_ms = _evaluate_repo(tasks, benchmark_dir, project, verbose=verbose) + finally: + _cleanup_index(project) + + return RepoResult(repo=spec.name, language=spec.language, ndcg10=ndcg10, p50_ms=p50_ms, index_ms=index_ms) + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Benchmark codebase-memory-mcp on the semble benchmark suite.") + add_filter_args(parser, verbose=True) + return parser.parse_args() + + +def main() -> None: + """Run the codebase-memory-mcp baseline benchmark.""" + args = _parse_args() + repo_specs, tasks = load_filtered_tasks(args.repo or None, args.language or None) + + print("codebase-memory-mcp (search_graph: BM25 + structural boosting)", file=sys.stderr) + print(f"{'Repo':<22} {'Language':<12} {'Index':>9} {'NDCG@10':>8} {'p50':>8}", file=sys.stderr) + print(f"{'-' * 22} {'-' * 12} {'-' * 9} {'-' * 8} {'-' * 8}", file=sys.stderr) + + results: list[RepoResult] = [] + for repo, repo_task_list in sorted(grouped_tasks(tasks).items()): + spec = repo_specs[repo] + if args.verbose: + print(f"\n--- {repo} ---", file=sys.stderr) + result = _run_repo(spec, repo_task_list, verbose=args.verbose) + if result is None: + continue + results.append(result) + print( + f"{repo:<22} {spec.language:<12} {result.index_ms:>8.0f}ms {result.ndcg10:>8.3f} {result.p50_ms:>7.1f}ms", + file=sys.stderr, + ) + + if not results: + return + + avg_ndcg10 = sum(r.ndcg10 for r in results) / len(results) + avg_p50 = sum(r.p50_ms for r in results) / len(results) + avg_index = sum(r.index_ms for r in results) / len(results) + print(f"{'-' * 22} {'-' * 12} {'-' * 9} {'-' * 8} {'-' * 8}", file=sys.stderr) + avg_label = f"Average ({len(results)})" + print( + f"{avg_label:<22} {'':<12} {avg_index:>8.0f}ms {avg_ndcg10:>8.3f} {avg_p50:>7.1f}ms", + file=sys.stderr, + ) + + summary = { + "tool": "codebase-memory-mcp", + "note": "search_graph: SQLite FTS5 BM25 with structural boosting (Function/Method +10, Route +8, Class +5); " + "'fast' index mode (no embeddings)", + "repos": [ + { + "repo": r.repo, + "language": r.language, + "ndcg10": round(r.ndcg10, 4), + "p50_ms": round(r.p50_ms, 1), + "index_ms": round(r.index_ms, 0), + } + for r in results + ], + "avg_ndcg10": round(avg_ndcg10, 4), + "avg_p50_ms": round(avg_p50, 1), + "avg_index_ms": round(avg_index, 0), + } + print(json.dumps(summary, indent=2)) + + if not args.repo and not args.language: + out = save_results("codebase-memory-mcp", summary) + print(f"\nResults saved to {out}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/baselines/coderankembed.py b/benchmarks/baselines/coderankembed.py index c93da095f..921aefee0 100644 --- a/benchmarks/baselines/coderankembed.py +++ b/benchmarks/baselines/coderankembed.py @@ -18,39 +18,63 @@ load_filtered_tasks, results_path, save_results, - summarize_modes, ) from benchmarks.metrics import ndcg_at_k, target_rank -from semble import SembleIndex -from semble.types import SearchResult +from semble.index.create import create_index_from_path +from semble.index.index import SembleIndex +from semble.types import ContentType, SearchResult _MODEL_NAME = "nomic-ai/CodeRankEmbed" _TOP_K = 10 _LATENCY_RUNS = 3 # transformer inference is slow; keep runs low +_ALPHA = 1.0 # SembleIndex.search()'s alpha: 1.0 = full semantic weight + +_UNSET = object() class _AsymmetricWrapper: - """Wrap SentenceTransformer with asymmetric query/document prompts.""" + """Wrap SentenceTransformer with asymmetric query/document prompts. + + semble only passes use_multiprocessing during index-time calls, never at query time, so its + presence is a reliable query/document discriminator (batch size is not: single-chunk files are + real one-element document batches). + """ def __init__(self, model: SentenceTransformer, max_seq_length: int = 512) -> None: self._model = model self._model.max_seq_length = max_seq_length - def encode(self, texts: Sequence[str]) -> np.ndarray: - """Encode texts with query or document prompt based on batch size.""" + def encode(self, texts: Sequence[str], use_multiprocessing: object = _UNSET) -> np.ndarray: + """Encode with the query prompt only when use_multiprocessing wasn't passed.""" text_list = list(texts) - if len(text_list) == 1: + if use_multiprocessing is _UNSET: return self._model.encode(text_list, prompt_name="query", batch_size=1) # type: ignore[return-value] return self._model.encode(text_list, batch_size=1) # type: ignore[return-value] +def _build_index(benchmark_dir: Path, model: _AsymmetricWrapper) -> SembleIndex: + """Build a SembleIndex using CodeRankEmbed embeddings for both BM25 enrichment and dense search.""" + bm25_index, semantic_index, chunks, _manifest = create_index_from_path( + benchmark_dir, + model=model, # type: ignore[arg-type] + content=(ContentType.CODE,), # type: ignore[arg-type] + ) + return SembleIndex( + model=model, # type: ignore[arg-type] + bm25_index=bm25_index, + semantic_index=semantic_index, + chunks=chunks, + model_path=_MODEL_NAME, + root=benchmark_dir, + ) + + @dataclass(frozen=True) class RepoResult: - """Per-repo benchmark result for one search mode.""" + """Per-repo benchmark result.""" repo: str language: str - mode: str chunks: int ndcg5: float ndcg10: float @@ -77,7 +101,7 @@ def _evaluate( results: list[SearchResult] = [] for _ in range(_LATENCY_RUNS): started = time.perf_counter() - results = index.search(task.query, top_k=_TOP_K) + results = index.search(task.query, top_k=_TOP_K, alpha=_ALPHA) query_latencies.append((time.perf_counter() - started) * 1000) latencies.append(float(np.median(query_latencies))) @@ -107,27 +131,26 @@ def _evaluate( return ndcg5_sum / total, ndcg10_sum / total, latencies, by_category -def _build_summary(results: list[RepoResult], modes: list[str]) -> dict[str, object]: +def _build_summary(results: list[RepoResult]) -> dict[str, object]: """Build the JSON summary dict from the current (possibly partial) results list.""" + n = len(results) return { "tool": "coderankembed", - "model": _MODEL_NAME, - "by_mode": summarize_modes(results, modes), + "note": f"{_MODEL_NAME} (137M params), semantic-only dense search (alpha=1.0)", "repos": [asdict(result) for result in results], + "avg_ndcg10": round(sum(r.ndcg10 for r in results) / n, 4) if n else 0.0, + "avg_p50_ms": round(sum(r.p50_ms for r in results) / n, 1) if n else 0.0, + "avg_index_ms": round(sum(r.index_ms for r in results) / n, 1) if n else 0.0, } -def _load_completed(out_path: Path, modes: list[str]) -> dict[str, list[RepoResult]]: - """Load repos where all requested modes are already saved in a previous run.""" +def _load_completed(out_path: Path) -> dict[str, RepoResult]: + """Load repos already saved in a previous run, keyed by repo name.""" if not out_path.exists(): return {} try: data = json.loads(out_path.read_text(encoding="utf-8")) - by_repo: dict[str, list[RepoResult]] = {} - for entry in data.get("repos", []): - result = RepoResult(**entry) - by_repo.setdefault(result.repo, []).append(result) - return {repo: results for repo, results in by_repo.items() if {result.mode for result in results} >= set(modes)} + return {entry["repo"]: RepoResult(**entry) for entry in data.get("repos", [])} except (json.JSONDecodeError, KeyError, TypeError): return {} @@ -136,36 +159,31 @@ def _bench( repo_tasks: dict[str, list[Task]], specs: dict[str, RepoSpec], model: _AsymmetricWrapper, - modes: list[str], out_path: Path | None, *, verbose: bool = False, ) -> list[RepoResult]: - """Index each repo once, evaluate each mode, and save after every repo.""" - completed = _load_completed(out_path, modes) if out_path else {} + """Index and evaluate each repo, saving after every repo.""" + completed = _load_completed(out_path) if out_path else {} if completed: print(f"Resuming: {len(completed)} repo(s) already done, skipping.", file=sys.stderr) - results: list[RepoResult] = [r for repo_results in completed.values() for r in repo_results] + results: list[RepoResult] = list(completed.values()) header = ( - f"{'Repo':<12} {'Language':<12} {'Mode':<10} {'Chunks':>6}" - f" {'Index':>9} {'NDCG@5':>8} {'NDCG@10':>8} {'p50':>8} {'p90':>8}" + f"{'Repo':<12} {'Language':<12} {'Chunks':>6} {'Index':>9} {'NDCG@5':>8} {'NDCG@10':>8} {'p50':>8} {'p90':>8}" ) print(header, file=sys.stderr) - print( - f"{'-' * 12} {'-' * 12} {'-' * 10} {'-' * 6} {'-' * 10} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8}", - file=sys.stderr, - ) + print(f"{'-' * 12} {'-' * 12} {'-' * 6} {'-' * 10} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8}", file=sys.stderr) for repo in sorted(completed): - for r in completed[repo]: - print( - f"{r.repo:<12} {r.language:<12} {r.mode:<10} {r.chunks:>6}" - f" {r.index_ms:>8.0f}ms {r.ndcg5:>8.3f} {r.ndcg10:>8.3f}" - f" {r.p50_ms:>7.2f}ms {r.p90_ms:>7.2f}ms (cached)", - file=sys.stderr, - ) + r = completed[repo] + print( + f"{r.repo:<12} {r.language:<12} {r.chunks:>6}" + f" {r.index_ms:>8.0f}ms {r.ndcg5:>8.3f} {r.ndcg10:>8.3f}" + f" {r.p50_ms:>7.2f}ms {r.p90_ms:>7.2f}ms (cached)", + file=sys.stderr, + ) for repo, tasks in sorted(repo_tasks.items()): if repo in completed: @@ -175,35 +193,31 @@ def _bench( print(f"\n--- {repo} ---", file=sys.stderr) started = time.perf_counter() - index = SembleIndex.from_path(spec.benchmark_dir) + index = _build_index(spec.benchmark_dir, model) index_ms = (time.perf_counter() - started) * 1000 - repo_results: list[RepoResult] = [] - for mode in modes: - ndcg5, ndcg10, latencies, by_category = _evaluate(index, tasks, verbose=verbose) - p50, p90 = np.percentile(latencies, [50, 90]).tolist() - result = RepoResult( - repo=repo, - language=spec.language, - mode=mode, - chunks=len(index.chunks), - ndcg5=ndcg5, - ndcg10=ndcg10, - p50_ms=p50, - p90_ms=p90, - index_ms=index_ms, - by_category=by_category, - ) - repo_results.append(result) - print( - f"{repo:<12} {spec.language:<12} {mode:<10} {len(index.chunks):>6}" - f" {index_ms:>8.0f}ms {ndcg5:>8.3f} {ndcg10:>8.3f} {p50:>7.2f}ms {p90:>7.2f}ms", - file=sys.stderr, - ) + ndcg5, ndcg10, latencies, by_category = _evaluate(index, tasks, verbose=verbose) + p50, p90 = np.percentile(latencies, [50, 90]).tolist() + result = RepoResult( + repo=repo, + language=spec.language, + chunks=len(index.chunks), + ndcg5=ndcg5, + ndcg10=ndcg10, + p50_ms=p50, + p90_ms=p90, + index_ms=index_ms, + by_category=by_category, + ) + results.append(result) + print( + f"{repo:<12} {spec.language:<12} {len(index.chunks):>6}" + f" {index_ms:>8.0f}ms {ndcg5:>8.3f} {ndcg10:>8.3f} {p50:>7.2f}ms {p90:>7.2f}ms", + file=sys.stderr, + ) - results.extend(repo_results) if out_path: - save_results("coderankembed", _build_summary(results, modes)) + save_results("coderankembed", _build_summary(results)) return results @@ -211,16 +225,12 @@ def _bench( def _parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description="Benchmark CodeRankEmbed on the semble benchmark suite.") add_filter_args(parser, verbose=True) - parser.add_argument( - "--mode", action="append", default=[], choices=["semantic", "hybrid"], help="Search mode(s) (default: both)." - ) return parser.parse_args() def main() -> None: """Run the CodeRankEmbed baseline benchmark.""" args = _parse_args() - modes = args.mode or ["semantic", "hybrid"] is_full_run = not args.repo and not args.language repo_specs, tasks = load_filtered_tasks(args.repo or None, args.language or None) @@ -233,24 +243,17 @@ def main() -> None: print(file=sys.stderr) out_path = results_path("coderankembed") if is_full_run else None - results = _bench(grouped_tasks(tasks), repo_specs, model, modes, out_path, verbose=args.verbose) + results = _bench(grouped_tasks(tasks), repo_specs, model, out_path, verbose=args.verbose) if not results: return print(file=sys.stderr) - for mode in modes: - mode_results = [r for r in results if r.mode == mode] - if not mode_results: - continue - avg_ndcg10 = sum(r.ndcg10 for r in mode_results) / len(mode_results) - avg_p50 = sum(r.p50_ms for r in mode_results) / len(mode_results) - print( - f" {mode:<10} avg ndcg@10={avg_ndcg10:.3f} avg p50={avg_p50:.1f}ms ({len(mode_results)} repos)", - file=sys.stderr, - ) + avg_ndcg10 = sum(r.ndcg10 for r in results) / len(results) + avg_p50 = sum(r.p50_ms for r in results) / len(results) + print(f" avg ndcg@10={avg_ndcg10:.3f} avg p50={avg_p50:.1f}ms ({len(results)} repos)", file=sys.stderr) - summary = _build_summary(results, modes) + summary = _build_summary(results) print(json.dumps(summary, indent=2)) if is_full_run: diff --git a/benchmarks/baselines/cs.py b/benchmarks/baselines/cs.py new file mode 100644 index 000000000..51712f400 --- /dev/null +++ b/benchmarks/baselines/cs.py @@ -0,0 +1,115 @@ +import json +import sys +import time +from dataclasses import dataclass +from pathlib import Path + +from benchmarks.data import ( + Task, + add_filter_args, + grouped_tasks, + load_filtered_tasks, + save_results, +) +from benchmarks.metrics import file_rank, ndcg_at_k +from benchmarks.tools import run_cs + +_TOP_K = 10 +_LATENCY_RUNS = 3 + + +@dataclass(frozen=True) +class RepoResult: + """Per-repo benchmark result.""" + + repo: str + language: str + ndcg10: float + p50_ms: float + + +def _evaluate_repo( + tasks: list[Task], + benchmark_dir: Path, + *, + verbose: bool = False, +) -> tuple[float, float]: + """Return (mean ndcg@10, p50 latency ms) for a list of tasks.""" + ndcg10_sum = 0.0 + latencies: list[float] = [] + + for task in tasks: + query_latencies: list[float] = [] + file_paths: list[str] = [] + for _ in range(_LATENCY_RUNS): + started = time.perf_counter() + file_paths = run_cs(task.query, benchmark_dir, top_k=_TOP_K) + query_latencies.append((time.perf_counter() - started) * 1000) + latencies.append(sorted(query_latencies)[_LATENCY_RUNS // 2]) + + relevant_ranks = [rank for t in task.all_relevant if (rank := file_rank(file_paths, t.path)) is not None] + q_ndcg10 = ndcg_at_k(relevant_ranks, len(task.all_relevant), _TOP_K) + ndcg10_sum += q_ndcg10 + + if verbose: + print( + f" ndcg@10={q_ndcg10:.3f} ranks={relevant_ranks} n_rel={len(task.all_relevant)} q={task.query!r}", + file=sys.stderr, + ) + print(f" targets: {', '.join(t.path for t in task.all_relevant)}", file=sys.stderr) + print(f" top-5: {[Path(fp).name for fp in file_paths[:5]]}", file=sys.stderr) + + latencies.sort() + return ndcg10_sum / len(tasks), latencies[len(latencies) // 2] + + +def main() -> None: + """Run the cs (Code Spelunker) baseline benchmark.""" + import argparse + + parser = argparse.ArgumentParser(description="Benchmark cs (Code Spelunker) on the semble benchmark suite.") + add_filter_args(parser, verbose=True) + args = parser.parse_args() + + repo_specs, tasks = load_filtered_tasks(args.repo or None, args.language or None) + + print("cs (structural BM25 ranker, no index)", file=sys.stderr) + print(f"{'Repo':<22} {'Language':<12} {'NDCG@10':>8} {'p50':>8}", file=sys.stderr) + print(f"{'-' * 22} {'-' * 12} {'-' * 8} {'-' * 8}", file=sys.stderr) + + results: list[RepoResult] = [] + for repo, repo_task_list in sorted(grouped_tasks(tasks).items()): + spec = repo_specs[repo] + if args.verbose: + print(f"\n--- {repo} ---", file=sys.stderr) + ndcg10, p50_ms = _evaluate_repo(repo_task_list, spec.benchmark_dir, verbose=args.verbose) + results.append(RepoResult(repo=repo, language=spec.language, ndcg10=ndcg10, p50_ms=p50_ms)) + print(f"{repo:<22} {spec.language:<12} {ndcg10:>8.3f} {p50_ms:>7.1f}ms", file=sys.stderr) + + if not results: + return + + avg_ndcg10 = sum(r.ndcg10 for r in results) / len(results) + avg_p50 = sum(r.p50_ms for r in results) / len(results) + print(f"{'-' * 22} {'-' * 12} {'-' * 8} {'-' * 8}", file=sys.stderr) + print(f"{'Average (' + str(len(results)) + ')':<22} {'':<12} {avg_ndcg10:>8.3f} {avg_p50:>7.1f}ms", file=sys.stderr) + + summary = { + "tool": "cs", + "note": "structural BM25 ranker, tree-sitter aware, no persistent index", + "repos": [ + {"repo": r.repo, "language": r.language, "ndcg10": round(r.ndcg10, 4), "p50_ms": round(r.p50_ms, 1)} + for r in results + ], + "avg_ndcg10": round(avg_ndcg10, 4), + "avg_p50_ms": round(avg_p50, 1), + } + print(json.dumps(summary, indent=2)) + + if not args.repo and not args.language: + out = save_results("cs", summary) + print(f"\nResults saved to {out}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/plot.py b/benchmarks/plot.py index 517f627e5..9bb0e9abf 100644 --- a/benchmarks/plot.py +++ b/benchmarks/plot.py @@ -25,7 +25,7 @@ class _Method(TypedDict): "name": "ripgrep", "ndcg10": 0.126, "index_ms": 0.0, # no persistent index; scans on the fly - "query_p50_ms": 12.08, + "query_p50_ms": 14.46, "color": "#606060", "params_m": 0, }, @@ -37,19 +37,43 @@ class _Method(TypedDict): "color": "#9b7bb0", "params_m": 0, }, + { + "name": "cs", + "ndcg10": 0.1997, + "index_ms": 0.0, # no persistent index; scans on the fly + "query_p50_ms": 22.1, + "color": "#5aa9a3", + "params_m": 0, + }, + { + "name": "codebase-memory-mcp", + "ndcg10": 0.6298, + "index_ms": 454.0, + "query_p50_ms": 46.3, + "color": "#a3a34a", + "params_m": 0, + }, + { + "name": "ck", + "ndcg10": 0.642, + "index_ms": 95892.7, + "query_p50_ms": 187.0, + "color": "#4a7ba3", + "params_m": 33, + }, { "name": "BM25", "ndcg10": 0.673, - "index_ms": 262.6, # same semble index infrastructure; BM25 component adds negligible overhead - "query_p50_ms": 0.019, + "index_ms": 46.6, # standalone BM25 build time, not shared with semble's dense index + "query_p50_ms": 0.17, # standalone bm25_index.get_scores() + top-k sort, not hybrid search() "color": "#3a9e7e", "params_m": 0, }, { "name": "ColGREP", "ndcg10": 0.6925, - "index_ms": 5750.6, - "query_p50_ms": 123.83, + "index_ms": 5359.0, + "query_p50_ms": 122.42, "color": "#e8a838", "params_m": 16, }, @@ -63,25 +87,17 @@ class _Method(TypedDict): }, { "name": "CodeRankEmbed", - "ndcg10": 0.7648, - "index_ms": 57269.4, - "query_p50_ms": 16.27, + "ndcg10": 0.8393, + "index_ms": 115859.8, + "query_p50_ms": 15.56, "color": "#d9634f", "params_m": 137, }, - { - "name": "CodeRankEmbed\nHybrid", - "ndcg10": 0.8617, - "index_ms": 57269.4, - "query_p50_ms": 16.27, - "color": "#922b21", - "params_m": 137, - }, { "name": "semble", "ndcg10": 0.8544, - "index_ms": 262.6, - "query_p50_ms": 1.49, + "index_ms": 518.2, + "query_p50_ms": 0.91, "color": "#1a5fa8", "params_m": 16, }, @@ -93,11 +109,11 @@ class _Method(TypedDict): _CBRT_LABEL_DELTA_WARM = 0.2 # Frontier methods per mode. -# Cold: incumbent prior-art curve (ripgrep → BM25 → ColGREP → CRE Hybrid); semble floats above it. -# Warm: BM25 dominates ripgrep (faster and higher NDCG), so incumbent curve is BM25 → CRE Hybrid. +# Cold: incumbent prior-art curve (ripgrep → BM25 → ColGREP → CodeRankEmbed); semble floats above it. +# Warm: BM25 dominates ripgrep (faster and higher NDCG), so incumbent curve is BM25 → CodeRankEmbed. _FRONTIER_NAMES: dict[str, set[str]] = { - "cold": {"ripgrep", "BM25", "ColGREP", "CodeRankEmbed\nHybrid"}, - "warm": {"BM25", "CodeRankEmbed\nHybrid"}, + "cold": {"ripgrep", "BM25", "ColGREP", "CodeRankEmbed"}, + "warm": {"BM25", "CodeRankEmbed"}, } diff --git a/benchmarks/results/ck-2e3d2dd2cf8e.json b/benchmarks/results/ck-2e3d2dd2cf8e.json new file mode 100644 index 000000000..ae078d25d --- /dev/null +++ b/benchmarks/results/ck-2e3d2dd2cf8e.json @@ -0,0 +1,450 @@ +{ + "tool": "ck", + "note": "hybrid regex + BAAI/bge-small-en-v1.5 (33M params) semantic search, RRF fusion", + "repos": [ + { + "repo": "abseil-cpp", + "language": "cpp", + "ndcg10": 0.6441, + "p50_ms": 431.1, + "index_ms": 732963.0 + }, + { + "repo": "aeson", + "language": "haskell", + "ndcg10": 0.8684, + "p50_ms": 150.0, + "index_ms": 99192.0 + }, + { + "repo": "aiohttp", + "language": "python", + "ndcg10": 0.6689, + "p50_ms": 145.4, + "index_ms": 77115.0 + }, + { + "repo": "alamofire", + "language": "swift", + "ndcg10": 0.7738, + "p50_ms": 144.8, + "index_ms": 24072.0 + }, + { + "repo": "axios", + "language": "javascript", + "ndcg10": 0.6031, + "p50_ms": 142.4, + "index_ms": 13177.0 + }, + { + "repo": "axum", + "language": "rust", + "ndcg10": 0.4694, + "p50_ms": 150.0, + "index_ms": 64744.0 + }, + { + "repo": "bash-it", + "language": "bash", + "ndcg10": 0.5934, + "p50_ms": 247.4, + "index_ms": 49291.0 + }, + { + "repo": "bats-core", + "language": "bash", + "ndcg10": 0.9215, + "p50_ms": 147.7, + "index_ms": 2356.0 + }, + { + "repo": "cats", + "language": "scala", + "ndcg10": 0.7211, + "p50_ms": 172.6, + "index_ms": 46817.0 + }, + { + "repo": "chi", + "language": "go", + "ndcg10": 0.2999, + "p50_ms": 159.5, + "index_ms": 19697.0 + }, + { + "repo": "circe", + "language": "scala", + "ndcg10": 0.72, + "p50_ms": 152.8, + "index_ms": 7351.0 + }, + { + "repo": "click", + "language": "python", + "ndcg10": 0.9156, + "p50_ms": 143.7, + "index_ms": 25511.0 + }, + { + "repo": "cobra", + "language": "go", + "ndcg10": 0.5923, + "p50_ms": 151.7, + "index_ms": 29510.0 + }, + { + "repo": "commons-lang", + "language": "java", + "ndcg10": 0.82, + "p50_ms": 194.6, + "index_ms": 114592.0 + }, + { + "repo": "curl", + "language": "c", + "ndcg10": 0.4848, + "p50_ms": 220.9, + "index_ms": 333137.0 + }, + { + "repo": "dapper", + "language": "csharp", + "ndcg10": 0.6627, + "p50_ms": 150.6, + "index_ms": 40006.0 + }, + { + "repo": "ecto", + "language": "elixir", + "ndcg10": 0.7575, + "p50_ms": 153.5, + "index_ms": 120090.0 + }, + { + "repo": "exposed", + "language": "kotlin", + "ndcg10": 0.4831, + "p50_ms": 156.1, + "index_ms": 28931.0 + }, + { + "repo": "express", + "language": "javascript", + "ndcg10": 0.9131, + "p50_ms": 134.7, + "index_ms": 9316.0 + }, + { + "repo": "fastapi", + "language": "python", + "ndcg10": 0.6479, + "p50_ms": 165.9, + "index_ms": 37138.0 + }, + { + "repo": "flask", + "language": "python", + "ndcg10": 0.6708, + "p50_ms": 143.8, + "index_ms": 23902.0 + }, + { + "repo": "fmtlib", + "language": "cpp", + "ndcg10": 0.784, + "p50_ms": 143.9, + "index_ms": 69699.0 + }, + { + "repo": "gin", + "language": "go", + "ndcg10": 0.481, + "p50_ms": 158.3, + "index_ms": 59892.0 + }, + { + "repo": "gson", + "language": "java", + "ndcg10": 0.5398, + "p50_ms": 188.1, + "index_ms": 52071.0 + }, + { + "repo": "guzzle", + "language": "php", + "ndcg10": 0.7665, + "p50_ms": 143.0, + "index_ms": 7501.0 + }, + { + "repo": "http4s", + "language": "scala", + "ndcg10": 0.7113, + "p50_ms": 182.0, + "index_ms": 37805.0 + }, + { + "repo": "httpx", + "language": "python", + "ndcg10": 0.7891, + "p50_ms": 140.6, + "index_ms": 22163.0 + }, + { + "repo": "jackson-databind", + "language": "java", + "ndcg10": 0.4578, + "p50_ms": 248.6, + "index_ms": 169660.0 + }, + { + "repo": "kotlinx-coroutines", + "language": "kotlin", + "ndcg10": 0.7108, + "p50_ms": 156.6, + "index_ms": 35395.0 + }, + { + "repo": "ktor", + "language": "kotlin", + "ndcg10": 0.5657, + "p50_ms": 157.2, + "index_ms": 15309.0 + }, + { + "repo": "laravel-framework", + "language": "php", + "ndcg10": 0.4068, + "p50_ms": 563.3, + "index_ms": 211546.0 + }, + { + "repo": "lazy.nvim", + "language": "lua", + "ndcg10": 0.6213, + "p50_ms": 143.8, + "index_ms": 11865.0 + }, + { + "repo": "libuv", + "language": "c", + "ndcg10": 0.4899, + "p50_ms": 167.1, + "index_ms": 108649.0 + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "ndcg10": 0.4835, + "p50_ms": 165.2, + "index_ms": 101720.0 + }, + { + "repo": "mini.nvim", + "language": "lua", + "ndcg10": 0.9262, + "p50_ms": 169.7, + "index_ms": 79435.0 + }, + { + "repo": "model2vec", + "language": "python", + "ndcg10": 0.5376, + "p50_ms": 143.9, + "index_ms": 7455.0 + }, + { + "repo": "monolog", + "language": "php", + "ndcg10": 0.6723, + "p50_ms": 158.0, + "index_ms": 13875.0 + }, + { + "repo": "newtonsoft-json", + "language": "csharp", + "ndcg10": 0.4989, + "p50_ms": 187.6, + "index_ms": 167571.0 + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "ndcg10": 0.6322, + "p50_ms": 154.4, + "index_ms": 76391.0 + }, + { + "repo": "nvm", + "language": "bash", + "ndcg10": 0.516, + "p50_ms": 202.5, + "index_ms": 40523.0 + }, + { + "repo": "pandoc", + "language": "haskell", + "ndcg10": 0.5764, + "p50_ms": 202.4, + "index_ms": 501281.0 + }, + { + "repo": "phoenix", + "language": "elixir", + "ndcg10": 0.7142, + "p50_ms": 148.3, + "index_ms": 74777.0 + }, + { + "repo": "plug", + "language": "elixir", + "ndcg10": 0.8855, + "p50_ms": 148.2, + "index_ms": 41623.0 + }, + { + "repo": "pydantic", + "language": "python", + "ndcg10": 0.5135, + "p50_ms": 158.9, + "index_ms": 128536.0 + }, + { + "repo": "rack", + "language": "ruby", + "ndcg10": 0.8138, + "p50_ms": 141.8, + "index_ms": 34345.0 + }, + { + "repo": "rails", + "language": "ruby", + "ndcg10": 0.4174, + "p50_ms": 198.7, + "index_ms": 71709.0 + }, + { + "repo": "redis", + "language": "c", + "ndcg10": 0.8421, + "p50_ms": 202.8, + "index_ms": 479720.0 + }, + { + "repo": "redux", + "language": "javascript", + "ndcg10": 0.801, + "p50_ms": 134.7, + "index_ms": 2058.0 + }, + { + "repo": "requests", + "language": "python", + "ndcg10": 0.9131, + "p50_ms": 135.3, + "index_ms": 12488.0 + }, + { + "repo": "serde", + "language": "rust", + "ndcg10": 0.3446, + "p50_ms": 252.6, + "index_ms": 195235.0 + }, + { + "repo": "sinatra", + "language": "ruby", + "ndcg10": 0.9815, + "p50_ms": 142.6, + "index_ms": 33710.0 + }, + { + "repo": "snapkit", + "language": "swift", + "ndcg10": 0.6681, + "p50_ms": 156.8, + "index_ms": 4276.0 + }, + { + "repo": "starlette", + "language": "python", + "ndcg10": 0.8318, + "p50_ms": 504.1, + "index_ms": 27513.0 + }, + { + "repo": "telescope.nvim", + "language": "lua", + "ndcg10": 0.6652, + "p50_ms": 157.0, + "index_ms": 19910.0 + }, + { + "repo": "tokio", + "language": "rust", + "ndcg10": 0.4416, + "p50_ms": 214.9, + "index_ms": 291068.0 + }, + { + "repo": "trpc", + "language": "typescript", + "ndcg10": 0.5825, + "p50_ms": 153.9, + "index_ms": 28329.0 + }, + { + "repo": "vapor", + "language": "swift", + "ndcg10": 0.5747, + "p50_ms": 172.2, + "index_ms": 27414.0 + }, + { + "repo": "vitest", + "language": "typescript", + "ndcg10": 0.4985, + "p50_ms": 188.5, + "index_ms": 85140.0 + }, + { + "repo": "xmonad", + "language": "haskell", + "ndcg10": 0.7539, + "p50_ms": 155.7, + "index_ms": 96648.0 + }, + { + "repo": "zig", + "language": "zig", + "ndcg10": 0.4127, + "p50_ms": 471.9, + "index_ms": null + }, + { + "repo": "zig-clap", + "language": "zig", + "ndcg10": 0.6531, + "p50_ms": 145.9, + "index_ms": 19527.0 + }, + { + "repo": "zls", + "language": "zig", + "ndcg10": 0.4671, + "p50_ms": 158.0, + "index_ms": 361145.0 + }, + { + "repo": "zod", + "language": "typescript", + "ndcg10": 0.2873, + "p50_ms": 203.1, + "index_ms": 221465.0 + } + ], + "avg_ndcg10": 0.6422, + "avg_p50_ms": 187.0, + "avg_index_ms": 95892.7 +} diff --git a/benchmarks/results/codebase-memory-mcp-906319556a46.json b/benchmarks/results/codebase-memory-mcp-906319556a46.json new file mode 100644 index 000000000..149709b21 --- /dev/null +++ b/benchmarks/results/codebase-memory-mcp-906319556a46.json @@ -0,0 +1,450 @@ +{ + "tool": "codebase-memory-mcp", + "note": "search_graph: SQLite FTS5 BM25 with structural boosting (Function/Method +10, Route +8, Class +5); 'fast' index mode (no embeddings)", + "repos": [ + { + "repo": "abseil-cpp", + "language": "cpp", + "ndcg10": 0.5653, + "p50_ms": 54.1, + "index_ms": 2146.0 + }, + { + "repo": "aeson", + "language": "haskell", + "ndcg10": 0.6496, + "p50_ms": 47.4, + "index_ms": 460.0 + }, + { + "repo": "aiohttp", + "language": "python", + "ndcg10": 0.5947, + "p50_ms": 46.7, + "index_ms": 349.0 + }, + { + "repo": "alamofire", + "language": "swift", + "ndcg10": 0.6935, + "p50_ms": 46.7, + "index_ms": 450.0 + }, + { + "repo": "axios", + "language": "javascript", + "ndcg10": 0.7437, + "p50_ms": 46.5, + "index_ms": 254.0 + }, + { + "repo": "axum", + "language": "rust", + "ndcg10": 0.4127, + "p50_ms": 45.6, + "index_ms": 354.0 + }, + { + "repo": "bash-it", + "language": "bash", + "ndcg10": 0.4959, + "p50_ms": 57.4, + "index_ms": 246.0 + }, + { + "repo": "bats-core", + "language": "bash", + "ndcg10": 0.8881, + "p50_ms": 60.1, + "index_ms": 363.0 + }, + { + "repo": "cats", + "language": "scala", + "ndcg10": 0.8438, + "p50_ms": 54.7, + "index_ms": 569.0 + }, + { + "repo": "chi", + "language": "go", + "ndcg10": 0.5163, + "p50_ms": 48.7, + "index_ms": 364.0 + }, + { + "repo": "circe", + "language": "scala", + "ndcg10": 0.6417, + "p50_ms": 60.6, + "index_ms": 351.0 + }, + { + "repo": "click", + "language": "python", + "ndcg10": 0.8701, + "p50_ms": 47.4, + "index_ms": 350.0 + }, + { + "repo": "cobra", + "language": "go", + "ndcg10": 0.7272, + "p50_ms": 44.9, + "index_ms": 463.0 + }, + { + "repo": "commons-lang", + "language": "java", + "ndcg10": 0.6029, + "p50_ms": 48.4, + "index_ms": 464.0 + }, + { + "repo": "curl", + "language": "c", + "ndcg10": 0.5325, + "p50_ms": 52.1, + "index_ms": 556.0 + }, + { + "repo": "dapper", + "language": "csharp", + "ndcg10": 0.7977, + "p50_ms": 45.1, + "index_ms": 356.0 + }, + { + "repo": "ecto", + "language": "elixir", + "ndcg10": 0.4916, + "p50_ms": 50.6, + "index_ms": 557.0 + }, + { + "repo": "exposed", + "language": "kotlin", + "ndcg10": 0.421, + "p50_ms": 52.7, + "index_ms": 358.0 + }, + { + "repo": "express", + "language": "javascript", + "ndcg10": 0.838, + "p50_ms": 50.4, + "index_ms": 256.0 + }, + { + "repo": "fastapi", + "language": "python", + "ndcg10": 0.6572, + "p50_ms": 46.4, + "index_ms": 362.0 + }, + { + "repo": "flask", + "language": "python", + "ndcg10": 0.5959, + "p50_ms": 48.8, + "index_ms": 348.0 + }, + { + "repo": "fmtlib", + "language": "cpp", + "ndcg10": 0.6249, + "p50_ms": 48.3, + "index_ms": 759.0 + }, + { + "repo": "gin", + "language": "go", + "ndcg10": 0.2731, + "p50_ms": 46.5, + "index_ms": 350.0 + }, + { + "repo": "gson", + "language": "java", + "ndcg10": 0.4465, + "p50_ms": 44.9, + "index_ms": 481.0 + }, + { + "repo": "guzzle", + "language": "php", + "ndcg10": 0.502, + "p50_ms": 40.6, + "index_ms": 354.0 + }, + { + "repo": "http4s", + "language": "scala", + "ndcg10": 0.6259, + "p50_ms": 42.8, + "index_ms": 364.0 + }, + { + "repo": "httpx", + "language": "python", + "ndcg10": 0.561, + "p50_ms": 41.5, + "index_ms": 364.0 + }, + { + "repo": "jackson-databind", + "language": "java", + "ndcg10": 0.6138, + "p50_ms": 47.1, + "index_ms": 794.0 + }, + { + "repo": "kotlinx-coroutines", + "language": "kotlin", + "ndcg10": 0.8262, + "p50_ms": 42.9, + "index_ms": 363.0 + }, + { + "repo": "ktor", + "language": "kotlin", + "ndcg10": 0.5867, + "p50_ms": 42.2, + "index_ms": 262.0 + }, + { + "repo": "laravel-framework", + "language": "php", + "ndcg10": 0.7073, + "p50_ms": 50.5, + "index_ms": 912.0 + }, + { + "repo": "lazy.nvim", + "language": "lua", + "ndcg10": 0.371, + "p50_ms": 44.0, + "index_ms": 260.0 + }, + { + "repo": "libuv", + "language": "c", + "ndcg10": 0.6422, + "p50_ms": 42.2, + "index_ms": 356.0 + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "ndcg10": 0.6694, + "p50_ms": 42.9, + "index_ms": 359.0 + }, + { + "repo": "mini.nvim", + "language": "lua", + "ndcg10": 0.709, + "p50_ms": 48.6, + "index_ms": 1123.0 + }, + { + "repo": "model2vec", + "language": "python", + "ndcg10": 0.5507, + "p50_ms": 41.0, + "index_ms": 245.0 + }, + { + "repo": "monolog", + "language": "php", + "ndcg10": 0.6147, + "p50_ms": 41.9, + "index_ms": 254.0 + }, + { + "repo": "newtonsoft-json", + "language": "csharp", + "ndcg10": 0.8574, + "p50_ms": 46.1, + "index_ms": 463.0 + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "ndcg10": 0.6985, + "p50_ms": 42.0, + "index_ms": 683.0 + }, + { + "repo": "nvm", + "language": "bash", + "ndcg10": 0.9196, + "p50_ms": 40.5, + "index_ms": 370.0 + }, + { + "repo": "pandoc", + "language": "haskell", + "ndcg10": 0.4985, + "p50_ms": 52.2, + "index_ms": 472.0 + }, + { + "repo": "phoenix", + "language": "elixir", + "ndcg10": 0.3496, + "p50_ms": 50.0, + "index_ms": 261.0 + }, + { + "repo": "plug", + "language": "elixir", + "ndcg10": 0.6757, + "p50_ms": 45.1, + "index_ms": 360.0 + }, + { + "repo": "pydantic", + "language": "python", + "ndcg10": 0.5136, + "p50_ms": 43.6, + "index_ms": 365.0 + }, + { + "repo": "rack", + "language": "ruby", + "ndcg10": 0.75, + "p50_ms": 42.3, + "index_ms": 360.0 + }, + { + "repo": "rails", + "language": "ruby", + "ndcg10": 0.5052, + "p50_ms": 45.5, + "index_ms": 364.0 + }, + { + "repo": "redis", + "language": "c", + "ndcg10": 0.7893, + "p50_ms": 48.7, + "index_ms": 1023.0 + }, + { + "repo": "redux", + "language": "javascript", + "ndcg10": 0.7275, + "p50_ms": 42.4, + "index_ms": 260.0 + }, + { + "repo": "requests", + "language": "python", + "ndcg10": 0.7679, + "p50_ms": 40.6, + "index_ms": 377.0 + }, + { + "repo": "serde", + "language": "rust", + "ndcg10": 0.4509, + "p50_ms": 41.4, + "index_ms": 361.0 + }, + { + "repo": "sinatra", + "language": "ruby", + "ndcg10": 0.8131, + "p50_ms": 40.6, + "index_ms": 250.0 + }, + { + "repo": "snapkit", + "language": "swift", + "ndcg10": 0.5381, + "p50_ms": 41.8, + "index_ms": 255.0 + }, + { + "repo": "starlette", + "language": "python", + "ndcg10": 0.6794, + "p50_ms": 41.8, + "index_ms": 361.0 + }, + { + "repo": "telescope.nvim", + "language": "lua", + "ndcg10": 0.6944, + "p50_ms": 42.2, + "index_ms": 246.0 + }, + { + "repo": "tokio", + "language": "rust", + "ndcg10": 0.4973, + "p50_ms": 43.2, + "index_ms": 459.0 + }, + { + "repo": "trpc", + "language": "typescript", + "ndcg10": 0.6347, + "p50_ms": 43.0, + "index_ms": 249.0 + }, + { + "repo": "vapor", + "language": "swift", + "ndcg10": 0.6596, + "p50_ms": 43.5, + "index_ms": 253.0 + }, + { + "repo": "vitest", + "language": "typescript", + "ndcg10": 0.4596, + "p50_ms": 42.9, + "index_ms": 366.0 + }, + { + "repo": "xmonad", + "language": "haskell", + "ndcg10": 0.7241, + "p50_ms": 43.0, + "index_ms": 256.0 + }, + { + "repo": "zig", + "language": "zig", + "ndcg10": 0.7571, + "p50_ms": 56.1, + "index_ms": 1002.0 + }, + { + "repo": "zig-clap", + "language": "zig", + "ndcg10": 0.7893, + "p50_ms": 45.5, + "index_ms": 366.0 + }, + { + "repo": "zls", + "language": "zig", + "ndcg10": 0.7518, + "p50_ms": 44.7, + "index_ms": 903.0 + }, + { + "repo": "zod", + "language": "typescript", + "ndcg10": 0.2719, + "p50_ms": 42.2, + "index_ms": 364.0 + } + ], + "avg_ndcg10": 0.6298, + "avg_p50_ms": 46.3, + "avg_index_ms": 454.0 +} diff --git a/benchmarks/results/coderankembed-812d8e74b9d0.json b/benchmarks/results/coderankembed-812d8e74b9d0.json new file mode 100644 index 000000000..c400e9b07 --- /dev/null +++ b/benchmarks/results/coderankembed-812d8e74b9d0.json @@ -0,0 +1,941 @@ +{ + "tool": "coderankembed", + "note": "nomic-ai/CodeRankEmbed (137M params), semantic-only dense search (alpha=1.0)", + "repos": [ + { + "repo": "abseil-cpp", + "language": "cpp", + "chunks": 16824, + "ndcg5": 0.8533745161718927, + "ndcg10": 0.8533745161718927, + "p50_ms": 19.0328545140801, + "p90_ms": 56.07666281575803, + "index_ms": 618604.4516669936, + "by_category": { + "architecture": 1.0, + "semantic": 0.8624373713244265, + "symbol": 0.7103099178571526 + } + }, + { + "repo": "aeson", + "language": "haskell", + "chunks": 788, + "ndcg5": 0.7836848582002587, + "ndcg10": 0.7836848582002587, + "p50_ms": 11.786229006247595, + "p90_ms": 17.135683295782666, + "index_ms": 28080.439207988093, + "by_category": { + "architecture": 0.6297424628754913, + "semantic": 0.8349989899751812 + } + }, + { + "repo": "aiohttp", + "language": "python", + "chunks": 1469, + "ndcg5": 0.8771295613880089, + "ndcg10": 0.8868620565112701, + "p50_ms": 8.768875006353483, + "p90_ms": 12.441166996723041, + "index_ms": 42771.746666985564, + "by_category": { + "architecture": 0.9770630826137678, + "semantic": 0.7538512898266998, + "symbol": 1.0 + } + }, + { + "repo": "alamofire", + "language": "swift", + "chunks": 1300, + "ndcg5": 0.969504606050801, + "ndcg10": 0.969504606050801, + "p50_ms": 7.793916985974647, + "p90_ms": 12.789429104304872, + "index_ms": 36898.45345800859, + "by_category": { + "architecture": 0.9732402630493958, + "semantic": 0.9518519392607121, + "symbol": 1.0 + } + }, + { + "repo": "axios", + "language": "javascript", + "chunks": 299, + "ndcg5": 0.8764068122572592, + "ndcg10": 0.8764068122572592, + "p50_ms": 8.376916506676935, + "p90_ms": 10.111495800083503, + "index_ms": 9295.566458982648, + "by_category": { + "architecture": 0.7301841859614265, + "semantic": 0.8752461461486721, + "symbol": 1.0 + } + }, + { + "repo": "axum", + "language": "rust", + "chunks": 1034, + "ndcg5": 0.7446394630357187, + "ndcg10": 0.7791164890577864, + "p50_ms": 7.577020995086059, + "p90_ms": 12.065470792003907, + "index_ms": 34879.40008298028, + "by_category": { + "architecture": 0.6712414374216045, + "semantic": 0.852176882837309, + "symbol": 0.6666666666666666 + } + }, + { + "repo": "bash-it", + "language": "bash", + "chunks": 1319, + "ndcg5": 0.632019654975484, + "ndcg10": 0.6876361727178842, + "p50_ms": 8.498729483108036, + "p90_ms": 9.165320891770534, + "index_ms": 46668.29699999653, + "by_category": { + "architecture": 0.75, + "semantic": 0.679608458123447, + "symbol": 0.6835626997505745 + } + }, + { + "repo": "bats-core", + "language": "bash", + "chunks": 97, + "ndcg5": 0.8526944590681549, + "ndcg10": 0.8526944590681549, + "p50_ms": 6.85818750935141, + "p90_ms": 8.274025018909015, + "index_ms": 3222.654999990482, + "by_category": { + "architecture": 0.703279918255948, + "semantic": 0.9331484425824198 + } + }, + { + "repo": "cats", + "language": "scala", + "chunks": 2393, + "ndcg5": 0.9113147192765458, + "ndcg10": 0.9205435041155242, + "p50_ms": 11.96185449953191, + "p90_ms": 16.94263780664187, + "index_ms": 85615.23762499564, + "by_category": { + "architecture": 0.8527175205776212, + "semantic": 0.9, + "symbol": 1.0 + } + }, + { + "repo": "chi", + "language": "go", + "chunks": 508, + "ndcg5": 0.705559947196655, + "ndcg10": 0.7240784657151736, + "p50_ms": 7.519791484810412, + "p90_ms": 9.318404400255531, + "index_ms": 17337.142165983096, + "by_category": { + "architecture": 0.5452588126217018, + "semantic": 0.8624288341269906, + "symbol": 0.6666666666666666 + } + }, + { + "repo": "circe", + "language": "scala", + "chunks": 379, + "ndcg5": 0.9210526315789473, + "ndcg10": 0.9210526315789473, + "p50_ms": 7.893166010035202, + "p90_ms": 10.75067458441481, + "index_ms": 12639.447207999183, + "by_category": { + "architecture": 0.8, + "semantic": 0.9545454545454546, + "symbol": 1.0 + } + }, + { + "repo": "click", + "language": "python", + "chunks": 610, + "ndcg5": 0.9630929753571458, + "ndcg10": 0.9630929753571458, + "p50_ms": 10.136500000953674, + "p90_ms": 12.08890361012891, + "index_ms": 18604.901832994074, + "by_category": { + "architecture": 0.9384882922619097, + "semantic": 0.9630929753571458, + "symbol": 1.0 + } + }, + { + "repo": "cobra", + "language": "go", + "chunks": 780, + "ndcg5": 0.7677394468013801, + "ndcg10": 0.7677394468013801, + "p50_ms": 7.656000496353954, + "p90_ms": 12.233508686767891, + "index_ms": 27104.321416991297, + "by_category": { + "architecture": 0.7103099178571526, + "semantic": 0.791461511059104, + "symbol": 0.7956176024115139 + } + }, + { + "repo": "commons-lang", + "language": "java", + "chunks": 6049, + "ndcg5": 0.863660295573871, + "ndcg10": 0.863660295573871, + "p50_ms": 12.666374997934327, + "p90_ms": 36.14383400417864, + "index_ms": 180353.87958300998, + "by_category": { + "architecture": 0.5377157309218195, + "semantic": 0.8945513581632737, + "symbol": 1.0 + } + }, + { + "repo": "curl", + "language": "c", + "chunks": 8904, + "ndcg5": 0.6315622065940932, + "ndcg10": 0.656234567789412, + "p50_ms": 9.51114550116472, + "p90_ms": 10.842887795297429, + "index_ms": 297927.92766701314, + "by_category": { + "architecture": 0.6583394119944418, + "semantic": 0.6545124225307509 + } + }, + { + "repo": "dapper", + "language": "csharp", + "chunks": 798, + "ndcg5": 0.7630929753571458, + "ndcg10": 0.795954834495746, + "p50_ms": 10.312229496776126, + "p90_ms": 11.937941718497314, + "index_ms": 24267.82504201401, + "by_category": { + "architecture": 0.33905179677700553, + "semantic": 0.8894530386774536, + "symbol": 1.0 + } + }, + { + "repo": "ecto", + "language": "elixir", + "chunks": 1431, + "ndcg5": 0.9288795954935503, + "ndcg10": 0.9288795954935503, + "p50_ms": 8.444375009275973, + "p90_ms": 18.15032540471293, + "index_ms": 46997.82987500657, + "by_category": { + "architecture": 1.0, + "semantic": 0.8960547934136506, + "symbol": 1.0 + } + }, + { + "repo": "exposed", + "language": "kotlin", + "chunks": 1515, + "ndcg5": 0.6806128427182248, + "ndcg10": 0.6908319625976491, + "p50_ms": 10.752416506875306, + "p90_ms": 15.28513717930764, + "index_ms": 46875.47220897977, + "by_category": { + "architecture": 0.8333333333333334, + "semantic": 0.5940456608537844, + "symbol": 1.0 + } + }, + { + "repo": "express", + "language": "javascript", + "chunks": 102, + "ndcg5": 0.975, + "ndcg10": 0.975, + "p50_ms": 8.340270491316915, + "p90_ms": 9.812800306826835, + "index_ms": 3373.775541986106, + "by_category": { + "architecture": 1.0, + "semantic": 0.95, + "symbol": 1.0 + } + }, + { + "repo": "fastapi", + "language": "python", + "chunks": 1188, + "ndcg5": 0.790381976157683, + "ndcg10": 0.8013023479985912, + "p50_ms": 8.624729016446508, + "p90_ms": 11.68960379145574, + "index_ms": 30003.29175000661, + "by_category": { + "architecture": 0.7124176122501534, + "semantic": 0.7548904592775939, + "symbol": 1.0 + } + }, + { + "repo": "flask", + "language": "python", + "chunks": 558, + "ndcg5": 0.8710644478386811, + "ndcg10": 0.8902541208671446, + "p50_ms": 8.24412499787286, + "p90_ms": 9.782000008272007, + "index_ms": 16811.63137499243, + "by_category": { + "architecture": 0.9381431141670256, + "semantic": 0.8634004532244912, + "symbol": 0.8769765845238192 + } + }, + { + "repo": "fmtlib", + "language": "cpp", + "chunks": 966, + "ndcg5": 0.9380929753571458, + "ndcg10": 0.9380929753571458, + "p50_ms": 8.561374997952953, + "p90_ms": 10.17405829334166, + "index_ms": 37705.68645800813, + "by_category": { + "architecture": 0.8769765845238192, + "semantic": 0.9379235538265327, + "symbol": 1.0 + } + }, + { + "repo": "gin", + "language": "go", + "chunks": 1171, + "ndcg5": 0.601246844003376, + "ndcg10": 0.6461730625292705, + "p50_ms": 10.590874502668157, + "p90_ms": 12.532420913339594, + "index_ms": 43848.21983301663, + "by_category": { + "architecture": 0.597365243962041, + "semantic": 0.5762972533466514, + "symbol": 1.0 + } + }, + { + "repo": "gson", + "language": "java", + "chunks": 2882, + "ndcg5": 0.7192955013573377, + "ndcg10": 0.7447400653774877, + "p50_ms": 9.612416994059458, + "p90_ms": 18.484416697174314, + "index_ms": 89205.03929100232, + "by_category": { + "architecture": 0.21703186658518844, + "semantic": 0.8026673841209, + "symbol": 1.0 + } + }, + { + "repo": "guzzle", + "language": "php", + "chunks": 386, + "ndcg5": 0.938503284734147, + "ndcg10": 0.9477320695731255, + "p50_ms": 7.437250009388663, + "p90_ms": 9.870354490703903, + "index_ms": 10894.163833989296, + "by_category": { + "architecture": 0.9732402630493958, + "semantic": 0.9257631232549479, + "symbol": 1.0 + } + }, + { + "repo": "http4s", + "language": "scala", + "chunks": 1849, + "ndcg5": 0.9319537411240482, + "ndcg10": 0.9319537411240482, + "p50_ms": 14.001688003190793, + "p90_ms": 17.276770892203785, + "index_ms": 61675.23533300846, + "by_category": { + "architecture": 0.9590717717793499, + "semantic": 0.9115613933673511, + "symbol": 1.0 + } + }, + { + "repo": "httpx", + "language": "python", + "chunks": 488, + "ndcg5": 0.9297379202551925, + "ndcg10": 0.9389487013798243, + "p50_ms": 8.105541986878961, + "p90_ms": 10.95104101113975, + "index_ms": 14709.17079100036, + "by_category": { + "architecture": 0.8557660725862154, + "semantic": 0.9697289134303114, + "symbol": 1.0 + } + }, + { + "repo": "jackson-databind", + "language": "java", + "chunks": 8975, + "ndcg5": 0.7203465950867115, + "ndcg10": 0.7615299791302552, + "p50_ms": 13.546957998187281, + "p90_ms": 36.670599700300954, + "index_ms": 278275.5487910181, + "by_category": { + "architecture": 0.5836179065485082, + "semantic": 0.7013900055402845, + "symbol": 1.0 + } + }, + { + "repo": "kotlinx-coroutines", + "language": "kotlin", + "chunks": 1597, + "ndcg5": 0.8511859507142916, + "ndcg10": 0.8678526173809583, + "p50_ms": 13.515687503968365, + "p90_ms": 16.637812781846154, + "index_ms": 50065.45329198707, + "by_category": { + "architecture": 0.8333333333333334, + "semantic": 0.8469323105442259, + "symbol": 1.0 + } + }, + { + "repo": "ktor", + "language": "kotlin", + "chunks": 760, + "ndcg5": 0.9007429416731274, + "ndcg10": 0.9109620615525518, + "p50_ms": 8.506999991368502, + "p90_ms": 12.562266891472975, + "index_ms": 23459.111250005662, + "by_category": { + "architecture": 0.8397905453446818, + "semantic": 0.9058153448054661, + "symbol": 1.0 + } + }, + { + "repo": "laravel-framework", + "language": "php", + "chunks": 11681, + "ndcg5": 0.6872200000795508, + "ndcg10": 0.7147016796158153, + "p50_ms": 10.221312521025538, + "p90_ms": 37.04688748402987, + "index_ms": 334185.789375013, + "by_category": { + "architecture": 0.454366192042393, + "semantic": 0.831393388272011, + "symbol": 0.9077324383928644 + } + }, + { + "repo": "lazy.nvim", + "language": "lua", + "chunks": 596, + "ndcg5": 0.7328428691643483, + "ndcg10": 0.767319895186416, + "p50_ms": 7.9822495026746765, + "p90_ms": 8.959858008893207, + "index_ms": 21001.784458989277, + "by_category": { + "architecture": 0.8056894514724523, + "semantic": 0.7359266218614772 + } + }, + { + "repo": "libuv", + "language": "c", + "chunks": 2638, + "ndcg5": 0.7207656053816912, + "ndcg10": 0.7316859772225992, + "p50_ms": 7.734645521850325, + "p90_ms": 8.556067288736813, + "index_ms": 87342.43150000111, + "by_category": { + "architecture": 1.0, + "semantic": 0.7018733080251103, + "symbol": 1.0 + } + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "chunks": 2179, + "ndcg5": 0.8745560513042175, + "ndcg10": 0.8745560513042175, + "p50_ms": 14.201603989931755, + "p90_ms": 15.181141986977309, + "index_ms": 58772.2294999985, + "by_category": { + "architecture": 0.5018158518630595, + "semantic": 0.9426508465146795, + "symbol": 1.0 + } + }, + { + "repo": "mini.nvim", + "language": "lua", + "chunks": 4381, + "ndcg5": 0.9815464876785729, + "ndcg10": 0.9815464876785729, + "p50_ms": 8.750395980314352, + "p90_ms": 9.586870195926167, + "index_ms": 151508.6681249959, + "by_category": { + "architecture": 1.0, + "semantic": 0.9769331095982161 + } + }, + { + "repo": "model2vec", + "language": "python", + "chunks": 203, + "ndcg5": 0.6907359332646117, + "ndcg10": 0.7327258336263148, + "p50_ms": 7.794124991050921, + "p90_ms": 9.435887791914865, + "index_ms": 5737.819083005888, + "by_category": { + "architecture": 0.7866418334952426, + "semantic": 0.6761171030825056, + "symbol": 0.7956176024115139 + } + }, + { + "repo": "monolog", + "language": "php", + "chunks": 774, + "ndcg5": 0.8363147192765459, + "ndcg10": 0.8773446510571661, + "p50_ms": 8.253187013906427, + "p90_ms": 10.765824420377612, + "index_ms": 22229.444458003854, + "by_category": { + "architecture": 0.6430676783119402, + "semantic": 0.9331554629583622, + "symbol": 1.0 + } + }, + { + "repo": "newtonsoft-json", + "language": "csharp", + "chunks": 4297, + "ndcg5": 0.8740663550396519, + "ndcg10": 0.8740663550396519, + "p50_ms": 21.796229004394263, + "p90_ms": 24.53397939971183, + "index_ms": 116233.86224999558, + "by_category": { + "architecture": 0.8917328912035968, + "semantic": 0.8300845230519507, + "symbol": 1.0 + } + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "chunks": 1599, + "ndcg5": 0.88921041678514, + "ndcg10": 0.8988817369660035, + "p50_ms": 8.592729005613364, + "p90_ms": 12.630186919705011, + "index_ms": 51208.99133299827, + "by_category": { + "architecture": 0.8352435693926591, + "semantic": 0.9130929753571457, + "symbol": 1.0 + } + }, + { + "repo": "nvm", + "language": "bash", + "chunks": 309, + "ndcg5": 0.9630929753571458, + "ndcg10": 0.9630929753571458, + "p50_ms": 6.4721664966782555, + "p90_ms": 7.284792006248608, + "index_ms": 10994.143082993105, + "by_category": { + "architecture": 0.8154648767857288, + "semantic": 0.9794960974206366 + } + }, + { + "repo": "pandoc", + "language": "haskell", + "chunks": 6178, + "ndcg5": 0.8227197786179612, + "ndcg10": 0.8227197786179612, + "p50_ms": 11.337478994391859, + "p90_ms": 28.626642312156036, + "index_ms": 188675.6729169865, + "by_category": { + "architecture": 0.7577007889556063, + "semantic": 0.8660657717261978 + } + }, + { + "repo": "phoenix", + "language": "elixir", + "chunks": 1046, + "ndcg5": 0.8388357287839533, + "ndcg10": 0.8388357287839533, + "p50_ms": 7.603417005157098, + "p90_ms": 11.49785018642433, + "index_ms": 31730.159083002945, + "by_category": { + "architecture": 0.7825423300834514, + "semantic": 0.8421425537348641, + "symbol": 1.0 + } + }, + { + "repo": "plug", + "language": "elixir", + "chunks": 509, + "ndcg5": 0.8946394630357186, + "ndcg10": 0.9104127068750051, + "p50_ms": 7.2731459949864075, + "p90_ms": 9.803424405981787, + "index_ms": 16300.368583993986, + "by_category": { + "architecture": 1.0, + "semantic": 0.8983803131377603, + "symbol": 0.8769765845238192 + } + }, + { + "repo": "pydantic", + "language": "python", + "chunks": 2868, + "ndcg5": 0.7539393393932413, + "ndcg10": 0.7539393393932413, + "p50_ms": 10.838687507202849, + "p90_ms": 16.788521409034733, + "index_ms": 83974.0884580242, + "by_category": { + "architecture": 0.5754480350207404, + "semantic": 0.8024650986435234, + "symbol": 0.9261859507142916 + } + }, + { + "repo": "rack", + "language": "ruby", + "chunks": 483, + "ndcg5": 0.9384882922619097, + "ndcg10": 0.9384882922619097, + "p50_ms": 7.511438001529314, + "p90_ms": 12.216107995482162, + "index_ms": 14195.551042008447, + "by_category": { + "architecture": 1.0, + "semantic": 0.8892789260714373, + "symbol": 1.0 + } + }, + { + "repo": "rails", + "language": "ruby", + "chunks": 870, + "ndcg5": 0.8081898867742552, + "ndcg10": 0.8293293784945878, + "p50_ms": 7.690916492720135, + "p90_ms": 12.306383418035693, + "index_ms": 23925.738499994623, + "by_category": { + "architecture": 0.7387871355486184, + "semantic": 0.9472756790816368, + "symbol": 0.8266433990956823 + } + }, + { + "repo": "redis", + "language": "c", + "chunks": 12306, + "ndcg5": 0.9258891280402999, + "ndcg10": 0.9258891280402999, + "p50_ms": 9.949478509952314, + "p90_ms": 11.505816713906825, + "index_ms": 392957.3756250029, + "by_category": { + "architecture": 0.9017782560805999, + "semantic": 0.95 + } + }, + { + "repo": "redux", + "language": "javascript", + "chunks": 87, + "ndcg5": 0.9060646130837483, + "ndcg10": 0.9227312797504149, + "p50_ms": 8.766562488744967, + "p90_ms": 9.908053881372325, + "index_ms": 2635.6589579954743, + "by_category": { + "architecture": 0.9150604180172514, + "semantic": 0.884918120767199, + "symbol": 1.0 + } + }, + { + "repo": "requests", + "language": "python", + "chunks": 316, + "ndcg5": 0.9740637329553106, + "ndcg10": 0.9740637329553106, + "p50_ms": 8.661604500957765, + "p90_ms": 11.957307983539069, + "index_ms": 8785.81466601463, + "by_category": { + "architecture": 1.0, + "semantic": 0.9351593323882765, + "symbol": 1.0 + } + }, + { + "repo": "serde", + "language": "rust", + "chunks": 2187, + "ndcg5": 0.5530803155822426, + "ndcg10": 0.6189275194482255, + "p50_ms": 8.605061986600049, + "p90_ms": 14.366328896721825, + "index_ms": 65978.21654201834, + "by_category": { + "architecture": 0.46716720638937514, + "semantic": 0.6178130541784601, + "symbol": 1.0 + } + }, + { + "repo": "sinatra", + "language": "ruby", + "chunks": 134, + "ndcg5": 0.9315464876785728, + "ndcg10": 0.9473197315178593, + "p50_ms": 7.369083003140986, + "p90_ms": 10.363466700073333, + "index_ms": 3805.43916602619, + "by_category": { + "architecture": 0.9261859507142916, + "semantic": 0.923940541865081, + "symbol": 1.0 + } + }, + { + "repo": "snapkit", + "language": "swift", + "chunks": 200, + "ndcg5": 0.7198838441880201, + "ndcg10": 0.779806691996841, + "p50_ms": 8.683749998454005, + "p90_ms": 9.78085818933323, + "index_ms": 5189.072165987454, + "by_category": { + "architecture": 0.6786456757240362, + "semantic": 0.7287928526838322, + "symbol": 1.0 + } + }, + { + "repo": "starlette", + "language": "python", + "chunks": 419, + "ndcg5": 0.9593340559371003, + "ndcg10": 0.9593340559371003, + "p50_ms": 7.483291992684826, + "p90_ms": 9.259957994800061, + "index_ms": 11871.105250000255, + "by_category": { + "architecture": 0.9223650158799187, + "semantic": 1.0, + "symbol": 1.0 + } + }, + { + "repo": "telescope.nvim", + "language": "lua", + "chunks": 1053, + "ndcg5": 0.7380484584370977, + "ndcg10": 0.7380484584370977, + "p50_ms": 7.18266649346333, + "p90_ms": 7.670300296740606, + "index_ms": 31199.50758299092, + "by_category": { + "architecture": 0.7929500775313779, + "semantic": 0.708486048155562 + } + }, + { + "repo": "tokio", + "language": "rust", + "chunks": 5338, + "ndcg5": 0.8271885654576014, + "ndcg10": 0.8633031368426707, + "p50_ms": 10.791770997457206, + "p90_ms": 23.625258306856267, + "index_ms": 174905.88658399065, + "by_category": { + "architecture": 0.8981596823946757, + "semantic": 0.7346380803106698, + "symbol": 1.0 + } + }, + { + "repo": "trpc", + "language": "typescript", + "chunks": 690, + "ndcg5": 0.7795831796771399, + "ndcg10": 0.8014239233589564, + "p50_ms": 10.035833009169437, + "p90_ms": 10.964604187756777, + "index_ms": 20863.401707989397, + "by_category": { + "architecture": 0.6618409361320702, + "semantic": 0.8191072825889378, + "symbol": 1.0 + } + }, + { + "repo": "vapor", + "language": "swift", + "chunks": 1485, + "ndcg5": 0.7336905362161261, + "ndcg10": 0.787093628607754, + "p50_ms": 8.143979008309543, + "p90_ms": 11.779046410811134, + "index_ms": 41482.889750011964, + "by_category": { + "architecture": 0.7021981388906108, + "semantic": 0.7596627253916607, + "symbol": 1.0 + } + }, + { + "repo": "vitest", + "language": "typescript", + "chunks": 2065, + "ndcg5": 0.7105159313011153, + "ndcg10": 0.7105159313011153, + "p50_ms": 8.503417004249059, + "p90_ms": 14.15438809490297, + "index_ms": 59996.19391700253, + "by_category": { + "architecture": 0.75, + "semantic": 0.6100289660020279, + "symbol": 1.0 + } + }, + { + "repo": "xmonad", + "language": "haskell", + "chunks": 241, + "ndcg5": 0.8077957301599588, + "ndcg10": 0.8256060895153599, + "p50_ms": 6.575833496754058, + "p90_ms": 8.91372139158193, + "index_ms": 7642.741333023878, + "by_category": { + "architecture": 0.8571841278062386, + "semantic": 0.8045540639881074 + } + }, + { + "repo": "zig", + "language": "zig", + "chunks": 26252, + "ndcg5": 0.9008891280402999, + "ndcg10": 0.9008891280402999, + "p50_ms": 17.58906249597203, + "p90_ms": 78.69548752496485, + "index_ms": 843119.6548340085, + "by_category": { + "architecture": 0.8769765845238192, + "semantic": 0.9051089886608554 + } + }, + { + "repo": "zig-clap", + "language": "zig", + "chunks": 193, + "ndcg5": 0.8946394630357186, + "ndcg10": 0.9113061297023852, + "p50_ms": 8.84543699794449, + "p90_ms": 10.260895613464527, + "index_ms": 5857.5270420114975, + "by_category": { + "architecture": 0.8154648767857288, + "semantic": 0.921955157804236 + } + }, + { + "repo": "zls", + "language": "zig", + "chunks": 2624, + "ndcg5": 0.919342640361727, + "ndcg10": 0.919342640361727, + "p50_ms": 10.488228988833725, + "p90_ms": 12.03385859844275, + "index_ms": 78771.71687499504, + "by_category": { + "architecture": 0.8571428571428571, + "semantic": 0.952834831325734 + } + }, + { + "repo": "zod", + "language": "typescript", + "chunks": 3576, + "ndcg5": 0.4214866888992802, + "ndcg10": 0.5021921418291659, + "p50_ms": 17.33481250994373, + "p90_ms": 18.42801250750199, + "index_ms": 123796.67454201262, + "by_category": { + "architecture": 0.3730576070078544, + "semantic": 0.49389866576772656, + "symbol": 0.7103099178571526 + } + } + ], + "avg_ndcg10": 0.8393, + "avg_p50_ms": 9.7, + "avg_index_ms": 84683.3 +} diff --git a/benchmarks/results/cs-906319556a46.json b/benchmarks/results/cs-906319556a46.json new file mode 100644 index 000000000..6a8f73c81 --- /dev/null +++ b/benchmarks/results/cs-906319556a46.json @@ -0,0 +1,386 @@ +{ + "tool": "cs", + "note": "structural BM25 ranker, tree-sitter aware, no persistent index", + "repos": [ + { + "repo": "abseil-cpp", + "language": "cpp", + "ndcg10": 0.1597, + "p50_ms": 38.2 + }, + { + "repo": "aeson", + "language": "haskell", + "ndcg10": 0.375, + "p50_ms": 20.8 + }, + { + "repo": "aiohttp", + "language": "python", + "ndcg10": 0.1535, + "p50_ms": 19.9 + }, + { + "repo": "alamofire", + "language": "swift", + "ndcg10": 0.2307, + "p50_ms": 20.2 + }, + { + "repo": "axios", + "language": "javascript", + "ndcg10": 0.1523, + "p50_ms": 19.4 + }, + { + "repo": "axum", + "language": "rust", + "ndcg10": 0.1693, + "p50_ms": 20.2 + }, + { + "repo": "bash-it", + "language": "bash", + "ndcg10": 0.1113, + "p50_ms": 25.2 + }, + { + "repo": "bats-core", + "language": "bash", + "ndcg10": 0.1, + "p50_ms": 18.2 + }, + { + "repo": "cats", + "language": "scala", + "ndcg10": 0.0646, + "p50_ms": 21.2 + }, + { + "repo": "chi", + "language": "go", + "ndcg10": 0.2645, + "p50_ms": 19.8 + }, + { + "repo": "circe", + "language": "scala", + "ndcg10": 0.0789, + "p50_ms": 18.7 + }, + { + "repo": "click", + "language": "python", + "ndcg10": 0.4417, + "p50_ms": 19.9 + }, + { + "repo": "cobra", + "language": "go", + "ndcg10": 0.1393, + "p50_ms": 20.7 + }, + { + "repo": "commons-lang", + "language": "java", + "ndcg10": 0.062, + "p50_ms": 22.1 + }, + { + "repo": "curl", + "language": "c", + "ndcg10": 0.1631, + "p50_ms": 25.1 + }, + { + "repo": "dapper", + "language": "csharp", + "ndcg10": 0.4281, + "p50_ms": 21.0 + }, + { + "repo": "ecto", + "language": "elixir", + "ndcg10": 0.5223, + "p50_ms": 20.4 + }, + { + "repo": "exposed", + "language": "kotlin", + "ndcg10": 0.0565, + "p50_ms": 20.4 + }, + { + "repo": "express", + "language": "javascript", + "ndcg10": 0.1122, + "p50_ms": 17.9 + }, + { + "repo": "fastapi", + "language": "python", + "ndcg10": 0.2776, + "p50_ms": 20.0 + }, + { + "repo": "flask", + "language": "python", + "ndcg10": 0.3914, + "p50_ms": 25.0 + }, + { + "repo": "fmtlib", + "language": "cpp", + "ndcg10": 0.3446, + "p50_ms": 22.7 + }, + { + "repo": "gin", + "language": "go", + "ndcg10": 0.1446, + "p50_ms": 23.3 + }, + { + "repo": "gson", + "language": "java", + "ndcg10": 0.2123, + "p50_ms": 26.0 + }, + { + "repo": "guzzle", + "language": "php", + "ndcg10": 0.1346, + "p50_ms": 22.6 + }, + { + "repo": "http4s", + "language": "scala", + "ndcg10": 0.1904, + "p50_ms": 22.4 + }, + { + "repo": "httpx", + "language": "python", + "ndcg10": 0.3866, + "p50_ms": 20.3 + }, + { + "repo": "jackson-databind", + "language": "java", + "ndcg10": 0.1351, + "p50_ms": 28.7 + }, + { + "repo": "kotlinx-coroutines", + "language": "kotlin", + "ndcg10": 0.2737, + "p50_ms": 20.9 + }, + { + "repo": "ktor", + "language": "kotlin", + "ndcg10": 0.1789, + "p50_ms": 19.8 + }, + { + "repo": "laravel-framework", + "language": "php", + "ndcg10": 0.1015, + "p50_ms": 45.6 + }, + { + "repo": "lazy.nvim", + "language": "lua", + "ndcg10": 0.0, + "p50_ms": 19.7 + }, + { + "repo": "libuv", + "language": "c", + "ndcg10": 0.1407, + "p50_ms": 20.0 + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "ndcg10": 0.1525, + "p50_ms": 20.8 + }, + { + "repo": "mini.nvim", + "language": "lua", + "ndcg10": 0.15, + "p50_ms": 20.1 + }, + { + "repo": "model2vec", + "language": "python", + "ndcg10": 0.1832, + "p50_ms": 18.4 + }, + { + "repo": "monolog", + "language": "php", + "ndcg10": 0.3031, + "p50_ms": 19.6 + }, + { + "repo": "newtonsoft-json", + "language": "csharp", + "ndcg10": 0.1624, + "p50_ms": 22.1 + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "ndcg10": 0.2816, + "p50_ms": 20.9 + }, + { + "repo": "nvm", + "language": "bash", + "ndcg10": 0.3, + "p50_ms": 23.2 + }, + { + "repo": "pandoc", + "language": "haskell", + "ndcg10": 0.05, + "p50_ms": 22.9 + }, + { + "repo": "phoenix", + "language": "elixir", + "ndcg10": 0.351, + "p50_ms": 20.9 + }, + { + "repo": "plug", + "language": "elixir", + "ndcg10": 0.319, + "p50_ms": 19.5 + }, + { + "repo": "pydantic", + "language": "python", + "ndcg10": 0.3685, + "p50_ms": 21.4 + }, + { + "repo": "rack", + "language": "ruby", + "ndcg10": 0.2566, + "p50_ms": 19.2 + }, + { + "repo": "rails", + "language": "ruby", + "ndcg10": 0.2788, + "p50_ms": 25.7 + }, + { + "repo": "redis", + "language": "c", + "ndcg10": 0.2211, + "p50_ms": 32.8 + }, + { + "repo": "redux", + "language": "javascript", + "ndcg10": 0.25, + "p50_ms": 19.8 + }, + { + "repo": "requests", + "language": "python", + "ndcg10": 0.3938, + "p50_ms": 22.0 + }, + { + "repo": "serde", + "language": "rust", + "ndcg10": 0.0801, + "p50_ms": 23.3 + }, + { + "repo": "sinatra", + "language": "ruby", + "ndcg10": 0.2307, + "p50_ms": 20.0 + }, + { + "repo": "snapkit", + "language": "swift", + "ndcg10": 0.1355, + "p50_ms": 20.1 + }, + { + "repo": "starlette", + "language": "python", + "ndcg10": 0.1457, + "p50_ms": 19.8 + }, + { + "repo": "telescope.nvim", + "language": "lua", + "ndcg10": 0.0, + "p50_ms": 20.5 + }, + { + "repo": "tokio", + "language": "rust", + "ndcg10": 0.3291, + "p50_ms": 24.1 + }, + { + "repo": "trpc", + "language": "typescript", + "ndcg10": 0.1465, + "p50_ms": 19.4 + }, + { + "repo": "vapor", + "language": "swift", + "ndcg10": 0.0873, + "p50_ms": 22.6 + }, + { + "repo": "vitest", + "language": "typescript", + "ndcg10": 0.0646, + "p50_ms": 22.3 + }, + { + "repo": "xmonad", + "language": "haskell", + "ndcg10": 0.0557, + "p50_ms": 18.6 + }, + { + "repo": "zig", + "language": "zig", + "ndcg10": 0.2631, + "p50_ms": 28.1 + }, + { + "repo": "zig-clap", + "language": "zig", + "ndcg10": 0.05, + "p50_ms": 18.4 + }, + { + "repo": "zls", + "language": "zig", + "ndcg10": 0.05, + "p50_ms": 19.9 + }, + { + "repo": "zod", + "language": "typescript", + "ndcg10": 0.225, + "p50_ms": 21.6 + } + ], + "avg_ndcg10": 0.1997, + "avg_p50_ms": 22.1 +} diff --git a/benchmarks/results/speed-0e2d85215c7e.json b/benchmarks/results/speed-0e2d85215c7e.json new file mode 100644 index 000000000..17bf62c37 --- /dev/null +++ b/benchmarks/results/speed-0e2d85215c7e.json @@ -0,0 +1,6259 @@ +{ + "repos": [ + "nvm", + "libuv", + "nlohmann-json", + "messagepack-csharp", + "phoenix", + "gin", + "aeson", + "gson", + "axios", + "ktor", + "telescope.nvim", + "monolog", + "flask", + "rack", + "axum", + "http4s", + "alamofire", + "trpc", + "zls" + ], + "summary": { + "semble": { + "avg_index_ms": 590.5, + "avg_p50_ms": 0.98, + "avg_p90_ms": 5.38, + "avg_p95_ms": 6.84, + "avg_p99_ms": 9.69 + }, + "bm25": { + "avg_index_ms": 45.0, + "avg_p50_ms": 0.77, + "avg_p90_ms": 4.36, + "avg_p95_ms": 5.93, + "avg_p99_ms": 9.14 + }, + "colgrep": { + "avg_index_ms": 5359.2, + "avg_p50_ms": 122.42, + "avg_p90_ms": 151.84, + "avg_p95_ms": 159.08, + "avg_p99_ms": 192.73 + }, + "ripgrep": { + "avg_index_ms": null, + "avg_p50_ms": 14.46, + "avg_p90_ms": 17.22, + "avg_p95_ms": 17.88, + "avg_p99_ms": 20.14 + } + }, + "results": [ + { + "repo": "nvm", + "language": "bash", + "tool": "semble", + "index_ms": 232.1, + "p50_ms": 0.64, + "p90_ms": 0.76, + "p95_ms": 0.89, + "p99_ms": 1.35, + "latencies_ms": [ + 5.991, + 0.986, + 0.889, + 0.801, + 0.707, + 0.676, + 0.617, + 0.604, + 0.592, + 0.74, + 0.769, + 1.208, + 0.886, + 0.73, + 0.726, + 0.731, + 0.682, + 0.67, + 0.696, + 0.664, + 0.68, + 0.633, + 0.658, + 0.618, + 0.729, + 0.655, + 0.923, + 0.735, + 0.75, + 0.73, + 0.866, + 0.687, + 0.668, + 0.724, + 0.708, + 0.707, + 0.678, + 0.7, + 0.665, + 0.677, + 0.756, + 0.692, + 0.696, + 0.694, + 1.305, + 0.686, + 0.556, + 0.538, + 0.54, + 0.541, + 0.753, + 0.712, + 0.649, + 0.64, + 0.647, + 0.6, + 0.575, + 0.583, + 0.575, + 0.579, + 0.612, + 0.595, + 0.624, + 0.62, + 0.614, + 0.631, + 0.636, + 0.754, + 0.674, + 0.639, + 0.636, + 0.624, + 0.623, + 0.578, + 0.572, + 0.64, + 0.612, + 0.593, + 0.603, + 0.592, + 0.52, + 0.499, + 0.493, + 0.5, + 0.477, + 0.598, + 0.585, + 0.58, + 0.583, + 0.587, + 0.567, + 0.545, + 0.55, + 0.586, + 0.547, + 0.64, + 0.573, + 0.551, + 0.538, + 0.707 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "bm25", + "index_ms": 9.8, + "p50_ms": 0.51, + "p90_ms": 0.55, + "p95_ms": 0.57, + "p99_ms": 0.6, + "latencies_ms": [ + 0.742, + 0.572, + 0.603, + 0.516, + 0.49, + 0.516, + 0.491, + 0.484, + 0.475, + 0.475, + 0.567, + 0.544, + 0.535, + 0.533, + 0.523, + 0.545, + 0.537, + 0.536, + 0.52, + 0.521, + 0.482, + 0.478, + 0.568, + 0.57, + 0.537, + 0.535, + 0.488, + 0.472, + 0.455, + 0.458, + 0.522, + 0.473, + 0.47, + 0.478, + 0.486, + 0.52, + 0.501, + 0.501, + 0.504, + 0.502, + 0.534, + 0.517, + 0.507, + 0.506, + 0.507, + 0.484, + 0.454, + 0.452, + 0.471, + 0.45, + 0.555, + 0.547, + 0.533, + 0.537, + 0.538, + 0.499, + 0.603, + 0.585, + 0.543, + 0.523, + 0.555, + 0.537, + 0.533, + 0.533, + 0.535, + 0.516, + 0.476, + 0.492, + 0.469, + 0.469, + 0.501, + 0.505, + 0.479, + 0.483, + 0.476, + 0.525, + 0.507, + 0.495, + 0.494, + 0.514, + 0.424, + 0.401, + 0.399, + 0.397, + 0.403, + 0.531, + 0.522, + 0.526, + 0.525, + 0.498, + 0.479, + 0.463, + 0.458, + 0.461, + 0.463, + 0.518, + 0.514, + 0.46, + 0.454, + 0.442 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "colgrep", + "index_ms": 2864.5, + "p50_ms": 125.32, + "p90_ms": 136.08, + "p95_ms": 137.17, + "p99_ms": 152.52, + "latencies_ms": [ + 156.356, + 124.814, + 125.824, + 128.156, + 136.071, + 119.393, + 117.59, + 117.071, + 129.408, + 136.157, + 132.054, + 120.523, + 118.213, + 119.091, + 124.128, + 121.806, + 128.689, + 121.772, + 128.562, + 127.593 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 17.01, + "p90_ms": 18.35, + "p95_ms": 19.14, + "p99_ms": 19.68, + "latencies_ms": [ + 20.133, + 18.881, + 19.293, + 17.718, + 16.033, + 17.088, + 17.672, + 17.794, + 19.364, + 18.422, + 18.338, + 15.707, + 17.025, + 16.96, + 17.43, + 17.093, + 16.641, + 17.0, + 17.249, + 17.332, + 17.762, + 17.746, + 15.755, + 16.515, + 16.552, + 17.679, + 17.31, + 17.674, + 18.195, + 16.534, + 16.737, + 16.069, + 16.381, + 16.867, + 18.239, + 16.983, + 17.39, + 16.366, + 17.438, + 17.049, + 15.709, + 16.72, + 19.133, + 17.615, + 16.411, + 17.762, + 17.677, + 15.82, + 15.909, + 15.367, + 15.458, + 16.434, + 15.731, + 17.484, + 16.647, + 16.48, + 15.775, + 16.286, + 16.003, + 16.163 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "semble", + "index_ms": 1143.3, + "p50_ms": 0.83, + "p90_ms": 1.18, + "p95_ms": 1.51, + "p99_ms": 1.63, + "latencies_ms": [ + 1.911, + 1.366, + 1.287, + 1.233, + 1.255, + 0.894, + 0.843, + 0.839, + 0.849, + 0.742, + 1.015, + 0.903, + 0.894, + 0.869, + 0.869, + 1.584, + 1.626, + 1.582, + 1.52, + 1.504, + 1.034, + 0.899, + 0.837, + 0.826, + 0.83, + 0.82, + 0.738, + 0.734, + 0.722, + 0.709, + 0.746, + 0.676, + 0.676, + 0.68, + 0.673, + 0.944, + 0.869, + 0.872, + 0.866, + 0.849, + 0.696, + 0.628, + 0.616, + 0.605, + 0.613, + 0.755, + 0.685, + 0.7, + 0.682, + 0.692, + 0.974, + 0.907, + 1.021, + 0.979, + 0.939, + 1.037, + 0.958, + 0.937, + 0.932, + 0.926, + 1.129, + 1.044, + 1.106, + 1.066, + 1.037, + 0.801, + 0.731, + 0.718, + 0.752, + 0.83, + 0.905, + 0.808, + 0.792, + 0.801, + 0.801, + 0.573, + 0.53, + 0.513, + 0.497, + 0.501, + 1.177, + 1.121, + 1.085, + 1.046, + 1.042, + 0.8, + 0.807, + 0.836, + 0.793, + 0.743, + 0.777, + 0.684, + 0.673, + 0.674, + 0.668, + 0.857, + 0.753, + 0.774, + 0.739, + 0.799 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "bm25", + "index_ms": 88.0, + "p50_ms": 0.67, + "p90_ms": 0.97, + "p95_ms": 1.29, + "p99_ms": 1.57, + "latencies_ms": [ + 1.636, + 1.112, + 1.059, + 1.02, + 1.021, + 0.594, + 0.546, + 0.536, + 0.55, + 0.529, + 0.831, + 0.768, + 0.753, + 0.84, + 0.793, + 1.57, + 1.454, + 1.322, + 1.295, + 1.288, + 0.809, + 0.737, + 0.727, + 0.737, + 0.712, + 0.671, + 0.626, + 0.616, + 0.652, + 0.619, + 0.621, + 0.592, + 0.575, + 0.574, + 0.578, + 0.793, + 0.735, + 0.717, + 0.706, + 0.728, + 0.539, + 0.491, + 0.489, + 0.477, + 0.482, + 0.631, + 0.577, + 0.587, + 0.574, + 0.569, + 0.844, + 0.761, + 0.734, + 0.73, + 0.725, + 0.872, + 0.801, + 0.786, + 0.777, + 0.784, + 0.935, + 0.867, + 0.854, + 0.845, + 0.839, + 0.644, + 0.591, + 0.583, + 0.574, + 0.566, + 0.675, + 0.613, + 0.606, + 0.602, + 0.603, + 0.534, + 0.458, + 0.449, + 0.443, + 0.441, + 0.959, + 0.831, + 0.824, + 0.839, + 0.815, + 0.672, + 0.603, + 0.606, + 0.671, + 0.627, + 0.602, + 0.525, + 0.53, + 0.527, + 0.497, + 0.744, + 0.66, + 0.742, + 0.779, + 0.737 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "colgrep", + "index_ms": 10659.8, + "p50_ms": 141.26, + "p90_ms": 156.42, + "p95_ms": 160.01, + "p99_ms": 185.65, + "latencies_ms": [ + 192.063, + 137.887, + 138.543, + 138.895, + 140.792, + 146.394, + 158.325, + 146.928, + 156.211, + 148.719, + 151.694, + 146.5, + 141.338, + 136.899, + 136.619, + 136.213, + 140.979, + 138.011, + 153.58, + 141.19 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.8, + "p90_ms": 19.17, + "p95_ms": 19.86, + "p99_ms": 23.55, + "latencies_ms": [ + 27.01, + 18.649, + 17.981, + 17.998, + 15.8, + 15.753, + 17.28, + 18.818, + 18.631, + 15.528, + 17.988, + 16.523, + 21.15, + 19.114, + 18.388, + 16.78, + 19.693, + 19.917, + 19.86, + 19.818, + 17.75, + 16.826, + 16.758, + 16.164, + 16.256, + 15.776, + 16.68, + 15.456, + 16.153, + 16.586, + 14.827, + 17.861, + 17.676, + 17.19, + 16.948, + 17.422, + 17.54, + 16.498, + 14.521, + 15.04, + 15.796, + 15.786, + 16.21, + 17.455, + 16.992, + 17.343, + 16.759, + 16.404, + 17.698, + 17.025, + 15.268, + 15.969, + 16.859, + 15.926, + 17.193, + 15.422, + 15.26, + 14.901, + 15.203, + 16.784 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "semble", + "index_ms": 793.2, + "p50_ms": 1.05, + "p90_ms": 5.16, + "p95_ms": 5.25, + "p99_ms": 5.84, + "latencies_ms": [ + 1.987, + 1.335, + 1.21, + 1.212, + 1.127, + 0.799, + 0.732, + 0.743, + 0.732, + 0.733, + 0.794, + 0.795, + 0.83, + 0.785, + 0.752, + 1.038, + 0.985, + 1.033, + 1.022, + 0.977, + 1.213, + 1.191, + 1.194, + 1.175, + 1.165, + 0.998, + 0.948, + 0.922, + 0.926, + 0.928, + 1.092, + 1.123, + 1.156, + 1.136, + 1.126, + 1.278, + 1.198, + 1.183, + 1.173, + 1.178, + 1.006, + 0.932, + 0.919, + 0.916, + 0.917, + 0.971, + 0.922, + 0.923, + 0.913, + 0.92, + 1.01, + 0.984, + 0.964, + 0.951, + 0.937, + 0.881, + 0.833, + 0.835, + 0.826, + 0.817, + 1.688, + 1.61, + 1.579, + 1.6, + 1.632, + 6.032, + 5.07, + 5.19, + 5.195, + 5.308, + 1.432, + 1.06, + 1.059, + 0.946, + 0.932, + 0.985, + 1.029, + 0.911, + 0.878, + 0.976, + 1.192, + 1.342, + 1.025, + 1.024, + 0.99, + 5.43, + 4.758, + 4.661, + 5.024, + 5.208, + 5.807, + 5.211, + 5.035, + 5.089, + 5.156, + 5.031, + 5.838, + 4.854, + 5.008, + 5.245 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "bm25", + "index_ms": 52.7, + "p50_ms": 0.93, + "p90_ms": 3.89, + "p95_ms": 4.22, + "p99_ms": 4.27, + "latencies_ms": [ + 1.511, + 1.035, + 0.949, + 0.928, + 1.032, + 1.157, + 0.858, + 0.698, + 0.647, + 0.63, + 0.778, + 0.747, + 0.678, + 0.66, + 0.825, + 1.135, + 1.127, + 0.991, + 0.918, + 0.97, + 1.179, + 1.1, + 1.027, + 1.006, + 1.555, + 1.269, + 0.917, + 0.88, + 0.816, + 0.913, + 1.04, + 0.954, + 0.97, + 0.927, + 0.934, + 1.083, + 1.026, + 1.028, + 1.032, + 1.046, + 0.916, + 0.826, + 0.783, + 0.776, + 0.764, + 0.805, + 0.769, + 0.789, + 0.761, + 0.762, + 0.817, + 0.775, + 0.772, + 0.86, + 0.839, + 0.781, + 0.732, + 0.717, + 0.735, + 0.72, + 1.47, + 1.415, + 1.387, + 1.379, + 1.378, + 4.223, + 4.128, + 4.24, + 4.23, + 4.4, + 1.058, + 0.866, + 0.843, + 0.817, + 0.802, + 0.698, + 0.658, + 0.655, + 0.783, + 0.729, + 0.894, + 0.846, + 0.825, + 0.832, + 0.824, + 3.798, + 3.87, + 3.769, + 3.865, + 3.767, + 4.113, + 4.15, + 4.246, + 4.272, + 4.125, + 3.202, + 3.19, + 3.194, + 3.18, + 3.274 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "colgrep", + "index_ms": 5204.4, + "p50_ms": 117.89, + "p90_ms": 132.46, + "p95_ms": 143.46, + "p99_ms": 189.14, + "latencies_ms": [ + 200.565, + 120.304, + 117.892, + 114.652, + 123.228, + 140.458, + 131.567, + 120.732, + 123.26, + 113.949, + 116.615, + 117.171, + 118.32, + 117.885, + 116.577, + 115.578, + 115.023, + 117.355, + 121.68, + 116.401 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.28, + "p90_ms": 15.58, + "p95_ms": 15.89, + "p99_ms": 17.68, + "latencies_ms": [ + 18.216, + 14.956, + 15.888, + 14.639, + 15.564, + 17.303, + 15.731, + 14.495, + 13.288, + 12.382, + 13.409, + 14.862, + 13.442, + 14.313, + 14.218, + 12.715, + 12.037, + 12.931, + 15.019, + 14.809, + 13.969, + 11.693, + 13.387, + 13.548, + 14.915, + 14.97, + 13.641, + 11.947, + 11.265, + 12.512, + 12.675, + 13.061, + 12.567, + 11.23, + 10.358, + 13.613, + 12.73, + 14.652, + 15.696, + 15.9, + 13.446, + 12.371, + 12.227, + 12.205, + 12.257, + 13.036, + 13.268, + 13.508, + 11.941, + 11.403, + 12.035, + 13.543, + 11.78, + 12.939, + 11.357, + 12.651, + 14.078, + 13.006, + 11.939, + 11.49 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "semble", + "index_ms": 959.9, + "p50_ms": 5.13, + "p90_ms": 6.98, + "p95_ms": 7.15, + "p99_ms": 7.64, + "latencies_ms": [ + 7.639, + 6.79, + 6.741, + 6.758, + 6.701, + 1.383, + 1.182, + 1.172, + 1.18, + 1.171, + 1.267, + 1.207, + 1.238, + 1.268, + 1.353, + 1.498, + 1.447, + 1.417, + 1.308, + 1.307, + 0.914, + 0.838, + 0.832, + 0.842, + 0.811, + 1.156, + 1.092, + 1.276, + 1.253, + 1.29, + 1.585, + 1.533, + 1.6, + 1.63, + 1.655, + 1.142, + 1.219, + 0.999, + 1.011, + 1.047, + 6.777, + 6.827, + 6.608, + 6.443, + 6.556, + 1.171, + 1.052, + 1.08, + 1.621, + 1.442, + 7.03, + 6.915, + 6.745, + 6.721, + 6.801, + 7.394, + 6.835, + 6.845, + 6.924, + 6.812, + 7.735, + 7.252, + 7.082, + 6.965, + 6.837, + 5.545, + 5.142, + 5.117, + 5.264, + 5.14, + 5.189, + 4.77, + 4.874, + 4.738, + 4.622, + 6.613, + 6.146, + 6.264, + 6.208, + 6.394, + 5.646, + 5.477, + 5.304, + 5.279, + 5.723, + 7.336, + 7.146, + 6.976, + 7.105, + 7.022, + 1.314, + 1.165, + 1.072, + 1.044, + 1.034, + 6.072, + 5.55, + 5.763, + 5.665, + 5.553 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "bm25", + "index_ms": 94.8, + "p50_ms": 4.38, + "p90_ms": 6.68, + "p95_ms": 8.06, + "p99_ms": 10.37, + "latencies_ms": [ + 12.019, + 10.184, + 8.456, + 10.352, + 9.471, + 1.886, + 1.592, + 2.12, + 2.91, + 2.298, + 2.125, + 1.71, + 1.61, + 1.507, + 1.617, + 1.806, + 1.993, + 1.869, + 2.084, + 1.476, + 1.057, + 1.496, + 1.548, + 0.89, + 0.926, + 1.749, + 1.439, + 1.099, + 2.18, + 1.589, + 2.02, + 1.429, + 1.581, + 1.906, + 2.203, + 1.373, + 1.084, + 1.32, + 0.978, + 0.953, + 7.466, + 8.039, + 6.707, + 5.993, + 5.757, + 1.076, + 0.927, + 1.093, + 0.942, + 0.868, + 5.711, + 5.983, + 6.444, + 6.096, + 6.036, + 5.676, + 5.716, + 7.537, + 6.449, + 6.55, + 6.881, + 6.598, + 6.379, + 6.314, + 6.261, + 4.947, + 5.006, + 5.194, + 5.167, + 5.172, + 4.618, + 4.398, + 4.092, + 4.113, + 4.356, + 5.465, + 5.641, + 6.178, + 5.848, + 5.717, + 4.36, + 4.429, + 4.506, + 4.624, + 4.368, + 6.491, + 6.237, + 6.679, + 6.273, + 6.426, + 1.362, + 1.088, + 1.116, + 1.005, + 0.901, + 5.258, + 5.121, + 5.391, + 5.045, + 4.842 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "colgrep", + "index_ms": 5191.0, + "p50_ms": 125.68, + "p90_ms": 142.08, + "p95_ms": 146.85, + "p99_ms": 156.91, + "latencies_ms": [ + 141.619, + 159.427, + 125.806, + 129.463, + 127.668, + 130.313, + 125.558, + 123.38, + 122.463, + 124.638, + 121.233, + 122.778, + 119.576, + 120.32, + 118.497, + 127.761, + 122.103, + 130.618, + 146.187, + 137.91 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.74, + "p90_ms": 17.76, + "p95_ms": 18.45, + "p99_ms": 20.93, + "latencies_ms": [ + 18.963, + 16.823, + 17.214, + 16.259, + 15.438, + 16.712, + 15.274, + 16.299, + 17.003, + 15.11, + 16.095, + 14.992, + 15.835, + 17.322, + 15.086, + 16.011, + 15.743, + 15.249, + 15.715, + 22.161, + 20.067, + 17.121, + 17.999, + 16.264, + 18.422, + 17.731, + 17.309, + 15.303, + 15.641, + 16.934, + 15.502, + 15.117, + 15.611, + 15.17, + 15.411, + 15.692, + 15.674, + 15.735, + 16.245, + 17.984, + 17.018, + 14.684, + 14.943, + 16.141, + 15.767, + 16.724, + 16.876, + 15.277, + 15.47, + 15.01, + 14.755, + 17.105, + 15.447, + 14.94, + 14.97, + 14.628, + 14.322, + 14.661, + 16.213, + 14.853 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "semble", + "index_ms": 582.2, + "p50_ms": 0.82, + "p90_ms": 5.22, + "p95_ms": 6.29, + "p99_ms": 7.37, + "latencies_ms": [ + 1.801, + 1.183, + 1.113, + 1.089, + 1.066, + 0.843, + 0.832, + 0.797, + 0.765, + 0.8, + 0.809, + 0.698, + 0.678, + 0.683, + 0.681, + 0.736, + 0.706, + 0.803, + 0.871, + 0.799, + 0.854, + 0.755, + 0.742, + 0.734, + 0.734, + 5.978, + 5.373, + 5.24, + 5.35, + 5.206, + 0.766, + 0.647, + 0.598, + 0.587, + 0.575, + 0.741, + 0.687, + 0.671, + 0.669, + 0.709, + 1.084, + 0.992, + 0.97, + 0.972, + 0.975, + 0.735, + 0.692, + 0.695, + 0.692, + 0.703, + 0.926, + 0.944, + 1.025, + 0.954, + 0.912, + 0.703, + 0.677, + 0.69, + 0.668, + 0.648, + 0.78, + 0.821, + 0.806, + 0.757, + 0.749, + 0.81, + 0.76, + 0.743, + 0.707, + 0.71, + 0.851, + 0.816, + 0.809, + 0.863, + 0.959, + 1.495, + 1.214, + 1.159, + 1.427, + 1.541, + 5.228, + 4.861, + 5.078, + 4.777, + 5.017, + 8.634, + 7.287, + 7.165, + 7.026, + 7.062, + 5.178, + 4.569, + 4.501, + 4.624, + 4.548 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "bm25", + "index_ms": 44.3, + "p50_ms": 0.65, + "p90_ms": 3.95, + "p95_ms": 4.44, + "p99_ms": 5.06, + "latencies_ms": [ + 1.39, + 1.036, + 0.957, + 0.933, + 1.0, + 0.642, + 0.564, + 0.54, + 0.539, + 0.555, + 0.647, + 0.836, + 0.737, + 0.667, + 0.601, + 0.696, + 0.669, + 0.593, + 0.585, + 0.643, + 0.716, + 0.627, + 0.596, + 0.589, + 0.584, + 4.086, + 4.132, + 4.138, + 4.227, + 4.095, + 0.529, + 0.484, + 0.473, + 0.475, + 0.467, + 0.627, + 0.613, + 0.66, + 0.604, + 0.579, + 0.751, + 0.716, + 0.701, + 0.698, + 0.694, + 0.592, + 0.556, + 0.553, + 0.57, + 0.565, + 0.804, + 0.769, + 0.751, + 0.749, + 0.731, + 0.541, + 0.508, + 0.503, + 0.503, + 0.522, + 0.616, + 0.574, + 0.567, + 0.567, + 0.561, + 0.613, + 0.598, + 0.58, + 0.581, + 0.574, + 0.643, + 0.614, + 0.61, + 0.607, + 0.608, + 0.923, + 0.914, + 0.889, + 0.962, + 0.922, + 3.741, + 3.597, + 3.564, + 3.64, + 3.733, + 4.987, + 5.05, + 5.105, + 5.054, + 4.947, + 3.611, + 3.633, + 3.577, + 3.672, + 3.612 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "colgrep", + "index_ms": 19008.0, + "p50_ms": 157.38, + "p90_ms": 184.89, + "p95_ms": 198.93, + "p99_ms": 256.44, + "latencies_ms": [ + 270.816, + 173.743, + 183.38, + 190.945, + 177.629, + 159.606, + 155.944, + 157.38, + 159.023, + 158.657, + 156.79, + 160.48, + 155.212, + 154.499, + 154.058, + 152.181, + 155.184, + 156.612, + 156.458 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 12.91, + "p90_ms": 14.26, + "p95_ms": 14.76, + "p99_ms": 18.29, + "latencies_ms": [ + 21.511, + 13.327, + 13.77, + 13.233, + 13.114, + 12.912, + 13.352, + 13.194, + 13.059, + 12.979, + 12.122, + 12.572, + 14.615, + 13.296, + 11.59, + 15.753, + 14.709, + 13.515, + 11.463, + 12.509, + 12.895, + 11.97, + 12.904, + 14.192, + 12.694, + 12.376, + 11.399, + 12.431, + 12.679, + 12.649, + 12.945, + 13.48, + 13.436, + 11.849, + 13.125, + 12.664, + 12.71, + 11.588, + 11.233, + 13.24, + 11.474, + 14.373, + 14.03, + 14.96, + 13.468, + 11.952, + 12.719, + 12.698, + 12.499, + 12.174, + 12.473, + 13.045, + 13.309, + 12.92, + 13.192, + 11.866, + 11.219 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "semble", + "index_ms": 558.4, + "p50_ms": 0.88, + "p90_ms": 3.94, + "p95_ms": 4.9, + "p99_ms": 5.71, + "latencies_ms": [ + 1.13, + 0.757, + 0.682, + 0.647, + 0.61, + 0.565, + 0.479, + 0.462, + 0.466, + 0.494, + 0.8, + 1.148, + 0.995, + 0.844, + 0.834, + 0.894, + 0.802, + 0.817, + 0.792, + 0.977, + 0.925, + 1.069, + 1.062, + 0.894, + 0.849, + 5.705, + 4.639, + 4.457, + 4.694, + 4.792, + 0.987, + 0.933, + 0.786, + 0.715, + 0.661, + 0.68, + 0.564, + 0.724, + 0.664, + 0.581, + 0.515, + 0.472, + 0.464, + 0.463, + 0.605, + 0.84, + 0.803, + 1.119, + 0.841, + 0.795, + 0.722, + 0.611, + 0.607, + 0.733, + 0.639, + 1.409, + 1.356, + 1.293, + 1.389, + 1.012, + 0.948, + 0.879, + 0.843, + 0.846, + 0.826, + 0.985, + 0.871, + 0.907, + 0.987, + 1.05, + 0.889, + 0.804, + 0.765, + 0.787, + 0.779, + 1.116, + 1.056, + 1.084, + 1.017, + 0.978, + 3.881, + 3.397, + 3.388, + 3.24, + 3.244, + 5.983, + 4.996, + 4.892, + 5.195, + 5.134, + 3.578, + 3.522, + 3.447, + 3.495, + 3.421, + 0.9, + 0.755, + 0.727, + 0.718, + 0.705 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "bm25", + "index_ms": 45.2, + "p50_ms": 0.7, + "p90_ms": 3.54, + "p95_ms": 4.03, + "p99_ms": 4.72, + "latencies_ms": [ + 0.849, + 0.513, + 0.467, + 0.454, + 0.45, + 0.451, + 0.404, + 0.395, + 0.394, + 0.393, + 0.589, + 0.54, + 0.533, + 0.536, + 0.535, + 0.643, + 0.608, + 0.628, + 1.003, + 1.001, + 0.736, + 0.725, + 0.737, + 0.673, + 0.629, + 3.559, + 3.64, + 3.537, + 3.554, + 3.996, + 0.658, + 0.543, + 0.532, + 0.525, + 0.652, + 0.726, + 0.711, + 0.505, + 0.445, + 0.436, + 0.444, + 0.413, + 0.407, + 0.411, + 0.42, + 0.666, + 0.696, + 0.674, + 0.591, + 0.568, + 0.602, + 0.554, + 0.685, + 0.8, + 0.918, + 0.988, + 1.193, + 1.125, + 0.853, + 0.733, + 0.964, + 1.377, + 0.879, + 0.733, + 0.674, + 0.728, + 0.733, + 0.73, + 0.684, + 0.662, + 0.697, + 0.63, + 0.912, + 0.768, + 0.674, + 1.102, + 1.046, + 0.855, + 0.765, + 0.755, + 3.281, + 3.367, + 3.539, + 3.313, + 3.256, + 4.715, + 4.648, + 4.809, + 4.683, + 4.634, + 2.925, + 2.863, + 3.103, + 3.082, + 2.925, + 0.76, + 0.623, + 0.598, + 0.58, + 0.588 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "colgrep", + "index_ms": 5442.0, + "p50_ms": 127.87, + "p90_ms": 150.74, + "p95_ms": 156.5, + "p99_ms": 178.09, + "latencies_ms": [ + 183.493, + 139.902, + 150.252, + 133.797, + 122.263, + 121.553, + 119.226, + 119.393, + 144.924, + 128.81, + 127.117, + 123.251, + 128.019, + 132.918, + 127.728, + 121.637, + 155.079, + 129.262, + 125.437, + 120.764 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.67, + "p90_ms": 15.34, + "p95_ms": 15.63, + "p99_ms": 15.95, + "latencies_ms": [ + 15.345, + 12.214, + 13.102, + 13.799, + 13.861, + 15.017, + 13.693, + 15.205, + 12.43, + 11.98, + 13.477, + 14.115, + 11.881, + 13.545, + 15.778, + 14.154, + 14.197, + 14.39, + 13.277, + 13.658, + 13.394, + 12.123, + 12.835, + 12.28, + 14.576, + 12.022, + 11.584, + 12.761, + 12.673, + 13.793, + 11.863, + 12.704, + 12.934, + 15.07, + 16.195, + 14.14, + 14.954, + 12.46, + 14.918, + 11.872, + 13.681, + 13.402, + 13.277, + 14.317, + 14.5, + 12.428, + 12.075, + 13.357, + 14.425, + 13.421, + 14.19, + 15.748, + 13.081, + 15.339, + 14.212, + 12.95, + 13.847, + 15.355, + 15.624, + 14.939 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "semble", + "index_ms": 475.4, + "p50_ms": 4.42, + "p90_ms": 8.31, + "p95_ms": 13.32, + "p99_ms": 22.32, + "latencies_ms": [ + 1.388, + 1.075, + 0.973, + 0.921, + 0.893, + 5.319, + 4.406, + 4.444, + 4.727, + 4.611, + 5.186, + 4.134, + 4.499, + 4.486, + 4.479, + 5.967, + 5.459, + 5.109, + 7.575, + 4.786, + 5.591, + 4.989, + 4.543, + 4.429, + 4.673, + 5.268, + 4.713, + 4.752, + 5.706, + 5.058, + 5.479, + 4.786, + 4.237, + 4.203, + 4.088, + 23.346, + 22.313, + 21.833, + 21.426, + 21.302, + 4.917, + 4.328, + 4.333, + 4.288, + 4.236, + 5.048, + 4.697, + 4.557, + 4.636, + 4.752, + 4.538, + 4.082, + 4.039, + 4.229, + 4.79, + 0.836, + 0.872, + 1.162, + 1.11, + 0.865, + 0.955, + 0.903, + 0.866, + 0.847, + 0.851, + 1.456, + 1.025, + 0.859, + 0.799, + 0.831, + 4.506, + 4.044, + 4.06, + 3.974, + 3.991, + 0.919, + 0.803, + 0.777, + 0.752, + 0.748, + 12.895, + 12.375, + 12.382, + 12.032, + 11.956, + 7.905, + 6.982, + 6.859, + 6.892, + 7.611, + 1.497, + 1.005, + 0.837, + 0.745, + 0.677, + 4.417, + 4.096, + 4.231, + 4.626, + 4.334 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "bm25", + "index_ms": 32.5, + "p50_ms": 3.14, + "p90_ms": 5.3, + "p95_ms": 11.78, + "p99_ms": 22.24, + "latencies_ms": [ + 1.168, + 0.835, + 0.808, + 0.733, + 0.717, + 3.383, + 3.231, + 3.373, + 3.444, + 3.379, + 3.389, + 3.397, + 3.393, + 3.278, + 3.375, + 2.991, + 3.152, + 3.092, + 2.95, + 3.011, + 3.115, + 3.13, + 3.201, + 3.105, + 3.191, + 3.385, + 3.282, + 3.657, + 3.563, + 4.42, + 3.408, + 3.241, + 3.211, + 3.233, + 3.211, + 23.044, + 22.181, + 22.231, + 21.608, + 21.833, + 4.275, + 3.947, + 3.85, + 3.705, + 3.908, + 3.729, + 3.825, + 3.7, + 3.676, + 3.744, + 3.157, + 3.018, + 3.054, + 3.017, + 3.078, + 0.671, + 0.618, + 0.603, + 0.606, + 0.672, + 0.727, + 0.7, + 0.68, + 0.679, + 0.676, + 0.655, + 0.653, + 0.638, + 0.632, + 0.63, + 3.164, + 3.084, + 3.136, + 3.078, + 2.953, + 0.645, + 0.658, + 0.626, + 0.584, + 0.577, + 11.262, + 11.238, + 11.227, + 11.135, + 11.264, + 4.652, + 4.529, + 4.534, + 4.57, + 4.597, + 0.639, + 0.58, + 0.569, + 0.565, + 0.575, + 3.081, + 3.002, + 2.979, + 2.958, + 2.97 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "colgrep", + "index_ms": 3558.3, + "p50_ms": 123.26, + "p90_ms": 132.4, + "p95_ms": 138.06, + "p99_ms": 159.88, + "latencies_ms": [ + 165.339, + 136.619, + 130.425, + 121.88, + 124.577, + 122.537, + 122.504, + 128.48, + 126.739, + 116.912, + 117.892, + 123.986, + 124.393, + 127.468, + 120.757, + 131.934, + 119.273, + 118.897, + 115.166, + 115.937 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 12.05, + "p90_ms": 14.61, + "p95_ms": 16.54, + "p99_ms": 19.1, + "latencies_ms": [ + 13.979, + 12.14, + 12.59, + 11.546, + 11.391, + 13.283, + 12.147, + 11.632, + 11.213, + 13.442, + 18.902, + 17.056, + 16.51, + 14.249, + 14.446, + 12.288, + 11.504, + 11.362, + 10.249, + 11.337, + 11.071, + 12.124, + 13.161, + 11.752, + 11.751, + 12.562, + 11.771, + 11.506, + 11.322, + 11.28, + 10.834, + 14.162, + 19.395, + 14.376, + 15.539, + 14.511, + 15.286, + 13.914, + 13.129, + 13.742, + 14.531, + 13.583, + 10.039, + 10.83, + 11.354, + 10.642, + 13.269, + 11.942, + 11.735, + 11.98, + 12.232, + 11.188, + 10.986, + 11.794, + 11.835, + 11.222, + 13.014, + 12.969, + 11.974, + 11.342 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "semble", + "index_ms": 1291.6, + "p50_ms": 1.6, + "p90_ms": 9.67, + "p95_ms": 10.59, + "p99_ms": 13.2, + "latencies_ms": [ + 14.594, + 13.091, + 12.54, + 12.632, + 13.186, + 1.876, + 1.665, + 1.627, + 1.61, + 1.605, + 1.414, + 1.377, + 1.329, + 1.31, + 1.312, + 8.908, + 8.287, + 9.635, + 9.036, + 8.836, + 7.863, + 7.247, + 7.094, + 6.987, + 6.948, + 1.629, + 1.504, + 1.59, + 1.544, + 1.492, + 1.399, + 1.356, + 1.337, + 1.332, + 1.427, + 1.282, + 1.215, + 1.193, + 1.357, + 1.326, + 1.37, + 1.442, + 1.233, + 1.212, + 1.184, + 0.938, + 0.913, + 0.889, + 0.896, + 0.886, + 1.35, + 1.379, + 1.336, + 1.3, + 1.298, + 1.336, + 1.293, + 1.274, + 1.302, + 1.325, + 1.481, + 1.444, + 1.406, + 1.414, + 1.416, + 1.601, + 1.57, + 1.521, + 1.5, + 1.517, + 1.663, + 1.601, + 1.642, + 1.617, + 1.623, + 1.841, + 1.811, + 1.79, + 1.911, + 1.939, + 9.872, + 9.169, + 8.947, + 9.05, + 8.764, + 7.799, + 7.146, + 6.96, + 6.841, + 6.722, + 10.068, + 9.668, + 9.734, + 10.487, + 9.685, + 7.788, + 7.226, + 7.326, + 7.163, + 7.058 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "bm25", + "index_ms": 107.0, + "p50_ms": 1.36, + "p90_ms": 8.84, + "p95_ms": 9.17, + "p99_ms": 10.93, + "latencies_ms": [ + 11.625, + 10.92, + 10.505, + 10.491, + 10.49, + 1.476, + 1.387, + 1.378, + 1.346, + 1.351, + 1.043, + 0.988, + 0.982, + 0.963, + 0.936, + 8.134, + 8.18, + 8.853, + 8.757, + 8.836, + 6.597, + 6.606, + 6.915, + 6.834, + 6.717, + 1.564, + 1.419, + 1.305, + 1.293, + 1.277, + 1.287, + 1.525, + 1.363, + 1.3, + 1.282, + 1.189, + 1.129, + 1.107, + 1.104, + 1.089, + 1.084, + 1.113, + 1.04, + 1.015, + 1.019, + 0.883, + 0.768, + 0.726, + 0.759, + 0.785, + 1.074, + 0.944, + 0.916, + 0.933, + 0.939, + 1.087, + 1.093, + 1.18, + 1.066, + 1.003, + 1.276, + 1.199, + 1.165, + 1.212, + 1.286, + 1.352, + 1.324, + 1.35, + 1.689, + 1.461, + 1.415, + 1.346, + 1.356, + 1.307, + 1.343, + 1.858, + 1.66, + 1.717, + 1.659, + 1.742, + 8.352, + 8.118, + 8.063, + 8.141, + 8.308, + 6.797, + 6.76, + 6.627, + 6.78, + 6.71, + 9.102, + 8.919, + 8.763, + 8.895, + 8.894, + 6.527, + 6.617, + 6.603, + 6.475, + 6.511 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "colgrep", + "index_ms": 9427.5, + "p50_ms": 125.15, + "p90_ms": 134.55, + "p95_ms": 136.16, + "p99_ms": 147.03, + "latencies_ms": [ + 149.746, + 132.019, + 135.443, + 134.452, + 127.785, + 125.943, + 122.619, + 123.23, + 122.195, + 122.427, + 122.494, + 124.201, + 124.354, + 122.154, + 123.718, + 123.394, + 130.246, + 129.158, + 129.502, + 127.046 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.66, + "p90_ms": 18.23, + "p95_ms": 18.71, + "p99_ms": 20.7, + "latencies_ms": [ + 23.281, + 16.524, + 15.735, + 17.149, + 13.894, + 16.128, + 15.772, + 18.908, + 17.741, + 15.792, + 16.257, + 17.568, + 15.747, + 16.456, + 15.615, + 16.345, + 17.282, + 17.063, + 14.834, + 14.089, + 13.394, + 14.917, + 17.114, + 15.429, + 16.458, + 16.845, + 15.471, + 14.985, + 14.953, + 15.785, + 15.977, + 15.554, + 15.965, + 17.219, + 17.73, + 18.23, + 17.024, + 16.807, + 16.63, + 18.701, + 17.43, + 17.038, + 15.544, + 16.748, + 15.61, + 16.773, + 18.226, + 16.844, + 17.965, + 18.793, + 17.981, + 17.481, + 16.489, + 17.499, + 18.524, + 16.155, + 17.603, + 16.698, + 16.341, + 17.677 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "semble", + "index_ms": 119.5, + "p50_ms": 0.82, + "p90_ms": 2.73, + "p95_ms": 3.16, + "p99_ms": 3.33, + "latencies_ms": [ + 1.333, + 0.79, + 0.699, + 0.654, + 0.634, + 0.649, + 0.551, + 0.514, + 0.501, + 0.503, + 0.637, + 0.676, + 0.631, + 0.598, + 0.596, + 0.689, + 0.626, + 0.711, + 0.843, + 0.678, + 3.219, + 2.461, + 2.268, + 2.279, + 2.24, + 2.287, + 1.966, + 1.802, + 1.783, + 1.74, + 2.931, + 2.526, + 2.401, + 2.498, + 2.519, + 0.654, + 0.569, + 0.693, + 0.641, + 0.543, + 0.626, + 0.574, + 0.586, + 0.57, + 0.569, + 0.698, + 0.664, + 0.776, + 0.669, + 0.649, + 0.6, + 0.577, + 0.6, + 0.579, + 0.562, + 0.632, + 0.634, + 0.598, + 0.583, + 0.579, + 0.703, + 0.677, + 0.672, + 0.679, + 0.872, + 0.801, + 0.682, + 0.671, + 0.883, + 1.898, + 1.49, + 1.39, + 1.538, + 1.493, + 2.184, + 1.427, + 1.361, + 1.674, + 1.53, + 1.677, + 1.781, + 1.391, + 1.112, + 1.585, + 1.419, + 3.637, + 3.168, + 3.101, + 3.0, + 3.278, + 3.324, + 2.609, + 2.71, + 2.657, + 2.465, + 3.16, + 2.607, + 2.507, + 3.006, + 2.59 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "bm25", + "index_ms": 9.4, + "p50_ms": 0.57, + "p90_ms": 1.91, + "p95_ms": 1.99, + "p99_ms": 2.06, + "latencies_ms": [ + 0.97, + 0.661, + 0.585, + 0.567, + 0.54, + 0.506, + 0.469, + 0.46, + 0.459, + 0.553, + 0.745, + 0.577, + 0.522, + 0.486, + 0.486, + 0.618, + 0.566, + 0.568, + 0.677, + 0.643, + 2.054, + 2.084, + 1.955, + 2.034, + 2.057, + 0.893, + 0.831, + 0.812, + 0.791, + 0.79, + 1.9, + 1.9, + 1.913, + 1.867, + 1.851, + 0.464, + 0.426, + 0.518, + 0.447, + 0.417, + 0.526, + 0.488, + 0.482, + 0.485, + 0.479, + 0.574, + 0.562, + 0.554, + 0.555, + 0.561, + 0.533, + 0.507, + 0.496, + 0.496, + 0.498, + 0.532, + 0.571, + 0.556, + 0.526, + 0.511, + 0.602, + 0.571, + 0.567, + 0.576, + 0.57, + 0.512, + 0.506, + 0.491, + 0.495, + 0.495, + 0.532, + 0.501, + 0.492, + 0.497, + 0.493, + 0.583, + 0.582, + 0.578, + 0.562, + 0.577, + 0.619, + 0.565, + 0.549, + 0.53, + 0.529, + 1.475, + 1.415, + 1.438, + 1.486, + 1.498, + 1.939, + 1.917, + 1.892, + 1.975, + 2.008, + 1.992, + 1.902, + 1.86, + 1.897, + 1.886 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "colgrep", + "index_ms": 1837.6, + "p50_ms": 112.68, + "p90_ms": 121.2, + "p95_ms": 125.98, + "p99_ms": 133.79, + "latencies_ms": [ + 135.739, + 108.823, + 110.647, + 109.423, + 112.697, + 109.626, + 117.432, + 114.531, + 119.737, + 118.358, + 120.732, + 125.465, + 116.396, + 113.607, + 110.165, + 110.96, + 109.932, + 110.34, + 112.659, + 110.925 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.83, + "p90_ms": 16.44, + "p95_ms": 16.61, + "p99_ms": 26.14, + "latencies_ms": [ + 15.173, + 15.426, + 12.159, + 13.394, + 13.374, + 12.205, + 12.352, + 14.332, + 14.163, + 16.436, + 14.431, + 14.115, + 13.829, + 14.579, + 15.322, + 14.62, + 16.292, + 14.353, + 16.617, + 14.646, + 14.772, + 14.772, + 15.568, + 15.835, + 14.772, + 15.892, + 14.209, + 13.142, + 16.29, + 15.178, + 16.608, + 16.576, + 16.255, + 16.304, + 14.335, + 14.663, + 15.73, + 15.384, + 17.494, + 14.895, + 16.231, + 16.479, + 38.579, + 15.268, + 15.285, + 15.872, + 15.172, + 15.408, + 14.576, + 15.959, + 14.587, + 15.241, + 14.16, + 15.062, + 13.492, + 12.444, + 13.658, + 12.203, + 13.046, + 14.677 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "semble", + "index_ms": 391.3, + "p50_ms": 1.2, + "p90_ms": 5.68, + "p95_ms": 6.05, + "p99_ms": 6.42, + "latencies_ms": [ + 7.543, + 6.201, + 6.209, + 6.225, + 6.413, + 1.376, + 1.274, + 1.24, + 1.162, + 1.387, + 1.079, + 0.909, + 0.847, + 0.836, + 0.83, + 1.105, + 1.076, + 1.06, + 1.076, + 1.077, + 1.004, + 0.951, + 0.945, + 0.931, + 1.031, + 1.06, + 0.991, + 0.953, + 0.947, + 0.955, + 0.847, + 0.804, + 0.772, + 0.781, + 0.772, + 0.683, + 0.657, + 0.645, + 0.765, + 0.701, + 3.645, + 3.398, + 3.647, + 3.554, + 3.435, + 0.959, + 0.939, + 0.923, + 0.912, + 0.914, + 1.098, + 1.069, + 1.058, + 1.061, + 1.099, + 5.585, + 5.665, + 5.573, + 5.32, + 5.588, + 5.646, + 5.069, + 5.088, + 5.184, + 4.969, + 4.202, + 3.665, + 3.675, + 3.566, + 3.409, + 5.032, + 4.586, + 4.485, + 4.527, + 4.692, + 4.128, + 3.948, + 3.908, + 3.639, + 3.834, + 5.921, + 6.039, + 5.849, + 5.836, + 5.901, + 1.197, + 1.197, + 1.187, + 1.122, + 1.084, + 0.896, + 0.854, + 0.84, + 0.859, + 0.837, + 4.151, + 3.727, + 3.697, + 3.47, + 3.453 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "bm25", + "index_ms": 28.3, + "p50_ms": 0.84, + "p90_ms": 4.69, + "p95_ms": 5.29, + "p99_ms": 5.43, + "latencies_ms": [ + 5.549, + 5.327, + 5.342, + 5.161, + 5.257, + 0.79, + 0.716, + 0.709, + 0.715, + 0.737, + 0.793, + 0.76, + 0.743, + 0.725, + 0.725, + 0.867, + 0.815, + 0.82, + 0.809, + 0.818, + 0.761, + 0.771, + 0.794, + 0.804, + 0.784, + 0.702, + 0.682, + 0.671, + 0.66, + 0.639, + 0.599, + 0.582, + 0.577, + 0.567, + 0.573, + 0.526, + 0.525, + 0.538, + 0.555, + 0.537, + 2.99, + 2.858, + 2.786, + 2.904, + 2.925, + 0.84, + 0.781, + 0.735, + 0.778, + 0.724, + 0.858, + 0.81, + 0.795, + 0.797, + 0.788, + 4.637, + 4.524, + 4.523, + 4.289, + 4.426, + 4.037, + 4.041, + 4.001, + 4.031, + 4.047, + 2.408, + 2.427, + 2.464, + 2.454, + 2.397, + 3.834, + 3.881, + 3.862, + 3.873, + 3.815, + 3.804, + 2.801, + 2.852, + 2.79, + 2.807, + 5.29, + 5.288, + 5.431, + 5.27, + 5.129, + 0.846, + 0.823, + 0.812, + 0.87, + 0.853, + 0.776, + 0.751, + 0.743, + 0.731, + 0.732, + 2.729, + 2.748, + 2.706, + 2.735, + 2.725 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "colgrep", + "index_ms": 4203.9, + "p50_ms": 132.17, + "p90_ms": 176.25, + "p95_ms": 185.71, + "p99_ms": 193.1, + "latencies_ms": [ + 194.947, + 175.255, + 185.222, + 146.384, + 140.544, + 145.885, + 139.338, + 143.563, + 138.686, + 118.646, + 117.767, + 119.017, + 116.176, + 120.418, + 117.155, + 116.15, + 130.298, + 134.045, + 126.093, + 130.171 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.18, + "p90_ms": 16.62, + "p95_ms": 17.85, + "p99_ms": 19.86, + "latencies_ms": [ + 15.297, + 14.958, + 14.298, + 15.825, + 16.618, + 14.295, + 14.531, + 14.158, + 14.176, + 14.759, + 13.962, + 16.115, + 20.619, + 19.328, + 17.829, + 18.264, + 16.13, + 15.878, + 16.542, + 16.676, + 14.897, + 16.03, + 15.424, + 15.194, + 16.657, + 15.782, + 14.178, + 13.404, + 14.485, + 13.313, + 13.58, + 13.461, + 15.052, + 12.865, + 12.555, + 12.388, + 13.029, + 12.541, + 12.954, + 12.09, + 12.15, + 13.447, + 13.308, + 12.022, + 12.828, + 11.737, + 13.317, + 12.221, + 15.606, + 14.835, + 14.41, + 13.408, + 13.819, + 13.734, + 12.055, + 12.88, + 12.431, + 12.948, + 14.175, + 15.08 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "semble", + "index_ms": 473.5, + "p50_ms": 0.84, + "p90_ms": 1.74, + "p95_ms": 2.79, + "p99_ms": 4.37, + "latencies_ms": [ + 1.285, + 0.918, + 0.878, + 0.865, + 0.843, + 0.809, + 0.742, + 0.733, + 0.76, + 0.825, + 0.898, + 0.821, + 0.824, + 0.811, + 1.059, + 1.185, + 1.78, + 1.901, + 1.736, + 1.725, + 1.984, + 1.558, + 1.854, + 2.724, + 1.246, + 1.212, + 1.532, + 1.591, + 1.221, + 0.987, + 0.834, + 0.739, + 0.719, + 0.864, + 0.798, + 0.932, + 0.831, + 0.823, + 0.909, + 1.108, + 1.133, + 1.049, + 1.094, + 0.978, + 0.818, + 4.9, + 4.175, + 4.369, + 4.109, + 4.018, + 1.075, + 0.912, + 0.834, + 0.802, + 0.873, + 1.167, + 0.962, + 0.832, + 0.776, + 0.768, + 0.85, + 0.794, + 0.779, + 0.772, + 0.787, + 1.018, + 0.945, + 0.911, + 0.922, + 0.933, + 0.839, + 0.794, + 0.805, + 0.82, + 0.911, + 0.834, + 0.752, + 0.73, + 0.742, + 0.779, + 0.911, + 0.8, + 0.772, + 0.765, + 0.762, + 0.775, + 0.724, + 0.717, + 0.703, + 0.724, + 0.74, + 0.687, + 0.683, + 0.758, + 0.703, + 0.813, + 0.962, + 0.97, + 0.829, + 0.766 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "bm25", + "index_ms": 35.4, + "p50_ms": 0.66, + "p90_ms": 0.79, + "p95_ms": 1.28, + "p99_ms": 3.21, + "latencies_ms": [ + 1.18, + 0.784, + 0.727, + 0.715, + 0.721, + 0.711, + 0.662, + 0.663, + 0.665, + 0.65, + 0.785, + 0.724, + 0.683, + 0.657, + 0.653, + 0.757, + 0.74, + 0.706, + 0.695, + 0.694, + 0.799, + 0.772, + 0.768, + 0.754, + 0.859, + 0.825, + 0.742, + 0.721, + 0.711, + 0.711, + 0.603, + 0.571, + 0.574, + 0.565, + 0.566, + 0.699, + 0.672, + 0.654, + 0.686, + 0.73, + 0.684, + 0.628, + 0.616, + 0.607, + 0.603, + 3.226, + 3.156, + 3.206, + 3.135, + 3.168, + 0.723, + 0.658, + 0.641, + 0.629, + 0.628, + 0.65, + 0.631, + 0.623, + 0.624, + 0.621, + 0.742, + 0.709, + 0.71, + 0.717, + 0.713, + 0.796, + 0.758, + 0.726, + 0.718, + 0.716, + 0.694, + 0.67, + 0.656, + 0.658, + 0.662, + 0.661, + 0.623, + 0.627, + 0.623, + 0.622, + 0.652, + 0.62, + 0.613, + 0.601, + 0.604, + 0.611, + 0.589, + 0.583, + 0.578, + 0.583, + 0.625, + 0.62, + 0.596, + 0.605, + 0.605, + 0.751, + 0.67, + 0.65, + 0.64, + 0.634 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "colgrep", + "index_ms": 4045.7, + "p50_ms": 124.38, + "p90_ms": 144.56, + "p95_ms": 146.8, + "p99_ms": 157.05, + "latencies_ms": [ + 159.617, + 128.982, + 142.265, + 120.829, + 128.587, + 128.187, + 121.303, + 146.125, + 134.464, + 144.387, + 124.982, + 114.622, + 114.102, + 123.771, + 115.702, + 112.531, + 115.236, + 123.758, + 134.306, + 123.22 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.67, + "p90_ms": 15.63, + "p95_ms": 16.39, + "p99_ms": 17.69, + "latencies_ms": [ + 15.09, + 16.776, + 15.302, + 15.253, + 14.483, + 16.366, + 13.379, + 13.013, + 13.124, + 12.289, + 12.85, + 16.111, + 13.831, + 13.507, + 13.52, + 13.644, + 14.723, + 13.972, + 15.541, + 14.324, + 13.978, + 14.059, + 15.165, + 14.598, + 13.725, + 13.716, + 13.534, + 12.699, + 17.995, + 17.479, + 15.277, + 15.42, + 15.135, + 15.062, + 13.348, + 15.612, + 15.812, + 13.87, + 13.753, + 12.975, + 12.302, + 13.704, + 12.805, + 11.589, + 11.303, + 11.146, + 11.477, + 11.409, + 10.793, + 13.223, + 12.39, + 13.72, + 12.372, + 12.245, + 12.447, + 11.773, + 11.782, + 11.519, + 11.357, + 13.237 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "semble", + "index_ms": 360.8, + "p50_ms": 0.97, + "p90_ms": 3.38, + "p95_ms": 3.47, + "p99_ms": 3.82, + "latencies_ms": [ + 1.321, + 0.88, + 0.828, + 0.816, + 0.812, + 1.095, + 1.052, + 0.972, + 0.949, + 0.926, + 0.872, + 0.799, + 0.793, + 0.792, + 0.785, + 0.751, + 0.697, + 0.876, + 0.78, + 0.735, + 4.144, + 3.427, + 3.426, + 3.453, + 3.426, + 3.552, + 3.133, + 3.109, + 3.152, + 3.11, + 3.755, + 3.257, + 3.258, + 3.281, + 3.307, + 0.936, + 0.879, + 0.878, + 0.888, + 0.863, + 1.076, + 1.051, + 1.174, + 1.131, + 1.085, + 0.981, + 0.941, + 0.925, + 0.919, + 0.917, + 0.953, + 0.956, + 1.003, + 0.956, + 0.932, + 0.917, + 0.893, + 0.88, + 0.882, + 0.87, + 0.795, + 0.767, + 0.761, + 0.782, + 0.812, + 0.895, + 0.874, + 0.836, + 0.819, + 0.806, + 1.009, + 0.956, + 0.95, + 0.946, + 0.948, + 1.03, + 0.971, + 0.964, + 0.957, + 0.972, + 1.11, + 1.07, + 1.065, + 1.101, + 1.192, + 1.094, + 1.024, + 0.989, + 0.979, + 0.974, + 3.819, + 3.463, + 3.38, + 3.256, + 3.37, + 3.632, + 3.225, + 3.167, + 3.18, + 3.168 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "bm25", + "index_ms": 24.0, + "p50_ms": 0.76, + "p90_ms": 2.52, + "p95_ms": 2.57, + "p99_ms": 2.61, + "latencies_ms": [ + 1.021, + 0.693, + 0.629, + 0.643, + 0.605, + 0.783, + 0.744, + 0.803, + 0.795, + 0.747, + 0.573, + 0.536, + 0.525, + 0.519, + 0.523, + 0.591, + 0.548, + 0.548, + 0.542, + 0.541, + 2.606, + 2.542, + 2.517, + 2.539, + 2.641, + 2.598, + 2.569, + 2.456, + 2.592, + 2.524, + 2.46, + 2.474, + 2.518, + 2.521, + 2.43, + 0.733, + 0.66, + 0.642, + 0.635, + 0.64, + 0.908, + 0.854, + 0.8, + 0.783, + 0.771, + 0.72, + 0.678, + 0.674, + 0.682, + 0.669, + 0.789, + 0.762, + 0.755, + 0.855, + 0.864, + 0.823, + 0.783, + 0.77, + 0.753, + 0.741, + 0.626, + 0.62, + 0.607, + 0.609, + 0.593, + 0.735, + 0.718, + 0.789, + 0.773, + 0.729, + 0.729, + 0.67, + 0.671, + 0.665, + 0.665, + 0.776, + 0.76, + 0.746, + 0.75, + 0.76, + 0.933, + 0.911, + 0.886, + 0.879, + 0.879, + 0.703, + 0.679, + 0.669, + 0.666, + 0.67, + 2.417, + 2.367, + 2.357, + 2.307, + 2.393, + 2.593, + 2.525, + 2.537, + 2.433, + 2.402 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "colgrep", + "index_ms": 4266.4, + "p50_ms": 122.98, + "p90_ms": 149.69, + "p95_ms": 156.68, + "p99_ms": 159.62, + "latencies_ms": [ + 139.566, + 117.528, + 117.928, + 118.893, + 125.379, + 117.623, + 114.503, + 120.576, + 117.532, + 113.903, + 115.237, + 116.32, + 156.485, + 146.1, + 136.252, + 134.135, + 160.353, + 148.938, + 139.656, + 129.317 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.14, + "p90_ms": 17.44, + "p95_ms": 20.16, + "p99_ms": 22.58, + "latencies_ms": [ + 17.142, + 16.954, + 16.647, + 16.945, + 16.883, + 21.99, + 23.437, + 20.146, + 17.227, + 18.693, + 16.032, + 15.788, + 17.801, + 15.089, + 16.413, + 17.405, + 15.113, + 14.787, + 15.725, + 15.789, + 16.15, + 16.525, + 14.939, + 14.723, + 14.693, + 16.359, + 14.511, + 15.832, + 16.309, + 16.069, + 14.639, + 14.929, + 14.145, + 14.137, + 16.02, + 15.074, + 14.03, + 12.994, + 14.679, + 15.083, + 13.684, + 15.213, + 20.379, + 15.663, + 15.779, + 15.561, + 14.458, + 15.174, + 13.384, + 12.362, + 13.446, + 12.495, + 11.522, + 13.242, + 12.963, + 11.518, + 12.948, + 13.174, + 13.131, + 13.557 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "semble", + "index_ms": 302.1, + "p50_ms": 0.61, + "p90_ms": 2.61, + "p95_ms": 3.45, + "p99_ms": 4.1, + "latencies_ms": [ + 0.975, + 0.574, + 0.534, + 0.484, + 0.768, + 0.895, + 0.573, + 0.47, + 0.455, + 0.467, + 0.426, + 0.395, + 0.558, + 0.547, + 0.471, + 0.474, + 0.425, + 0.971, + 0.81, + 0.931, + 0.763, + 0.556, + 0.463, + 0.466, + 0.435, + 0.428, + 0.394, + 0.578, + 0.761, + 0.474, + 0.69, + 0.515, + 0.491, + 0.492, + 0.452, + 0.653, + 0.503, + 0.428, + 0.421, + 0.411, + 0.435, + 0.402, + 0.41, + 0.415, + 0.602, + 0.619, + 0.51, + 0.496, + 0.482, + 0.529, + 0.956, + 0.781, + 0.674, + 0.636, + 0.64, + 0.78, + 0.773, + 0.703, + 0.749, + 0.978, + 0.969, + 0.76, + 0.916, + 0.792, + 0.75, + 0.571, + 0.548, + 0.54, + 0.86, + 0.614, + 0.829, + 0.736, + 0.742, + 1.076, + 1.037, + 4.188, + 3.634, + 3.254, + 3.304, + 3.538, + 2.728, + 2.2, + 2.436, + 2.302, + 2.167, + 4.12, + 3.461, + 3.374, + 3.423, + 3.492, + 0.459, + 0.583, + 0.441, + 0.395, + 0.388, + 0.374, + 0.329, + 0.321, + 0.318, + 0.325, + 0.774, + 0.667, + 0.661, + 0.677, + 0.679 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "bm25", + "index_ms": 19.7, + "p50_ms": 0.43, + "p90_ms": 2.09, + "p95_ms": 2.56, + "p99_ms": 3.12, + "latencies_ms": [ + 0.784, + 0.525, + 0.425, + 0.403, + 0.401, + 0.463, + 0.406, + 0.399, + 0.396, + 0.4, + 0.38, + 0.35, + 0.345, + 0.352, + 0.35, + 0.38, + 0.364, + 0.368, + 0.373, + 0.383, + 0.358, + 0.327, + 0.321, + 0.312, + 0.31, + 0.374, + 0.387, + 0.435, + 0.378, + 0.37, + 0.418, + 0.38, + 0.374, + 0.382, + 0.37, + 0.368, + 0.346, + 0.345, + 0.338, + 0.338, + 0.334, + 0.323, + 0.323, + 0.324, + 0.333, + 0.447, + 0.402, + 0.385, + 0.385, + 0.385, + 0.578, + 0.55, + 0.537, + 0.607, + 0.57, + 0.638, + 0.611, + 0.593, + 0.597, + 0.602, + 0.684, + 0.693, + 0.646, + 0.635, + 0.629, + 0.465, + 0.439, + 0.463, + 0.446, + 0.44, + 0.659, + 0.623, + 0.631, + 0.616, + 0.61, + 3.101, + 3.092, + 3.099, + 3.141, + 3.121, + 2.05, + 2.061, + 2.034, + 2.113, + 2.068, + 2.511, + 2.427, + 2.412, + 2.545, + 2.559, + 0.463, + 0.486, + 0.429, + 0.379, + 0.361, + 0.271, + 0.243, + 0.239, + 0.242, + 0.239, + 0.653, + 0.568, + 0.563, + 0.555, + 0.704 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "colgrep", + "index_ms": 2394.2, + "p50_ms": 112.08, + "p90_ms": 122.66, + "p95_ms": 125.48, + "p99_ms": 146.06, + "latencies_ms": [ + 151.205, + 111.255, + 109.941, + 109.401, + 110.037, + 109.345, + 122.66, + 116.407, + 115.891, + 120.353, + 112.105, + 108.931, + 111.086, + 125.479, + 112.263, + 111.106, + 111.84, + 110.048, + 112.258, + 113.013, + 112.077 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.06, + "p90_ms": 14.0, + "p95_ms": 14.23, + "p99_ms": 17.18, + "latencies_ms": [ + 14.214, + 13.9, + 13.55, + 11.764, + 13.065, + 12.912, + 13.986, + 13.955, + 13.587, + 12.792, + 13.276, + 13.057, + 13.244, + 13.226, + 13.103, + 13.464, + 12.966, + 11.904, + 11.618, + 12.346, + 13.483, + 13.368, + 12.702, + 11.834, + 11.632, + 12.357, + 11.849, + 11.897, + 13.147, + 12.474, + 12.51, + 12.121, + 12.752, + 12.99, + 12.827, + 14.279, + 12.609, + 13.13, + 12.257, + 13.116, + 14.073, + 14.231, + 13.83, + 13.741, + 13.998, + 12.915, + 11.682, + 13.227, + 11.315, + 12.74, + 11.302, + 12.662, + 13.493, + 12.921, + 13.213, + 13.177, + 21.919, + 13.174, + 13.12, + 12.857, + 12.841, + 12.618, + 14.256 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "semble", + "index_ms": 231.3, + "p50_ms": 0.63, + "p90_ms": 3.65, + "p95_ms": 4.97, + "p99_ms": 6.18, + "latencies_ms": [ + 2.128, + 1.791, + 3.488, + 0.645, + 0.587, + 0.734, + 0.647, + 0.629, + 0.625, + 0.619, + 0.474, + 0.439, + 0.442, + 0.458, + 0.441, + 0.545, + 0.489, + 0.477, + 0.476, + 0.473, + 3.641, + 2.992, + 3.001, + 3.081, + 3.246, + 6.363, + 5.585, + 6.158, + 5.578, + 5.359, + 4.487, + 3.737, + 3.699, + 3.634, + 3.785, + 0.713, + 0.612, + 0.584, + 0.696, + 0.655, + 0.603, + 0.553, + 0.565, + 0.552, + 0.526, + 0.623, + 0.597, + 0.596, + 0.604, + 0.714, + 0.589, + 0.536, + 0.539, + 0.606, + 0.635, + 0.766, + 0.688, + 0.626, + 0.614, + 0.606, + 0.605, + 0.565, + 0.565, + 0.552, + 0.55, + 0.547, + 0.521, + 0.51, + 0.514, + 0.525, + 0.675, + 0.611, + 0.585, + 0.576, + 0.581, + 0.715, + 0.654, + 0.637, + 0.647, + 0.958, + 3.453, + 3.01, + 2.918, + 2.897, + 2.897, + 3.197, + 2.805, + 2.819, + 2.806, + 2.878 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "bm25", + "index_ms": 15.8, + "p50_ms": 0.5, + "p90_ms": 3.16, + "p95_ms": 3.76, + "p99_ms": 4.22, + "latencies_ms": [ + 0.824, + 0.572, + 0.486, + 0.472, + 0.452, + 0.597, + 0.56, + 0.532, + 0.529, + 0.517, + 0.365, + 0.346, + 0.342, + 0.324, + 0.321, + 0.436, + 0.411, + 0.415, + 0.426, + 0.439, + 2.756, + 2.691, + 2.681, + 2.624, + 2.622, + 4.185, + 4.157, + 4.222, + 4.157, + 4.248, + 3.283, + 3.225, + 3.205, + 3.215, + 3.152, + 0.539, + 0.498, + 0.49, + 0.477, + 0.479, + 0.433, + 0.41, + 0.409, + 0.411, + 0.412, + 0.556, + 0.519, + 0.53, + 0.505, + 0.485, + 0.454, + 0.43, + 0.421, + 0.42, + 0.431, + 0.517, + 0.495, + 0.477, + 0.483, + 0.48, + 0.484, + 0.475, + 0.501, + 0.546, + 0.494, + 0.488, + 0.438, + 0.436, + 0.434, + 0.433, + 0.518, + 0.498, + 0.477, + 0.473, + 0.481, + 0.582, + 0.538, + 0.516, + 0.513, + 0.521, + 2.223, + 2.115, + 2.103, + 2.262, + 2.128, + 2.745, + 2.594, + 2.559, + 2.611, + 2.69 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "colgrep", + "index_ms": 2904.7, + "p50_ms": 112.74, + "p90_ms": 122.19, + "p95_ms": 126.52, + "p99_ms": 138.12, + "latencies_ms": [ + 141.023, + 111.288, + 111.306, + 111.573, + 113.376, + 113.491, + 111.053, + 114.051, + 120.5, + 111.053, + 121.428, + 117.625, + 123.961, + 114.854, + 109.797, + 112.108, + 111.867, + 109.906 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.46, + "p90_ms": 15.04, + "p95_ms": 15.87, + "p99_ms": 16.2, + "latencies_ms": [ + 14.789, + 13.641, + 13.237, + 14.065, + 12.075, + 13.956, + 13.559, + 13.942, + 13.0, + 13.411, + 13.769, + 13.199, + 14.207, + 12.076, + 11.787, + 13.877, + 11.839, + 11.47, + 12.513, + 11.281, + 12.486, + 12.864, + 15.006, + 14.91, + 13.006, + 11.149, + 11.996, + 13.272, + 11.846, + 14.011, + 14.214, + 13.763, + 13.657, + 12.712, + 11.723, + 11.543, + 12.396, + 12.964, + 11.749, + 11.689, + 12.812, + 14.057, + 13.722, + 16.326, + 15.057, + 15.995, + 15.796, + 14.94, + 15.766, + 14.565, + 16.085, + 13.625, + 13.369, + 13.501 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "semble", + "index_ms": 432.8, + "p50_ms": 1.06, + "p90_ms": 4.38, + "p95_ms": 5.05, + "p99_ms": 6.45, + "latencies_ms": [ + 1.512, + 1.116, + 0.992, + 1.111, + 1.141, + 1.079, + 1.215, + 1.029, + 1.189, + 1.043, + 7.536, + 6.271, + 6.286, + 6.285, + 6.441, + 0.93, + 0.749, + 0.709, + 0.687, + 0.686, + 0.675, + 0.636, + 0.716, + 0.685, + 0.671, + 0.757, + 0.698, + 0.689, + 0.671, + 0.673, + 4.984, + 4.576, + 4.376, + 4.433, + 4.687, + 1.259, + 1.292, + 1.165, + 1.029, + 1.014, + 0.831, + 0.786, + 0.776, + 0.767, + 0.772, + 0.839, + 0.796, + 0.81, + 0.922, + 1.01, + 0.946, + 0.807, + 0.803, + 0.777, + 0.78, + 3.512, + 3.397, + 3.399, + 3.533, + 3.661, + 4.866, + 4.267, + 4.272, + 4.334, + 4.294, + 4.356, + 3.807, + 3.736, + 3.709, + 3.738, + 0.844, + 0.766, + 0.755, + 0.751, + 0.778, + 0.957, + 0.87, + 0.853, + 0.844, + 0.841, + 0.679, + 0.641, + 0.634, + 0.63, + 0.63, + 3.454, + 3.527, + 3.438, + 3.353, + 3.388, + 3.533, + 3.515, + 3.409, + 3.411, + 3.421, + 3.656, + 3.591, + 3.561, + 3.573, + 3.606 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "bm25", + "index_ms": 32.9, + "p50_ms": 0.71, + "p90_ms": 3.34, + "p95_ms": 3.53, + "p99_ms": 4.73, + "latencies_ms": [ + 1.064, + 0.734, + 0.683, + 0.668, + 0.662, + 0.685, + 0.65, + 0.646, + 0.645, + 0.652, + 4.931, + 4.724, + 4.672, + 4.663, + 4.635, + 0.571, + 0.523, + 0.511, + 0.507, + 0.511, + 0.569, + 0.561, + 0.546, + 0.552, + 0.548, + 0.624, + 0.588, + 0.584, + 0.674, + 0.654, + 3.469, + 3.386, + 3.399, + 3.341, + 3.302, + 0.867, + 0.815, + 0.909, + 0.874, + 0.844, + 0.644, + 0.602, + 0.592, + 0.593, + 0.593, + 0.622, + 0.593, + 0.586, + 0.59, + 0.577, + 0.643, + 0.612, + 0.655, + 0.662, + 0.635, + 2.919, + 2.876, + 2.906, + 2.919, + 2.867, + 3.34, + 3.297, + 3.362, + 3.285, + 3.293, + 3.09, + 3.088, + 3.166, + 3.122, + 3.08, + 0.639, + 0.616, + 0.607, + 0.608, + 0.6, + 0.735, + 0.702, + 0.703, + 0.713, + 0.712, + 0.587, + 0.559, + 0.558, + 0.547, + 0.539, + 2.794, + 2.747, + 2.737, + 2.738, + 2.745, + 2.879, + 2.834, + 2.839, + 3.005, + 2.929, + 2.946, + 2.91, + 2.903, + 2.879, + 2.9 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "colgrep", + "index_ms": 6670.6, + "p50_ms": 122.0, + "p90_ms": 130.4, + "p95_ms": 132.53, + "p99_ms": 143.66, + "latencies_ms": [ + 146.439, + 120.878, + 118.29, + 119.267, + 130.246, + 123.088, + 123.948, + 122.381, + 131.793, + 121.014, + 120.122, + 125.845, + 124.269, + 121.611, + 123.189, + 121.147, + 120.979, + 121.057, + 125.985, + 121.454 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.83, + "p90_ms": 16.83, + "p95_ms": 17.29, + "p99_ms": 17.76, + "latencies_ms": [ + 16.861, + 17.555, + 17.65, + 17.281, + 16.801, + 15.395, + 16.364, + 15.447, + 16.075, + 16.158, + 15.068, + 16.783, + 14.72, + 14.022, + 15.52, + 14.752, + 15.851, + 15.21, + 14.008, + 16.83, + 16.087, + 17.267, + 17.916, + 16.147, + 15.082, + 14.78, + 15.215, + 14.642, + 15.277, + 12.917, + 14.212, + 14.137, + 13.676, + 14.703, + 13.561, + 13.604, + 14.208, + 15.783, + 14.524, + 14.008, + 13.929, + 15.158, + 14.367, + 14.927, + 13.375, + 14.454, + 12.863, + 13.875, + 14.822, + 12.366, + 12.864, + 14.732, + 15.246, + 12.25, + 12.414, + 14.845, + 15.239, + 14.915, + 13.283, + 13.96 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "semble", + "index_ms": 1007.3, + "p50_ms": 4.82, + "p90_ms": 7.23, + "p95_ms": 7.71, + "p99_ms": 7.88, + "latencies_ms": [ + 7.521, + 6.161, + 6.061, + 5.871, + 5.858, + 1.151, + 1.061, + 1.051, + 1.054, + 1.013, + 6.092, + 5.419, + 5.482, + 5.522, + 6.589, + 5.47, + 4.904, + 4.961, + 4.953, + 4.867, + 0.98, + 0.918, + 0.899, + 0.906, + 0.933, + 0.951, + 0.843, + 0.915, + 0.846, + 0.831, + 8.402, + 7.71, + 7.783, + 7.777, + 7.716, + 1.021, + 0.987, + 0.979, + 0.969, + 0.973, + 1.011, + 0.993, + 0.991, + 1.111, + 1.016, + 1.052, + 1.023, + 1.008, + 1.002, + 1.02, + 6.32, + 5.678, + 5.62, + 5.503, + 5.615, + 1.109, + 0.999, + 0.991, + 0.966, + 1.039, + 7.555, + 6.931, + 7.002, + 7.207, + 7.059, + 1.036, + 0.983, + 0.963, + 0.968, + 0.963, + 7.872, + 7.378, + 7.145, + 7.245, + 7.226, + 0.782, + 0.753, + 0.752, + 0.754, + 0.743, + 6.255, + 5.829, + 5.669, + 5.611, + 5.671, + 4.979, + 4.87, + 4.949, + 4.965, + 4.871, + 4.703, + 4.68, + 4.684, + 4.745, + 4.77, + 5.838, + 5.419, + 5.399, + 5.442, + 5.525 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "bm25", + "index_ms": 72.4, + "p50_ms": 4.05, + "p90_ms": 5.98, + "p95_ms": 6.09, + "p99_ms": 6.42, + "latencies_ms": [ + 5.121, + 4.699, + 4.709, + 4.793, + 4.861, + 1.021, + 0.917, + 0.9, + 0.906, + 0.902, + 4.913, + 4.787, + 4.647, + 4.636, + 4.613, + 4.378, + 4.423, + 4.253, + 4.31, + 4.303, + 0.802, + 0.746, + 0.741, + 0.751, + 0.736, + 0.659, + 0.662, + 0.687, + 0.657, + 0.637, + 5.967, + 5.986, + 5.925, + 5.899, + 5.938, + 0.925, + 0.868, + 0.849, + 0.857, + 0.845, + 0.833, + 0.813, + 0.801, + 0.802, + 0.843, + 0.846, + 0.83, + 0.822, + 0.802, + 0.795, + 4.601, + 4.703, + 4.678, + 4.705, + 4.588, + 0.885, + 0.845, + 0.86, + 0.856, + 0.821, + 6.03, + 6.047, + 6.077, + 5.988, + 5.982, + 0.75, + 0.703, + 0.696, + 0.696, + 0.688, + 6.423, + 6.377, + 6.315, + 6.487, + 6.3, + 0.632, + 0.593, + 0.586, + 0.583, + 0.626, + 4.639, + 4.584, + 4.645, + 4.572, + 4.646, + 4.024, + 3.986, + 3.981, + 3.973, + 4.048, + 4.087, + 4.072, + 4.069, + 4.042, + 4.083, + 5.216, + 5.176, + 5.215, + 5.343, + 5.263 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "colgrep", + "index_ms": 4386.3, + "p50_ms": 123.86, + "p90_ms": 133.23, + "p95_ms": 136.36, + "p99_ms": 163.9, + "latencies_ms": [ + 170.786, + 122.615, + 122.3, + 122.422, + 124.33, + 123.899, + 125.751, + 124.706, + 130.17, + 133.089, + 134.545, + 131.686, + 132.605, + 120.439, + 120.365, + 121.446, + 123.179, + 119.542, + 120.928, + 123.817 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.18, + "p90_ms": 17.33, + "p95_ms": 17.72, + "p99_ms": 18.3, + "latencies_ms": [ + 16.223, + 15.989, + 16.597, + 16.141, + 17.077, + 17.956, + 17.321, + 15.797, + 16.88, + 16.709, + 16.457, + 16.87, + 16.995, + 15.963, + 17.962, + 16.977, + 16.404, + 14.785, + 16.74, + 15.842, + 15.469, + 14.701, + 14.051, + 16.132, + 14.586, + 15.911, + 17.055, + 15.668, + 16.804, + 17.482, + 15.942, + 17.21, + 14.661, + 15.553, + 15.916, + 17.292, + 18.783, + 16.483, + 17.239, + 15.32, + 15.697, + 16.288, + 16.575, + 17.379, + 17.281, + 15.499, + 17.704, + 16.936, + 15.716, + 16.767, + 15.327, + 15.551, + 14.399, + 15.425, + 14.338, + 16.992, + 13.758, + 14.759, + 14.441, + 13.511 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "semble", + "index_ms": 464.1, + "p50_ms": 0.96, + "p90_ms": 4.74, + "p95_ms": 6.12, + "p99_ms": 7.23, + "latencies_ms": [ + 1.428, + 1.051, + 0.957, + 0.933, + 0.923, + 1.02, + 0.967, + 0.956, + 0.958, + 0.962, + 0.776, + 0.814, + 0.841, + 0.759, + 0.748, + 0.84, + 0.781, + 0.745, + 0.737, + 0.74, + 0.804, + 0.75, + 0.736, + 0.739, + 0.833, + 0.648, + 0.568, + 0.538, + 0.527, + 0.513, + 0.619, + 0.573, + 0.576, + 0.562, + 0.569, + 0.829, + 0.799, + 0.794, + 0.788, + 0.811, + 0.863, + 0.794, + 0.783, + 0.779, + 0.769, + 0.846, + 0.81, + 0.803, + 0.778, + 0.775, + 6.09, + 5.287, + 5.159, + 5.217, + 5.199, + 0.939, + 0.906, + 0.895, + 0.895, + 0.886, + 7.218, + 6.909, + 6.81, + 6.765, + 8.146, + 4.689, + 4.263, + 4.228, + 4.196, + 4.189, + 3.867, + 3.392, + 3.409, + 3.389, + 3.373, + 3.659, + 3.167, + 3.187, + 3.211, + 3.209, + 4.265, + 3.801, + 3.904, + 3.874, + 3.898, + 1.095, + 0.967, + 0.967, + 0.976, + 0.958, + 1.061, + 1.031, + 1.02, + 1.016, + 1.115, + 4.605, + 4.107, + 3.973, + 3.898, + 4.002 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "bm25", + "index_ms": 41.7, + "p50_ms": 0.85, + "p90_ms": 3.84, + "p95_ms": 4.88, + "p99_ms": 6.45, + "latencies_ms": [ + 1.247, + 0.896, + 0.852, + 0.834, + 0.872, + 0.927, + 0.842, + 0.824, + 0.824, + 0.823, + 0.683, + 0.655, + 0.662, + 0.658, + 0.684, + 0.694, + 0.653, + 0.642, + 0.694, + 0.688, + 0.701, + 0.668, + 0.67, + 0.673, + 0.678, + 0.47, + 0.434, + 0.429, + 0.437, + 0.43, + 0.521, + 0.484, + 0.487, + 0.515, + 0.493, + 0.747, + 0.771, + 0.744, + 0.712, + 0.698, + 0.721, + 0.685, + 0.694, + 0.689, + 0.677, + 0.648, + 0.61, + 0.604, + 0.599, + 0.601, + 4.796, + 4.72, + 4.732, + 4.685, + 4.663, + 0.797, + 0.77, + 0.753, + 0.753, + 0.756, + 6.485, + 6.45, + 6.393, + 6.38, + 6.392, + 3.746, + 3.709, + 3.716, + 3.667, + 3.737, + 3.119, + 3.061, + 2.979, + 3.044, + 3.039, + 2.976, + 2.893, + 3.04, + 2.95, + 2.9, + 3.313, + 3.212, + 3.18, + 3.186, + 3.176, + 0.956, + 0.86, + 0.869, + 0.879, + 0.875, + 0.936, + 0.912, + 0.958, + 0.932, + 0.913, + 3.306, + 3.306, + 3.302, + 3.293, + 3.288 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "colgrep", + "index_ms": 3964.2, + "p50_ms": 116.59, + "p90_ms": 126.12, + "p95_ms": 161.57, + "p99_ms": 419.76, + "latencies_ms": [ + 144.58, + 484.304, + 117.563, + 116.703, + 120.372, + 116.479, + 123.743, + 120.036, + 124.069, + 123.128, + 113.509, + 116.749, + 115.459, + 115.93, + 114.959, + 113.668, + 112.401, + 115.047, + 113.465, + 116.117 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.41, + "p90_ms": 15.29, + "p95_ms": 15.69, + "p99_ms": 16.88, + "latencies_ms": [ + 15.274, + 15.639, + 11.758, + 13.101, + 12.937, + 14.048, + 14.119, + 15.44, + 16.778, + 16.618, + 15.57, + 13.857, + 13.122, + 12.906, + 13.399, + 13.072, + 13.412, + 14.375, + 14.241, + 14.884, + 12.068, + 12.999, + 13.154, + 13.686, + 14.162, + 13.445, + 13.578, + 15.089, + 17.024, + 15.259, + 14.241, + 13.434, + 13.176, + 14.302, + 12.828, + 13.404, + 12.706, + 13.578, + 13.982, + 13.082, + 12.748, + 13.436, + 11.107, + 12.044, + 12.271, + 13.123, + 13.267, + 12.326, + 13.862, + 12.187, + 13.724, + 13.193, + 15.22, + 12.412, + 12.12, + 10.865, + 12.116, + 11.904, + 13.384, + 13.448 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "semble", + "index_ms": 306.1, + "p50_ms": 3.02, + "p90_ms": 3.92, + "p95_ms": 5.14, + "p99_ms": 5.24, + "latencies_ms": [ + 5.181, + 4.046, + 3.918, + 3.987, + 4.022, + 0.946, + 0.83, + 0.798, + 0.786, + 0.777, + 1.019, + 1.1, + 1.073, + 1.166, + 1.029, + 3.749, + 3.636, + 3.668, + 3.645, + 3.669, + 0.977, + 0.921, + 0.9, + 0.905, + 0.901, + 0.911, + 0.867, + 0.851, + 0.912, + 0.912, + 0.802, + 0.757, + 0.779, + 0.74, + 0.731, + 1.031, + 0.987, + 0.979, + 0.977, + 1.047, + 3.47, + 3.333, + 3.377, + 3.396, + 3.385, + 3.723, + 3.65, + 3.766, + 3.846, + 3.731, + 3.88, + 3.774, + 3.727, + 3.811, + 3.697, + 0.92, + 0.862, + 0.854, + 0.868, + 0.864, + 3.534, + 3.066, + 3.005, + 3.044, + 3.142, + 3.297, + 2.832, + 2.876, + 2.816, + 2.795, + 3.538, + 3.079, + 2.922, + 2.973, + 2.893, + 3.189, + 2.837, + 2.688, + 2.628, + 2.627, + 3.14, + 2.666, + 2.616, + 2.615, + 3.779, + 5.278, + 5.239, + 5.169, + 5.166, + 5.137, + 3.929, + 3.89, + 3.887, + 3.847, + 3.854, + 3.569, + 3.491, + 3.489, + 3.549, + 3.517 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "bm25", + "index_ms": 22.1, + "p50_ms": 2.25, + "p90_ms": 2.97, + "p95_ms": 3.16, + "p99_ms": 4.17, + "latencies_ms": [ + 3.108, + 2.974, + 2.96, + 2.971, + 2.991, + 0.642, + 0.615, + 0.61, + 0.605, + 0.607, + 0.708, + 0.688, + 0.761, + 0.726, + 0.692, + 2.802, + 2.814, + 2.813, + 2.811, + 2.787, + 0.676, + 0.643, + 0.64, + 0.632, + 0.635, + 0.752, + 0.74, + 0.685, + 0.67, + 0.669, + 0.634, + 0.616, + 0.613, + 0.604, + 0.608, + 0.836, + 0.816, + 0.783, + 0.775, + 0.772, + 2.974, + 2.84, + 2.842, + 2.943, + 2.93, + 2.776, + 2.721, + 2.843, + 2.77, + 2.714, + 2.945, + 2.98, + 2.888, + 2.856, + 3.001, + 0.823, + 0.787, + 0.777, + 0.768, + 0.767, + 2.511, + 2.531, + 2.505, + 2.434, + 2.483, + 2.188, + 2.128, + 2.108, + 2.075, + 2.094, + 2.232, + 2.183, + 2.18, + 2.167, + 2.172, + 2.289, + 2.26, + 2.284, + 2.267, + 2.3, + 2.154, + 2.147, + 2.139, + 2.177, + 2.178, + 4.185, + 4.16, + 4.113, + 4.137, + 4.172, + 2.938, + 2.927, + 2.916, + 2.848, + 2.962, + 2.91, + 2.903, + 2.841, + 2.78, + 2.822 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "colgrep", + "index_ms": 2900.9, + "p50_ms": 119.96, + "p90_ms": 122.2, + "p95_ms": 124.08, + "p99_ms": 143.27, + "latencies_ms": [ + 148.072, + 118.043, + 119.153, + 117.613, + 119.546, + 122.817, + 118.502, + 121.122, + 120.414, + 119.865, + 119.771, + 120.868, + 119.773, + 120.045, + 119.072, + 122.132, + 118.668, + 122.049, + 121.418, + 121.648 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.87, + "p90_ms": 16.43, + "p95_ms": 16.82, + "p99_ms": 17.1, + "latencies_ms": [ + 16.956, + 16.047, + 13.635, + 13.249, + 13.377, + 16.058, + 14.963, + 13.547, + 15.453, + 14.335, + 16.493, + 12.949, + 13.626, + 13.989, + 15.072, + 15.326, + 14.985, + 17.315, + 16.421, + 14.121, + 14.332, + 14.55, + 16.813, + 14.989, + 13.497, + 14.018, + 14.621, + 15.758, + 15.053, + 14.762, + 14.899, + 14.849, + 15.456, + 15.132, + 16.912, + 16.38, + 14.908, + 16.032, + 15.105, + 14.026, + 12.808, + 12.397, + 13.078, + 14.291, + 16.094, + 13.815, + 14.444, + 13.379, + 15.257, + 13.863, + 13.534, + 14.748, + 14.732, + 16.509, + 15.111, + 15.916, + 14.698, + 15.261, + 14.646, + 15.253 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "semble", + "index_ms": 1095.1, + "p50_ms": 0.86, + "p90_ms": 1.28, + "p95_ms": 2.07, + "p99_ms": 8.27, + "latencies_ms": [ + 1.75, + 1.312, + 1.274, + 1.286, + 1.262, + 9.274, + 8.206, + 8.263, + 8.136, + 8.121, + 0.82, + 0.696, + 0.682, + 0.674, + 0.664, + 0.967, + 0.897, + 0.909, + 0.976, + 0.993, + 0.983, + 0.895, + 0.894, + 0.9, + 0.89, + 1.163, + 1.056, + 1.067, + 1.024, + 1.019, + 0.725, + 0.67, + 0.662, + 0.65, + 0.662, + 0.706, + 0.88, + 0.831, + 0.703, + 0.669, + 0.939, + 0.87, + 0.923, + 0.954, + 0.855, + 1.427, + 1.332, + 1.235, + 1.163, + 1.145, + 0.85, + 0.789, + 0.784, + 0.789, + 0.817, + 0.842, + 0.789, + 0.789, + 0.791, + 0.825, + 0.956, + 0.907, + 0.975, + 0.957, + 0.901, + 0.729, + 0.683, + 0.66, + 0.664, + 0.649, + 0.788, + 0.724, + 0.9, + 0.898, + 0.787, + 0.839, + 0.828, + 0.755, + 0.736, + 0.73, + 0.714, + 0.708, + 0.677, + 0.682, + 0.722, + 1.184, + 1.041, + 1.031, + 1.053, + 1.035, + 0.894, + 0.825, + 0.818, + 0.812, + 0.79, + 0.86, + 0.784, + 0.763, + 0.769, + 0.962 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "bm25", + "index_ms": 78.0, + "p50_ms": 0.68, + "p90_ms": 1.02, + "p95_ms": 1.81, + "p99_ms": 7.44, + "latencies_ms": [ + 1.524, + 1.075, + 1.015, + 0.992, + 0.99, + 7.436, + 7.352, + 7.183, + 7.287, + 7.579, + 0.658, + 0.543, + 0.542, + 0.517, + 0.511, + 0.811, + 0.739, + 0.733, + 0.723, + 0.721, + 0.72, + 0.75, + 0.776, + 0.727, + 0.692, + 0.86, + 0.815, + 0.795, + 0.792, + 0.795, + 0.609, + 0.564, + 0.562, + 0.55, + 0.54, + 0.563, + 0.54, + 0.529, + 0.537, + 0.527, + 0.757, + 0.722, + 0.722, + 0.713, + 0.713, + 0.948, + 0.896, + 0.886, + 0.892, + 0.927, + 0.656, + 0.626, + 0.659, + 0.601, + 0.567, + 0.597, + 0.573, + 0.582, + 0.561, + 0.556, + 0.765, + 0.742, + 0.73, + 0.732, + 0.73, + 0.607, + 0.552, + 0.504, + 0.487, + 0.482, + 0.682, + 0.649, + 0.643, + 0.636, + 0.639, + 0.759, + 0.675, + 0.678, + 0.663, + 0.647, + 0.547, + 0.55, + 0.494, + 0.491, + 0.495, + 1.063, + 0.964, + 1.145, + 1.074, + 1.015, + 0.754, + 0.665, + 0.648, + 0.644, + 0.693, + 0.686, + 0.617, + 0.606, + 0.603, + 0.601 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "colgrep", + "index_ms": 2895.3, + "p50_ms": 115.94, + "p90_ms": 121.12, + "p95_ms": 124.8, + "p99_ms": 147.93, + "latencies_ms": [ + 153.709, + 117.919, + 114.798, + 115.74, + 116.142, + 123.274, + 120.879, + 116.171, + 117.405, + 116.233, + 114.42, + 115.493, + 118.269, + 114.297, + 117.271, + 114.364, + 114.123, + 114.315, + 115.476, + 114.455 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.1, + "p90_ms": 14.45, + "p95_ms": 14.78, + "p99_ms": 15.92, + "latencies_ms": [ + 17.203, + 13.248, + 12.689, + 14.315, + 14.325, + 14.155, + 13.228, + 13.006, + 12.944, + 13.725, + 14.399, + 14.095, + 13.164, + 15.035, + 12.611, + 14.192, + 13.495, + 12.928, + 13.248, + 11.406, + 12.961, + 13.268, + 14.772, + 14.19, + 13.312, + 14.703, + 12.552, + 13.569, + 14.214, + 13.038, + 12.254, + 12.405, + 12.372, + 14.433, + 13.113, + 14.123, + 12.525, + 12.04, + 12.401, + 12.004, + 14.247, + 12.19, + 12.262, + 13.096, + 13.874, + 12.298, + 14.216, + 12.225, + 11.2, + 12.111, + 12.063, + 12.278, + 12.986, + 14.615, + 14.125, + 12.645, + 12.0, + 12.031, + 12.214, + 14.907 + ] + } + ] +} diff --git a/benchmarks/results/speed-906319556a46.json b/benchmarks/results/speed-906319556a46.json new file mode 100644 index 000000000..53b35b157 --- /dev/null +++ b/benchmarks/results/speed-906319556a46.json @@ -0,0 +1,6259 @@ +{ + "repos": [ + "nvm", + "libuv", + "nlohmann-json", + "messagepack-csharp", + "phoenix", + "gin", + "aeson", + "gson", + "axios", + "ktor", + "telescope.nvim", + "monolog", + "flask", + "rack", + "axum", + "http4s", + "alamofire", + "trpc", + "zls" + ], + "summary": { + "semble": { + "avg_index_ms": 586.4, + "avg_p50_ms": 0.97, + "avg_p90_ms": 5.34, + "avg_p95_ms": 6.68, + "avg_p99_ms": 9.6 + }, + "bm25": { + "avg_index_ms": 586.4, + "avg_p50_ms": 0.77, + "avg_p90_ms": 4.53, + "avg_p95_ms": 5.58, + "avg_p99_ms": 8.91 + }, + "colgrep": { + "avg_index_ms": 5168.3, + "avg_p50_ms": 120.25, + "avg_p90_ms": 143.44, + "avg_p95_ms": 157.6, + "avg_p99_ms": 177.96 + }, + "ripgrep": { + "avg_index_ms": null, + "avg_p50_ms": 14.55, + "avg_p90_ms": 16.89, + "avg_p95_ms": 17.45, + "avg_p99_ms": 18.81 + } + }, + "results": [ + { + "repo": "nvm", + "language": "bash", + "tool": "semble", + "index_ms": 206.3, + "p50_ms": 0.6, + "p90_ms": 0.68, + "p95_ms": 0.73, + "p99_ms": 0.91, + "latencies_ms": [ + 2.363, + 0.733, + 0.818, + 0.681, + 0.618, + 0.604, + 0.566, + 0.553, + 0.569, + 0.559, + 0.718, + 0.636, + 0.642, + 0.636, + 0.675, + 0.708, + 0.628, + 0.618, + 0.64, + 0.62, + 0.633, + 0.589, + 0.598, + 0.579, + 0.577, + 0.52, + 0.518, + 0.532, + 0.52, + 0.497, + 0.598, + 0.567, + 0.564, + 0.565, + 0.556, + 0.623, + 0.61, + 0.592, + 0.604, + 0.595, + 0.64, + 0.623, + 0.611, + 0.604, + 0.614, + 0.568, + 0.54, + 0.539, + 0.5, + 0.497, + 0.666, + 0.632, + 0.636, + 0.763, + 0.73, + 0.629, + 0.595, + 0.604, + 0.577, + 0.576, + 0.621, + 0.626, + 0.605, + 0.589, + 0.591, + 0.658, + 0.63, + 0.586, + 0.606, + 0.6, + 0.595, + 0.598, + 0.566, + 0.566, + 0.564, + 0.636, + 0.621, + 0.602, + 0.596, + 0.603, + 0.521, + 0.494, + 0.481, + 0.491, + 0.494, + 0.609, + 0.584, + 0.588, + 0.58, + 0.587, + 0.563, + 0.551, + 0.562, + 0.661, + 0.652, + 0.897, + 0.626, + 0.565, + 0.571, + 0.865 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "bm25", + "index_ms": 206.3, + "p50_ms": 0.5, + "p90_ms": 0.56, + "p95_ms": 0.62, + "p99_ms": 0.69, + "latencies_ms": [ + 0.695, + 0.58, + 0.531, + 0.5, + 0.504, + 0.552, + 0.504, + 0.488, + 0.494, + 0.509, + 0.549, + 0.526, + 0.529, + 0.539, + 0.533, + 0.55, + 0.557, + 0.53, + 0.531, + 0.524, + 0.488, + 0.464, + 0.461, + 0.463, + 0.463, + 0.474, + 0.45, + 0.443, + 0.449, + 0.451, + 0.517, + 0.64, + 0.618, + 0.539, + 0.498, + 0.549, + 0.537, + 0.505, + 0.492, + 0.491, + 0.522, + 0.524, + 0.496, + 0.498, + 0.5, + 0.478, + 0.463, + 0.461, + 0.449, + 0.456, + 0.555, + 0.537, + 0.558, + 0.539, + 0.62, + 0.527, + 0.481, + 0.474, + 0.487, + 0.479, + 0.544, + 0.535, + 0.527, + 0.524, + 0.538, + 0.485, + 0.462, + 0.47, + 0.46, + 0.454, + 0.497, + 0.484, + 0.479, + 0.484, + 0.546, + 0.595, + 0.521, + 0.587, + 0.686, + 0.673, + 0.531, + 0.438, + 0.416, + 0.421, + 0.411, + 0.523, + 0.601, + 0.55, + 0.503, + 0.52, + 0.495, + 0.473, + 0.48, + 0.495, + 0.494, + 0.536, + 0.495, + 0.484, + 0.47, + 0.489 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "colgrep", + "index_ms": 3492.5, + "p50_ms": 119.02, + "p90_ms": 142.32, + "p95_ms": 155.95, + "p99_ms": 171.86, + "latencies_ms": [ + 175.833, + 132.9, + 127.21, + 140.926, + 139.46, + 139.456, + 125.451, + 154.905, + 130.612, + 117.407, + 118.401, + 115.411, + 116.189, + 116.987, + 115.65, + 115.95, + 119.595, + 116.344, + 115.394, + 118.45 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.81, + "p90_ms": 18.4, + "p95_ms": 19.39, + "p99_ms": 34.77, + "latencies_ms": [ + 25.557, + 15.828, + 15.562, + 15.922, + 14.745, + 14.848, + 16.454, + 16.611, + 15.127, + 15.197, + 16.74, + 16.32, + 15.827, + 17.047, + 16.647, + 16.911, + 15.863, + 18.854, + 17.684, + 19.423, + 17.104, + 16.133, + 16.097, + 16.747, + 16.259, + 17.126, + 15.846, + 17.191, + 16.003, + 17.52, + 17.502, + 16.128, + 18.348, + 17.216, + 19.387, + 18.094, + 17.836, + 16.123, + 16.589, + 15.713, + 16.828, + 48.035, + 17.391, + 17.451, + 16.921, + 16.391, + 17.154, + 16.513, + 16.569, + 17.403, + 17.348, + 19.197, + 17.379, + 18.286, + 16.797, + 18.304, + 17.459, + 16.938, + 15.897, + 16.257 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "semble", + "index_ms": 1186.6, + "p50_ms": 0.89, + "p90_ms": 1.35, + "p95_ms": 1.61, + "p99_ms": 1.93, + "latencies_ms": [ + 1.786, + 1.347, + 1.266, + 1.239, + 1.221, + 0.759, + 0.698, + 0.696, + 0.68, + 0.705, + 1.018, + 0.89, + 0.913, + 0.928, + 1.038, + 2.16, + 1.866, + 1.673, + 1.609, + 1.925, + 1.11, + 0.904, + 0.847, + 0.829, + 0.819, + 0.849, + 0.741, + 0.723, + 0.737, + 0.708, + 0.743, + 0.683, + 0.678, + 0.68, + 0.68, + 0.937, + 0.883, + 0.87, + 0.902, + 0.874, + 0.738, + 0.619, + 0.733, + 0.753, + 0.68, + 0.811, + 0.725, + 0.719, + 0.809, + 0.961, + 1.275, + 1.193, + 1.158, + 1.1, + 1.049, + 1.187, + 1.14, + 1.14, + 1.116, + 1.034, + 1.264, + 1.183, + 1.16, + 1.121, + 1.02, + 0.87, + 0.83, + 0.864, + 0.89, + 0.845, + 0.976, + 0.91, + 0.888, + 0.892, + 0.969, + 0.683, + 0.635, + 0.61, + 0.605, + 0.589, + 1.36, + 1.242, + 1.433, + 1.431, + 1.36, + 0.976, + 0.867, + 0.852, + 0.916, + 0.881, + 0.888, + 0.81, + 0.865, + 0.809, + 0.859, + 1.047, + 0.92, + 0.935, + 0.927, + 0.887 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "bm25", + "index_ms": 1186.6, + "p50_ms": 0.84, + "p90_ms": 1.28, + "p95_ms": 1.43, + "p99_ms": 1.52, + "latencies_ms": [ + 1.43, + 1.276, + 1.327, + 1.283, + 1.514, + 0.899, + 0.882, + 0.747, + 0.678, + 0.664, + 1.01, + 0.882, + 0.849, + 0.897, + 0.899, + 1.637, + 1.514, + 1.437, + 1.42, + 1.424, + 0.984, + 0.891, + 0.851, + 0.804, + 0.829, + 0.796, + 0.646, + 0.619, + 0.611, + 0.61, + 0.625, + 0.582, + 0.618, + 0.657, + 0.697, + 0.893, + 0.991, + 0.967, + 0.885, + 0.828, + 0.63, + 0.567, + 0.552, + 0.553, + 0.556, + 0.747, + 0.69, + 0.674, + 0.666, + 0.685, + 0.97, + 0.872, + 0.897, + 0.9, + 0.883, + 1.006, + 0.938, + 0.915, + 0.903, + 0.923, + 1.105, + 0.974, + 0.943, + 1.43, + 1.14, + 0.798, + 0.831, + 0.831, + 0.755, + 0.831, + 0.945, + 0.853, + 0.812, + 0.759, + 0.734, + 0.642, + 0.711, + 0.751, + 0.641, + 0.556, + 1.232, + 1.054, + 1.011, + 1.273, + 1.069, + 0.8, + 0.922, + 0.824, + 0.7, + 0.619, + 0.602, + 0.541, + 0.542, + 0.55, + 0.55, + 1.053, + 0.98, + 0.894, + 0.84, + 0.803 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "colgrep", + "index_ms": 10736.3, + "p50_ms": 136.94, + "p90_ms": 148.63, + "p95_ms": 150.01, + "p99_ms": 169.98, + "latencies_ms": [ + 174.978, + 139.055, + 148.626, + 141.392, + 143.957, + 136.972, + 134.287, + 132.107, + 133.649, + 133.535, + 135.589, + 136.913, + 133.802, + 134.991, + 138.101, + 130.921, + 148.694, + 133.255, + 144.059, + 138.674 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.09, + "p90_ms": 17.12, + "p95_ms": 17.37, + "p99_ms": 18.55, + "latencies_ms": [ + 16.891, + 16.898, + 16.162, + 14.988, + 15.638, + 14.691, + 16.01, + 17.015, + 16.138, + 19.56, + 15.804, + 16.163, + 15.724, + 14.842, + 14.736, + 13.781, + 13.067, + 15.263, + 14.975, + 13.818, + 15.472, + 15.81, + 16.504, + 15.179, + 14.737, + 15.341, + 16.108, + 14.763, + 16.86, + 15.707, + 16.074, + 16.12, + 16.865, + 15.466, + 16.871, + 17.854, + 17.345, + 16.431, + 16.593, + 16.594, + 16.907, + 16.155, + 15.788, + 16.806, + 15.827, + 17.321, + 15.972, + 14.78, + 15.182, + 17.107, + 16.363, + 15.795, + 17.237, + 17.003, + 15.455, + 16.151, + 15.936, + 17.806, + 16.123, + 16.431 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "semble", + "index_ms": 789.2, + "p50_ms": 1.09, + "p90_ms": 5.1, + "p95_ms": 5.22, + "p99_ms": 5.67, + "latencies_ms": [ + 1.888, + 1.357, + 1.282, + 1.244, + 1.25, + 0.858, + 0.759, + 0.745, + 0.746, + 0.743, + 0.828, + 0.813, + 0.808, + 0.831, + 0.843, + 1.068, + 1.018, + 0.992, + 0.985, + 1.006, + 1.295, + 1.377, + 1.338, + 1.289, + 1.218, + 1.049, + 1.011, + 1.008, + 1.012, + 1.08, + 1.2, + 1.136, + 1.098, + 1.081, + 1.129, + 1.293, + 1.278, + 1.269, + 1.414, + 1.333, + 1.084, + 1.052, + 1.107, + 1.088, + 1.011, + 1.037, + 0.967, + 0.945, + 1.031, + 1.119, + 1.1, + 1.027, + 1.001, + 0.976, + 0.95, + 0.915, + 0.849, + 0.881, + 0.886, + 0.848, + 1.731, + 1.733, + 1.667, + 1.661, + 1.717, + 6.112, + 5.251, + 5.199, + 5.223, + 5.143, + 1.111, + 0.966, + 0.949, + 0.951, + 0.988, + 0.849, + 0.806, + 0.816, + 0.843, + 0.871, + 1.071, + 0.985, + 0.976, + 0.995, + 1.155, + 5.218, + 4.86, + 4.758, + 4.611, + 4.529, + 5.634, + 5.197, + 5.092, + 5.138, + 5.088, + 4.945, + 4.523, + 4.476, + 5.663, + 4.508 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "bm25", + "index_ms": 789.2, + "p50_ms": 0.94, + "p90_ms": 4.04, + "p95_ms": 4.35, + "p99_ms": 4.57, + "latencies_ms": [ + 1.232, + 0.988, + 0.973, + 0.95, + 0.926, + 0.712, + 0.714, + 0.658, + 0.641, + 0.632, + 0.707, + 0.675, + 0.674, + 0.681, + 0.691, + 0.97, + 0.922, + 0.891, + 0.927, + 0.926, + 1.091, + 1.035, + 1.018, + 1.036, + 1.05, + 0.874, + 0.848, + 0.82, + 0.806, + 0.801, + 0.986, + 0.967, + 0.979, + 1.028, + 1.05, + 1.206, + 1.153, + 1.114, + 1.091, + 1.06, + 0.844, + 0.826, + 0.82, + 0.844, + 0.872, + 1.008, + 0.882, + 0.824, + 0.801, + 0.808, + 0.87, + 0.867, + 0.853, + 0.831, + 0.836, + 0.892, + 0.879, + 0.791, + 0.748, + 0.744, + 1.545, + 1.567, + 2.033, + 2.187, + 1.871, + 4.828, + 4.572, + 4.379, + 4.243, + 4.293, + 1.044, + 0.926, + 0.876, + 0.88, + 0.9, + 0.834, + 0.832, + 0.822, + 0.718, + 0.906, + 1.148, + 0.993, + 0.968, + 0.921, + 0.88, + 4.012, + 3.973, + 3.925, + 3.982, + 3.96, + 4.319, + 4.354, + 4.354, + 4.496, + 4.345, + 3.415, + 3.577, + 3.452, + 3.417, + 3.48 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "colgrep", + "index_ms": 4965.3, + "p50_ms": 118.97, + "p90_ms": 129.3, + "p95_ms": 133.06, + "p99_ms": 142.45, + "latencies_ms": [ + 144.79, + 125.68, + 128.953, + 132.447, + 128.364, + 119.117, + 121.722, + 115.857, + 117.357, + 118.817, + 117.778, + 116.889, + 117.704, + 119.615, + 120.341, + 120.484, + 117.598, + 117.772, + 117.949, + 117.121 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 11.87, + "p90_ms": 13.65, + "p95_ms": 14.15, + "p99_ms": 15.76, + "latencies_ms": [ + 13.635, + 11.878, + 9.733, + 13.99, + 15.217, + 10.821, + 10.486, + 13.522, + 12.125, + 12.137, + 12.648, + 11.027, + 10.851, + 10.997, + 11.296, + 11.065, + 9.968, + 13.757, + 13.086, + 12.097, + 11.26, + 11.445, + 11.835, + 11.597, + 12.135, + 12.956, + 13.62, + 11.692, + 12.289, + 11.172, + 11.389, + 10.482, + 11.735, + 13.178, + 11.536, + 12.184, + 9.628, + 11.43, + 10.9, + 11.67, + 12.341, + 12.313, + 14.793, + 16.541, + 11.855, + 12.699, + 9.955, + 11.963, + 9.5, + 13.483, + 12.949, + 13.555, + 11.963, + 11.644, + 13.258, + 11.037, + 14.113, + 11.978, + 11.134, + 10.46 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "semble", + "index_ms": 1001.2, + "p50_ms": 4.92, + "p90_ms": 6.81, + "p95_ms": 6.87, + "p99_ms": 7.13, + "latencies_ms": [ + 7.513, + 6.831, + 6.829, + 6.865, + 6.796, + 1.361, + 1.212, + 1.168, + 1.412, + 1.591, + 1.314, + 1.181, + 1.176, + 1.173, + 1.216, + 1.49, + 1.475, + 1.415, + 1.368, + 1.369, + 1.109, + 0.915, + 0.85, + 0.815, + 0.83, + 1.14, + 1.076, + 1.061, + 1.042, + 1.049, + 1.252, + 1.272, + 1.245, + 1.179, + 1.201, + 1.016, + 0.985, + 1.011, + 1.014, + 0.934, + 6.491, + 6.366, + 6.525, + 6.487, + 6.229, + 1.067, + 1.041, + 1.029, + 0.985, + 0.957, + 6.421, + 6.472, + 6.658, + 6.365, + 6.301, + 6.977, + 6.379, + 6.334, + 6.406, + 6.441, + 7.127, + 6.84, + 6.76, + 6.811, + 7.128, + 5.992, + 5.294, + 5.288, + 5.297, + 5.228, + 5.333, + 4.857, + 4.826, + 4.984, + 4.816, + 6.794, + 6.399, + 6.229, + 6.211, + 6.35, + 5.478, + 4.944, + 4.912, + 4.794, + 4.931, + 6.969, + 6.725, + 6.639, + 6.785, + 6.867, + 1.208, + 1.1, + 1.083, + 1.08, + 1.033, + 6.192, + 5.804, + 5.96, + 6.102, + 6.324 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "bm25", + "index_ms": 1001.2, + "p50_ms": 4.0, + "p90_ms": 5.62, + "p95_ms": 6.29, + "p99_ms": 6.61, + "latencies_ms": [ + 5.744, + 5.441, + 5.489, + 5.307, + 5.169, + 0.899, + 0.826, + 0.808, + 0.807, + 0.822, + 0.947, + 0.96, + 0.876, + 0.846, + 0.825, + 1.066, + 1.023, + 0.988, + 0.973, + 0.986, + 0.635, + 0.569, + 0.558, + 0.551, + 0.547, + 0.784, + 0.748, + 0.757, + 0.757, + 0.737, + 0.989, + 0.927, + 0.911, + 0.909, + 0.949, + 0.744, + 0.671, + 0.668, + 0.671, + 0.652, + 5.324, + 5.325, + 5.264, + 6.702, + 5.653, + 1.2, + 1.026, + 0.873, + 0.795, + 0.773, + 5.539, + 5.615, + 5.57, + 5.539, + 5.472, + 5.645, + 5.545, + 5.796, + 5.586, + 5.601, + 6.286, + 6.446, + 6.331, + 6.548, + 6.613, + 5.244, + 5.336, + 5.291, + 5.001, + 5.038, + 4.241, + 4.263, + 4.005, + 3.943, + 4.072, + 5.328, + 5.464, + 5.43, + 5.372, + 5.335, + 3.999, + 3.995, + 4.122, + 3.908, + 3.948, + 5.496, + 5.41, + 5.383, + 5.424, + 5.446, + 0.871, + 0.9, + 0.862, + 0.823, + 0.797, + 4.601, + 4.564, + 4.632, + 4.575, + 4.504 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "colgrep", + "index_ms": 4545.8, + "p50_ms": 121.72, + "p90_ms": 131.62, + "p95_ms": 132.96, + "p99_ms": 147.68, + "latencies_ms": [ + 151.363, + 118.613, + 120.988, + 119.553, + 127.612, + 120.045, + 131.577, + 128.89, + 131.988, + 123.629, + 129.092, + 122.076, + 120.827, + 124.028, + 118.663, + 121.358, + 117.193, + 117.042, + 122.416, + 117.1 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.94, + "p90_ms": 17.65, + "p95_ms": 18.21, + "p99_ms": 18.71, + "latencies_ms": [ + 18.193, + 16.051, + 16.55, + 16.13, + 17.653, + 16.133, + 16.155, + 18.602, + 14.609, + 15.183, + 14.907, + 15.92, + 14.498, + 14.482, + 15.126, + 15.934, + 16.492, + 16.31, + 16.534, + 13.865, + 14.869, + 16.653, + 14.205, + 14.53, + 17.779, + 15.645, + 18.726, + 15.986, + 15.235, + 13.866, + 16.367, + 16.436, + 15.938, + 16.126, + 14.524, + 16.221, + 14.32, + 15.514, + 16.54, + 15.75, + 15.408, + 15.706, + 15.399, + 14.478, + 17.448, + 17.504, + 13.992, + 15.947, + 17.659, + 16.672, + 14.86, + 15.873, + 16.909, + 16.455, + 14.823, + 17.625, + 18.704, + 15.393, + 16.08, + 14.528 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "semble", + "index_ms": 528.9, + "p50_ms": 0.93, + "p90_ms": 5.06, + "p95_ms": 6.11, + "p99_ms": 7.0, + "latencies_ms": [ + 1.52, + 1.111, + 1.076, + 1.043, + 1.115, + 0.886, + 0.789, + 0.772, + 0.755, + 0.741, + 0.753, + 0.688, + 0.68, + 0.682, + 0.689, + 0.723, + 0.693, + 0.696, + 0.696, + 0.677, + 0.778, + 0.724, + 0.717, + 0.721, + 0.711, + 5.866, + 5.049, + 5.172, + 5.074, + 5.084, + 0.716, + 0.647, + 0.607, + 0.574, + 0.565, + 0.749, + 0.703, + 0.712, + 0.694, + 0.682, + 1.072, + 1.001, + 0.981, + 0.99, + 1.001, + 0.846, + 0.802, + 0.737, + 0.708, + 0.709, + 0.944, + 0.939, + 0.917, + 0.95, + 1.102, + 0.804, + 0.954, + 1.094, + 0.798, + 0.711, + 0.883, + 0.902, + 0.764, + 0.724, + 0.743, + 0.87, + 0.935, + 0.956, + 0.932, + 0.944, + 1.048, + 0.898, + 0.828, + 0.969, + 1.069, + 1.321, + 1.156, + 1.114, + 1.122, + 1.096, + 5.038, + 4.586, + 4.531, + 4.488, + 4.404, + 7.418, + 6.978, + 6.692, + 6.699, + 6.888, + 5.097, + 4.432, + 4.501, + 4.485, + 4.383 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "bm25", + "index_ms": 528.9, + "p50_ms": 0.64, + "p90_ms": 4.09, + "p95_ms": 4.5, + "p99_ms": 5.35, + "latencies_ms": [ + 1.121, + 1.201, + 1.09, + 1.156, + 0.998, + 0.594, + 0.56, + 0.531, + 0.541, + 0.895, + 0.705, + 0.605, + 0.571, + 0.566, + 0.597, + 0.596, + 0.547, + 0.53, + 0.526, + 0.529, + 0.641, + 0.588, + 0.58, + 0.574, + 0.574, + 4.258, + 4.25, + 4.18, + 4.218, + 4.174, + 0.601, + 0.522, + 0.492, + 0.484, + 0.488, + 0.66, + 0.593, + 0.597, + 0.585, + 0.591, + 0.781, + 0.783, + 0.756, + 0.771, + 0.73, + 0.6, + 0.563, + 0.555, + 0.564, + 0.558, + 0.808, + 0.771, + 0.765, + 0.752, + 0.767, + 0.577, + 0.532, + 0.535, + 0.534, + 0.538, + 0.636, + 0.598, + 0.618, + 0.585, + 0.594, + 0.63, + 0.596, + 0.597, + 0.603, + 0.601, + 0.78, + 0.7, + 0.668, + 0.637, + 0.637, + 0.946, + 0.868, + 0.85, + 0.84, + 0.841, + 3.89, + 3.952, + 3.843, + 3.823, + 3.798, + 5.13, + 5.123, + 5.056, + 5.415, + 5.346, + 3.935, + 3.831, + 3.671, + 3.707, + 3.748 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "colgrep", + "index_ms": 17081.4, + "p50_ms": 160.58, + "p90_ms": 186.95, + "p95_ms": 203.23, + "p99_ms": 273.51, + "latencies_ms": [ + 291.076, + 185.322, + 170.91, + 172.878, + 193.471, + 185.068, + 159.144, + 164.334, + 153.677, + 155.646, + 158.082, + 166.332, + 150.996, + 160.037, + 159.851, + 160.584, + 157.511, + 162.591, + 156.537 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 12.77, + "p90_ms": 14.29, + "p95_ms": 14.65, + "p99_ms": 18.27, + "latencies_ms": [ + 21.74, + 13.982, + 14.978, + 13.844, + 11.824, + 12.606, + 12.813, + 12.728, + 12.772, + 13.272, + 12.841, + 13.117, + 14.099, + 13.501, + 10.935, + 14.277, + 12.822, + 11.72, + 14.314, + 13.712, + 12.78, + 11.61, + 13.24, + 14.12, + 13.58, + 13.245, + 12.498, + 12.254, + 12.408, + 12.348, + 13.926, + 10.959, + 13.102, + 14.246, + 12.051, + 12.433, + 13.512, + 12.25, + 12.222, + 12.341, + 11.11, + 12.671, + 14.35, + 15.538, + 13.227, + 12.468, + 12.598, + 11.704, + 12.283, + 10.79, + 12.321, + 13.856, + 12.401, + 12.613, + 14.566, + 12.38, + 11.351 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "semble", + "index_ms": 540.9, + "p50_ms": 0.75, + "p90_ms": 3.51, + "p95_ms": 4.58, + "p99_ms": 4.92, + "latencies_ms": [ + 3.224, + 0.729, + 0.638, + 0.609, + 0.596, + 0.529, + 0.464, + 0.483, + 0.479, + 0.461, + 0.756, + 0.698, + 0.691, + 0.688, + 0.687, + 0.796, + 0.83, + 0.862, + 0.8, + 0.77, + 0.786, + 0.74, + 0.723, + 0.728, + 0.74, + 4.922, + 4.199, + 4.033, + 4.166, + 4.117, + 0.662, + 0.617, + 0.609, + 0.703, + 0.686, + 0.592, + 0.571, + 0.545, + 0.545, + 0.547, + 0.483, + 0.452, + 0.445, + 0.448, + 0.465, + 0.725, + 0.695, + 0.685, + 0.687, + 0.682, + 0.679, + 0.65, + 0.605, + 0.576, + 0.561, + 0.873, + 0.84, + 0.827, + 0.819, + 0.82, + 0.813, + 0.783, + 0.779, + 0.831, + 0.851, + 0.83, + 0.788, + 0.781, + 0.779, + 0.781, + 0.771, + 0.735, + 0.728, + 0.734, + 0.727, + 0.913, + 0.994, + 0.995, + 0.92, + 0.893, + 3.455, + 2.994, + 3.041, + 3.035, + 3.007, + 5.045, + 4.589, + 4.598, + 4.582, + 4.648, + 3.236, + 3.217, + 3.193, + 3.16, + 3.127, + 0.756, + 0.712, + 0.746, + 0.738, + 0.711 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "bm25", + "index_ms": 540.9, + "p50_ms": 0.61, + "p90_ms": 2.75, + "p95_ms": 3.34, + "p99_ms": 4.21, + "latencies_ms": [ + 0.539, + 0.468, + 0.459, + 0.46, + 0.457, + 0.455, + 0.429, + 0.411, + 0.404, + 0.393, + 0.565, + 0.541, + 0.536, + 0.535, + 0.619, + 0.654, + 0.635, + 0.611, + 0.601, + 0.623, + 0.664, + 0.626, + 0.584, + 0.58, + 0.572, + 3.296, + 3.235, + 3.203, + 3.174, + 3.278, + 0.531, + 0.498, + 0.494, + 0.486, + 0.487, + 0.444, + 0.43, + 0.421, + 0.422, + 0.417, + 0.418, + 0.415, + 0.409, + 0.414, + 0.41, + 0.618, + 0.596, + 0.542, + 0.529, + 0.524, + 0.533, + 0.515, + 0.515, + 0.517, + 0.522, + 0.679, + 0.653, + 0.627, + 0.626, + 0.629, + 0.637, + 0.629, + 0.653, + 0.658, + 0.627, + 0.659, + 0.638, + 0.637, + 0.651, + 0.651, + 0.639, + 0.61, + 0.599, + 0.593, + 0.595, + 0.735, + 0.722, + 0.717, + 0.771, + 0.728, + 2.698, + 2.644, + 2.641, + 2.695, + 2.681, + 4.209, + 4.274, + 4.203, + 4.205, + 4.186, + 2.6, + 2.642, + 2.57, + 2.564, + 2.562, + 0.592, + 0.549, + 0.589, + 0.586, + 0.546 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "colgrep", + "index_ms": 5208.3, + "p50_ms": 122.23, + "p90_ms": 143.45, + "p95_ms": 166.81, + "p99_ms": 170.46, + "latencies_ms": [ + 171.378, + 120.617, + 118.026, + 122.953, + 120.385, + 120.53, + 118.348, + 118.674, + 117.604, + 128.723, + 166.571, + 134.705, + 126.227, + 122.702, + 121.845, + 122.609, + 121.473, + 119.73, + 131.185, + 140.885 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.34, + "p90_ms": 17.38, + "p95_ms": 17.74, + "p99_ms": 18.2, + "latencies_ms": [ + 16.775, + 16.508, + 13.759, + 14.172, + 14.644, + 16.087, + 15.051, + 14.784, + 14.07, + 14.503, + 14.765, + 15.028, + 15.051, + 14.895, + 16.148, + 16.583, + 15.099, + 13.592, + 15.241, + 14.096, + 15.974, + 17.38, + 18.272, + 18.09, + 18.143, + 17.388, + 17.718, + 16.242, + 16.169, + 15.525, + 15.344, + 15.098, + 14.189, + 14.447, + 13.225, + 14.793, + 15.332, + 13.991, + 14.523, + 15.886, + 15.76, + 13.086, + 13.653, + 13.37, + 13.264, + 14.48, + 13.147, + 13.12, + 16.457, + 16.801, + 16.387, + 15.919, + 17.245, + 17.384, + 17.149, + 16.819, + 15.926, + 15.831, + 17.265, + 16.386 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "semble", + "index_ms": 487.2, + "p50_ms": 4.27, + "p90_ms": 7.63, + "p95_ms": 12.37, + "p99_ms": 21.71, + "latencies_ms": [ + 1.525, + 1.178, + 0.992, + 1.011, + 0.97, + 5.185, + 4.68, + 4.698, + 4.629, + 4.632, + 5.248, + 4.452, + 4.255, + 4.378, + 4.392, + 4.875, + 4.535, + 4.579, + 4.443, + 4.253, + 4.953, + 4.326, + 4.316, + 4.388, + 4.26, + 5.042, + 6.336, + 4.948, + 4.793, + 4.848, + 4.603, + 4.213, + 4.429, + 4.29, + 4.109, + 22.916, + 21.702, + 21.38, + 21.183, + 21.21, + 5.039, + 4.391, + 4.312, + 4.304, + 4.3, + 5.05, + 4.597, + 4.539, + 4.559, + 4.57, + 4.256, + 3.849, + 3.827, + 3.829, + 3.912, + 0.786, + 0.707, + 0.73, + 0.741, + 0.787, + 0.806, + 0.796, + 0.767, + 0.836, + 0.796, + 0.765, + 0.731, + 0.732, + 0.741, + 0.719, + 4.074, + 3.649, + 3.658, + 3.672, + 3.618, + 0.767, + 0.745, + 0.733, + 0.717, + 0.725, + 11.903, + 11.374, + 11.345, + 11.191, + 11.254, + 7.235, + 6.395, + 6.404, + 6.463, + 6.343, + 0.695, + 0.655, + 0.656, + 0.666, + 0.696, + 4.189, + 3.793, + 3.816, + 3.843, + 3.784 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "bm25", + "index_ms": 487.2, + "p50_ms": 3.04, + "p90_ms": 5.32, + "p95_ms": 11.92, + "p99_ms": 21.01, + "latencies_ms": [ + 0.878, + 0.739, + 0.723, + 0.71, + 0.7, + 3.252, + 3.275, + 3.203, + 3.174, + 3.182, + 3.178, + 3.177, + 3.186, + 3.214, + 3.167, + 3.077, + 2.924, + 2.905, + 2.909, + 2.924, + 3.035, + 3.014, + 3.041, + 3.07, + 3.013, + 3.304, + 3.255, + 3.254, + 3.238, + 3.244, + 2.986, + 2.951, + 2.936, + 2.947, + 2.94, + 20.542, + 20.513, + 20.387, + 21.013, + 21.075, + 4.143, + 3.999, + 3.956, + 4.008, + 3.938, + 3.762, + 3.622, + 3.726, + 3.678, + 3.52, + 3.138, + 3.075, + 3.02, + 3.152, + 3.176, + 0.681, + 0.609, + 0.598, + 0.597, + 0.614, + 0.702, + 0.677, + 0.712, + 0.697, + 0.677, + 0.64, + 0.629, + 0.6, + 0.607, + 0.59, + 2.924, + 2.932, + 2.994, + 2.96, + 2.905, + 0.638, + 0.605, + 0.598, + 0.603, + 0.622, + 11.375, + 11.471, + 11.39, + 11.244, + 11.173, + 4.672, + 4.657, + 4.618, + 4.531, + 4.503, + 0.616, + 0.634, + 0.633, + 0.606, + 0.571, + 3.086, + 3.018, + 3.123, + 3.159, + 3.034 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "colgrep", + "index_ms": 3528.6, + "p50_ms": 121.52, + "p90_ms": 134.57, + "p95_ms": 136.09, + "p99_ms": 150.14, + "latencies_ms": [ + 153.654, + 115.391, + 116.711, + 116.893, + 120.607, + 117.654, + 128.624, + 122.763, + 135.162, + 132.464, + 134.508, + 128.684, + 134.273, + 121.772, + 126.907, + 117.835, + 115.552, + 116.333, + 116.132, + 121.275 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 12.99, + "p90_ms": 14.49, + "p95_ms": 14.72, + "p99_ms": 15.48, + "latencies_ms": [ + 14.862, + 14.578, + 13.914, + 13.932, + 12.649, + 12.621, + 14.064, + 11.922, + 12.803, + 10.571, + 11.736, + 11.54, + 12.315, + 11.844, + 11.92, + 12.237, + 10.379, + 12.995, + 11.659, + 11.433, + 11.778, + 11.363, + 10.967, + 11.108, + 12.521, + 12.143, + 11.371, + 12.13, + 13.372, + 12.64, + 13.114, + 13.679, + 11.733, + 14.115, + 13.409, + 15.298, + 14.127, + 14.117, + 13.988, + 13.787, + 14.146, + 12.383, + 14.016, + 14.71, + 13.227, + 12.703, + 12.981, + 11.157, + 14.122, + 15.734, + 13.382, + 13.957, + 13.754, + 13.128, + 14.478, + 14.554, + 14.347, + 12.88, + 13.446, + 12.519 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "semble", + "index_ms": 1238.2, + "p50_ms": 1.57, + "p90_ms": 9.53, + "p95_ms": 10.08, + "p99_ms": 14.55, + "latencies_ms": [ + 15.539, + 13.393, + 13.861, + 13.772, + 14.543, + 2.118, + 1.868, + 1.72, + 1.704, + 1.699, + 1.507, + 1.418, + 1.393, + 1.409, + 1.427, + 9.683, + 9.395, + 9.302, + 9.494, + 9.597, + 8.288, + 8.003, + 7.879, + 7.519, + 7.393, + 1.803, + 1.661, + 1.563, + 1.55, + 1.534, + 1.582, + 1.567, + 1.569, + 1.538, + 1.443, + 1.445, + 1.273, + 1.392, + 1.31, + 1.235, + 1.305, + 1.231, + 1.155, + 1.142, + 1.131, + 0.921, + 0.87, + 0.883, + 0.937, + 0.912, + 1.315, + 1.256, + 1.256, + 1.232, + 1.277, + 1.319, + 1.276, + 1.277, + 1.204, + 1.199, + 1.439, + 1.381, + 1.386, + 1.378, + 1.351, + 1.488, + 1.455, + 1.442, + 1.437, + 1.485, + 1.656, + 1.547, + 1.543, + 1.547, + 1.553, + 2.023, + 1.836, + 1.755, + 1.715, + 1.706, + 9.395, + 8.662, + 8.681, + 8.726, + 8.687, + 7.385, + 6.99, + 6.988, + 6.905, + 6.925, + 9.904, + 9.636, + 9.527, + 9.425, + 9.641, + 8.0, + 8.611, + 7.54, + 7.388, + 7.25 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "bm25", + "index_ms": 1238.2, + "p50_ms": 1.22, + "p90_ms": 8.12, + "p95_ms": 9.16, + "p99_ms": 10.63, + "latencies_ms": [ + 10.628, + 10.93, + 10.584, + 10.496, + 10.478, + 1.455, + 1.399, + 1.348, + 1.337, + 1.327, + 0.987, + 0.95, + 0.945, + 0.941, + 0.998, + 7.953, + 7.903, + 7.975, + 7.956, + 8.03, + 6.011, + 5.982, + 5.987, + 5.953, + 6.004, + 1.289, + 1.225, + 1.21, + 1.211, + 1.216, + 1.203, + 1.198, + 1.205, + 1.216, + 1.175, + 1.058, + 1.017, + 1.043, + 1.022, + 1.056, + 1.057, + 0.99, + 1.102, + 1.032, + 0.983, + 0.737, + 0.708, + 0.689, + 0.691, + 0.667, + 0.962, + 0.912, + 0.901, + 0.958, + 0.971, + 0.986, + 0.919, + 0.927, + 0.897, + 0.891, + 1.205, + 1.134, + 1.115, + 1.162, + 1.168, + 1.219, + 1.161, + 1.179, + 1.147, + 1.156, + 1.268, + 1.211, + 1.194, + 1.182, + 1.18, + 1.535, + 1.506, + 1.463, + 1.454, + 1.543, + 7.995, + 7.623, + 7.609, + 7.564, + 7.522, + 6.526, + 6.592, + 6.579, + 6.527, + 6.488, + 9.01, + 9.005, + 8.995, + 9.086, + 8.899, + 6.527, + 6.475, + 6.462, + 6.503, + 6.431 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "colgrep", + "index_ms": 9375.4, + "p50_ms": 124.68, + "p90_ms": 128.72, + "p95_ms": 130.9, + "p99_ms": 156.17, + "latencies_ms": [ + 162.491, + 125.654, + 124.08, + 128.107, + 124.973, + 121.754, + 127.52, + 121.864, + 123.352, + 124.369, + 123.435, + 123.145, + 124.199, + 125.062, + 124.382, + 123.194, + 129.236, + 127.432, + 128.661, + 126.782 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.39, + "p90_ms": 17.51, + "p95_ms": 17.65, + "p99_ms": 21.3, + "latencies_ms": [ + 25.245, + 14.865, + 14.07, + 15.364, + 15.073, + 16.123, + 14.737, + 14.582, + 14.122, + 17.045, + 16.215, + 15.712, + 14.836, + 15.53, + 15.483, + 16.799, + 17.008, + 18.558, + 17.009, + 17.306, + 15.367, + 16.511, + 16.74, + 16.559, + 15.842, + 18.298, + 17.328, + 17.487, + 16.78, + 16.777, + 17.126, + 17.159, + 17.62, + 17.063, + 15.27, + 15.239, + 14.599, + 17.609, + 17.586, + 14.678, + 16.688, + 16.423, + 16.287, + 17.497, + 16.287, + 16.329, + 17.365, + 16.339, + 16.197, + 16.875, + 16.356, + 16.413, + 15.732, + 16.757, + 16.938, + 16.376, + 15.904, + 16.921, + 15.31, + 13.283 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "semble", + "index_ms": 112.3, + "p50_ms": 0.67, + "p90_ms": 2.24, + "p95_ms": 2.42, + "p99_ms": 2.78, + "latencies_ms": [ + 1.205, + 0.729, + 0.667, + 0.641, + 0.62, + 0.523, + 0.498, + 0.477, + 0.487, + 0.477, + 0.617, + 0.592, + 0.571, + 0.566, + 0.566, + 0.634, + 0.611, + 0.6, + 0.716, + 0.673, + 2.889, + 2.189, + 2.199, + 2.219, + 2.214, + 2.123, + 1.673, + 1.671, + 1.686, + 1.679, + 2.781, + 2.362, + 2.415, + 2.381, + 2.329, + 0.576, + 0.536, + 0.527, + 0.518, + 0.515, + 0.588, + 0.557, + 0.551, + 0.556, + 0.546, + 0.729, + 0.709, + 0.683, + 0.65, + 0.653, + 0.594, + 0.566, + 0.55, + 0.559, + 0.556, + 0.601, + 0.568, + 0.57, + 0.564, + 0.566, + 0.718, + 0.71, + 0.713, + 0.687, + 0.683, + 0.648, + 0.631, + 0.624, + 0.601, + 0.599, + 0.573, + 0.553, + 0.553, + 0.551, + 0.568, + 0.684, + 0.674, + 0.687, + 0.688, + 0.659, + 0.75, + 0.723, + 0.713, + 0.707, + 0.699, + 2.566, + 2.188, + 2.157, + 2.167, + 2.146, + 2.549, + 2.139, + 2.095, + 2.049, + 2.054, + 2.65, + 2.246, + 2.242, + 2.191, + 2.22 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "bm25", + "index_ms": 112.3, + "p50_ms": 0.53, + "p90_ms": 1.8, + "p95_ms": 1.84, + "p99_ms": 1.85, + "latencies_ms": [ + 0.794, + 0.533, + 0.51, + 0.502, + 0.497, + 0.429, + 0.413, + 0.408, + 0.411, + 0.415, + 0.505, + 0.487, + 0.468, + 0.455, + 0.445, + 0.602, + 0.556, + 0.538, + 0.522, + 0.527, + 1.848, + 1.779, + 1.803, + 1.796, + 1.851, + 0.777, + 0.763, + 0.762, + 0.753, + 0.741, + 1.8, + 1.814, + 1.841, + 1.783, + 1.775, + 0.427, + 0.412, + 0.431, + 0.423, + 0.417, + 0.499, + 0.465, + 0.455, + 0.455, + 0.449, + 0.539, + 0.537, + 0.576, + 0.56, + 0.549, + 0.515, + 0.491, + 0.505, + 0.481, + 0.476, + 0.499, + 0.495, + 0.477, + 0.497, + 0.49, + 0.555, + 0.539, + 0.556, + 0.543, + 0.554, + 0.498, + 0.51, + 0.482, + 0.469, + 0.485, + 0.483, + 0.474, + 0.481, + 0.48, + 0.464, + 0.561, + 0.529, + 0.536, + 0.532, + 0.536, + 0.551, + 0.549, + 0.512, + 0.5, + 0.499, + 1.42, + 1.382, + 1.358, + 1.361, + 1.381, + 1.839, + 1.84, + 1.857, + 1.824, + 1.823, + 1.798, + 1.751, + 1.82, + 1.792, + 1.72 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "colgrep", + "index_ms": 1686.7, + "p50_ms": 117.03, + "p90_ms": 121.82, + "p95_ms": 124.69, + "p99_ms": 133.84, + "latencies_ms": [ + 136.124, + 110.538, + 110.822, + 113.246, + 117.726, + 114.398, + 116.326, + 118.371, + 121.491, + 124.088, + 121.571, + 120.469, + 112.527, + 115.266, + 112.56, + 120.651, + 116.339, + 118.259, + 120.802, + 112.251 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.71, + "p90_ms": 15.87, + "p95_ms": 16.46, + "p99_ms": 17.19, + "latencies_ms": [ + 16.887, + 14.078, + 14.43, + 12.375, + 13.405, + 14.445, + 13.76, + 15.071, + 15.077, + 14.958, + 14.442, + 15.461, + 17.626, + 15.316, + 13.776, + 14.712, + 14.247, + 15.865, + 15.49, + 13.887, + 15.534, + 15.515, + 14.156, + 16.438, + 16.799, + 14.76, + 16.001, + 15.071, + 15.69, + 15.504, + 14.199, + 15.484, + 13.853, + 13.942, + 14.819, + 15.316, + 15.433, + 15.532, + 15.911, + 15.62, + 14.701, + 12.858, + 14.558, + 10.754, + 11.617, + 11.834, + 13.173, + 12.683, + 13.287, + 13.722, + 15.274, + 15.522, + 14.919, + 13.981, + 11.778, + 11.346, + 12.733, + 13.143, + 14.933, + 12.498 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "semble", + "index_ms": 413.0, + "p50_ms": 1.78, + "p90_ms": 5.71, + "p95_ms": 5.94, + "p99_ms": 6.48, + "latencies_ms": [ + 7.413, + 6.286, + 6.181, + 6.301, + 6.475, + 1.235, + 1.086, + 1.025, + 1.001, + 1.008, + 0.867, + 0.913, + 0.998, + 0.9, + 0.858, + 1.121, + 1.055, + 1.038, + 1.047, + 1.042, + 0.993, + 0.983, + 1.047, + 1.107, + 1.011, + 2.695, + 2.232, + 2.145, + 3.285, + 2.036, + 1.449, + 1.332, + 1.411, + 1.464, + 1.519, + 1.277, + 1.13, + 1.038, + 1.077, + 1.083, + 4.351, + 4.453, + 5.134, + 5.25, + 4.51, + 1.464, + 1.169, + 1.121, + 1.109, + 1.319, + 1.371, + 1.481, + 1.46, + 1.172, + 1.085, + 5.45, + 5.435, + 5.424, + 5.314, + 5.371, + 5.708, + 4.983, + 5.043, + 5.063, + 5.018, + 3.979, + 3.612, + 3.558, + 3.555, + 3.786, + 5.192, + 4.781, + 4.788, + 5.219, + 4.764, + 4.327, + 3.542, + 3.533, + 3.49, + 3.428, + 5.932, + 5.881, + 5.885, + 5.775, + 5.898, + 1.373, + 1.237, + 1.145, + 1.169, + 1.228, + 0.984, + 0.886, + 0.88, + 0.876, + 0.857, + 4.076, + 3.933, + 3.678, + 3.51, + 3.555 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "bm25", + "index_ms": 413.0, + "p50_ms": 0.92, + "p90_ms": 4.75, + "p95_ms": 5.43, + "p99_ms": 5.82, + "latencies_ms": [ + 5.387, + 5.355, + 5.401, + 5.394, + 5.422, + 0.789, + 0.738, + 0.712, + 0.723, + 0.702, + 0.736, + 0.712, + 0.709, + 0.706, + 0.707, + 0.855, + 0.839, + 0.824, + 0.814, + 0.917, + 0.867, + 0.785, + 0.763, + 0.749, + 0.744, + 0.679, + 0.662, + 0.652, + 0.646, + 0.661, + 0.626, + 0.599, + 0.581, + 0.575, + 0.578, + 0.543, + 0.512, + 0.52, + 0.518, + 0.505, + 2.869, + 2.788, + 3.072, + 2.95, + 2.848, + 0.759, + 0.72, + 0.826, + 0.806, + 0.753, + 0.88, + 0.821, + 0.807, + 0.808, + 0.815, + 4.613, + 4.595, + 4.642, + 4.683, + 4.545, + 4.275, + 4.167, + 4.225, + 4.433, + 4.259, + 2.588, + 2.689, + 2.575, + 2.538, + 2.494, + 4.041, + 4.017, + 4.094, + 4.235, + 4.249, + 3.057, + 3.131, + 3.054, + 3.026, + 3.032, + 5.536, + 5.514, + 5.82, + 5.824, + 5.775, + 1.103, + 0.928, + 0.883, + 1.054, + 1.185, + 0.963, + 0.809, + 0.778, + 0.76, + 0.762, + 3.011, + 3.049, + 2.98, + 2.946, + 4.419 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "colgrep", + "index_ms": 3759.3, + "p50_ms": 115.63, + "p90_ms": 119.86, + "p95_ms": 126.49, + "p99_ms": 138.88, + "latencies_ms": [ + 141.983, + 117.253, + 119.094, + 125.675, + 119.219, + 114.2, + 114.791, + 115.076, + 115.108, + 115.785, + 115.359, + 116.823, + 113.43, + 117.142, + 113.937, + 114.913, + 115.483, + 115.011, + 116.558, + 115.802 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.57, + "p90_ms": 16.01, + "p95_ms": 16.36, + "p99_ms": 17.29, + "latencies_ms": [ + 15.102, + 13.007, + 13.907, + 13.72, + 14.377, + 13.193, + 11.885, + 12.779, + 12.766, + 13.164, + 12.714, + 11.587, + 14.224, + 14.556, + 12.15, + 13.807, + 13.35, + 15.438, + 14.094, + 14.091, + 16.857, + 15.786, + 15.029, + 14.415, + 14.073, + 14.756, + 14.703, + 14.861, + 13.721, + 14.582, + 14.071, + 14.606, + 12.946, + 14.482, + 15.403, + 14.714, + 15.486, + 16.768, + 15.995, + 14.921, + 15.229, + 14.696, + 16.339, + 14.55, + 14.042, + 14.127, + 12.918, + 14.954, + 14.206, + 15.125, + 17.905, + 15.707, + 16.091, + 15.75, + 15.452, + 15.137, + 16.296, + 14.911, + 14.978, + 14.457 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "semble", + "index_ms": 447.7, + "p50_ms": 0.79, + "p90_ms": 0.93, + "p95_ms": 1.6, + "p99_ms": 3.82, + "latencies_ms": [ + 1.495, + 0.968, + 0.886, + 0.87, + 0.851, + 0.886, + 0.831, + 0.776, + 0.745, + 0.733, + 0.859, + 0.809, + 0.791, + 0.792, + 0.81, + 0.845, + 0.816, + 0.804, + 0.96, + 0.882, + 0.962, + 0.933, + 0.896, + 0.881, + 0.878, + 0.884, + 0.844, + 0.823, + 0.823, + 0.946, + 0.788, + 0.713, + 0.697, + 0.689, + 0.688, + 0.829, + 0.79, + 0.776, + 0.771, + 0.772, + 0.784, + 0.777, + 0.81, + 0.797, + 0.777, + 4.519, + 3.808, + 3.793, + 3.66, + 3.725, + 0.85, + 0.796, + 0.78, + 0.772, + 0.775, + 0.804, + 0.771, + 0.772, + 0.757, + 0.755, + 0.846, + 0.815, + 0.801, + 0.781, + 0.77, + 0.932, + 0.901, + 0.898, + 0.921, + 0.924, + 0.831, + 0.817, + 0.789, + 0.789, + 0.79, + 0.788, + 0.754, + 0.738, + 0.739, + 0.732, + 0.796, + 0.764, + 0.762, + 0.77, + 0.747, + 0.788, + 0.746, + 0.715, + 0.712, + 0.706, + 0.708, + 0.683, + 0.692, + 0.678, + 0.681, + 0.812, + 0.78, + 0.787, + 0.752, + 0.835 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "bm25", + "index_ms": 447.7, + "p50_ms": 0.67, + "p90_ms": 0.76, + "p95_ms": 0.91, + "p99_ms": 3.19, + "latencies_ms": [ + 0.794, + 0.711, + 0.702, + 0.696, + 0.696, + 0.68, + 0.667, + 0.658, + 0.65, + 0.648, + 0.689, + 0.693, + 0.676, + 0.648, + 0.64, + 0.719, + 0.715, + 0.689, + 0.706, + 0.694, + 0.771, + 0.763, + 0.774, + 0.76, + 0.738, + 0.769, + 0.754, + 0.716, + 0.713, + 0.719, + 0.591, + 0.576, + 0.57, + 0.566, + 0.579, + 0.691, + 0.681, + 0.652, + 0.667, + 0.688, + 0.662, + 0.725, + 0.716, + 0.649, + 0.61, + 3.246, + 3.117, + 3.192, + 3.112, + 3.077, + 0.665, + 0.637, + 0.631, + 0.715, + 0.669, + 0.659, + 0.632, + 0.633, + 0.629, + 0.621, + 0.744, + 0.718, + 0.709, + 0.717, + 0.695, + 0.74, + 0.721, + 0.769, + 0.764, + 0.727, + 0.68, + 0.669, + 0.655, + 0.649, + 0.647, + 0.654, + 0.663, + 0.62, + 0.614, + 0.616, + 0.635, + 0.602, + 0.6, + 0.677, + 0.647, + 0.616, + 0.6, + 0.591, + 0.598, + 0.607, + 0.634, + 0.579, + 0.573, + 0.574, + 0.572, + 0.671, + 0.626, + 0.646, + 0.624, + 0.659 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "colgrep", + "index_ms": 3642.8, + "p50_ms": 112.98, + "p90_ms": 114.72, + "p95_ms": 120.09, + "p99_ms": 138.59, + "latencies_ms": [ + 143.222, + 112.916, + 114.08, + 112.697, + 112.992, + 114.178, + 112.549, + 112.861, + 113.716, + 113.116, + 112.975, + 113.133, + 112.13, + 114.189, + 114.26, + 112.932, + 112.739, + 111.981, + 112.299, + 118.87 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 11.62, + "p90_ms": 13.33, + "p95_ms": 13.45, + "p99_ms": 13.94, + "latencies_ms": [ + 13.883, + 11.838, + 11.447, + 11.827, + 10.72, + 11.843, + 11.114, + 10.261, + 11.824, + 13.441, + 12.365, + 11.632, + 11.394, + 12.789, + 11.298, + 11.033, + 11.535, + 10.084, + 11.421, + 11.297, + 10.787, + 13.316, + 12.082, + 11.549, + 11.29, + 12.425, + 11.339, + 11.438, + 12.18, + 10.501, + 11.286, + 12.034, + 14.012, + 13.576, + 12.222, + 11.472, + 10.721, + 12.628, + 10.995, + 11.984, + 11.24, + 11.122, + 11.491, + 13.254, + 11.306, + 13.423, + 12.006, + 11.924, + 10.815, + 11.568, + 11.712, + 11.33, + 11.618, + 11.625, + 13.44, + 12.914, + 12.99, + 11.731, + 11.653, + 11.218 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "semble", + "index_ms": 352.2, + "p50_ms": 0.97, + "p90_ms": 3.32, + "p95_ms": 3.49, + "p99_ms": 3.82, + "latencies_ms": [ + 1.269, + 0.919, + 0.892, + 0.843, + 0.82, + 1.001, + 0.936, + 0.919, + 0.93, + 0.916, + 0.863, + 0.799, + 0.794, + 0.911, + 0.853, + 0.773, + 0.716, + 0.702, + 0.726, + 0.701, + 4.109, + 3.439, + 3.329, + 3.42, + 3.489, + 3.559, + 3.109, + 3.157, + 3.182, + 3.141, + 3.759, + 3.291, + 3.294, + 3.328, + 3.258, + 0.914, + 0.888, + 0.88, + 0.886, + 0.867, + 1.074, + 1.048, + 1.051, + 1.032, + 1.049, + 0.966, + 0.942, + 0.94, + 0.934, + 0.989, + 0.985, + 0.969, + 0.938, + 0.934, + 0.919, + 0.918, + 0.896, + 0.879, + 0.878, + 0.874, + 0.845, + 0.787, + 0.77, + 0.794, + 0.786, + 0.878, + 0.835, + 0.812, + 0.814, + 0.837, + 1.009, + 0.969, + 1.043, + 0.996, + 0.981, + 1.021, + 0.987, + 0.971, + 0.97, + 0.958, + 1.097, + 1.073, + 1.16, + 1.119, + 1.081, + 1.0, + 0.988, + 0.963, + 0.969, + 0.978, + 3.813, + 3.299, + 3.302, + 3.323, + 3.245, + 3.487, + 3.039, + 3.004, + 3.034, + 3.072 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "bm25", + "index_ms": 352.2, + "p50_ms": 0.75, + "p90_ms": 2.48, + "p95_ms": 2.53, + "p99_ms": 2.61, + "latencies_ms": [ + 0.715, + 0.674, + 0.641, + 0.615, + 0.62, + 0.763, + 0.738, + 0.728, + 0.724, + 0.723, + 0.548, + 0.57, + 0.549, + 0.54, + 0.536, + 0.586, + 0.576, + 0.554, + 0.552, + 0.545, + 2.584, + 2.545, + 2.605, + 2.622, + 2.568, + 2.526, + 2.484, + 2.457, + 2.463, + 2.489, + 2.484, + 2.452, + 2.409, + 2.419, + 2.476, + 0.703, + 0.663, + 0.651, + 0.664, + 0.652, + 0.824, + 0.806, + 0.818, + 0.82, + 0.8, + 0.747, + 0.711, + 0.686, + 0.699, + 0.706, + 0.798, + 0.787, + 0.8, + 0.777, + 0.768, + 0.788, + 0.771, + 0.782, + 0.855, + 0.854, + 0.688, + 0.676, + 0.64, + 0.623, + 0.613, + 0.756, + 0.732, + 0.742, + 0.713, + 0.733, + 0.714, + 0.664, + 0.691, + 0.683, + 0.664, + 0.777, + 0.743, + 0.723, + 0.758, + 0.77, + 0.884, + 0.87, + 0.866, + 0.864, + 0.863, + 0.74, + 0.742, + 0.709, + 0.659, + 0.68, + 2.41, + 2.335, + 2.365, + 2.395, + 2.3, + 2.423, + 2.442, + 2.532, + 2.478, + 2.455 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "colgrep", + "index_ms": 4126.2, + "p50_ms": 119.14, + "p90_ms": 125.31, + "p95_ms": 127.72, + "p99_ms": 139.99, + "latencies_ms": [ + 143.052, + 119.128, + 117.118, + 118.38, + 125.137, + 126.918, + 117.895, + 119.146, + 118.759, + 116.892, + 119.336, + 119.186, + 116.612, + 118.535, + 121.398, + 120.319, + 118.445, + 119.599, + 119.274, + 113.655 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.73, + "p90_ms": 15.0, + "p95_ms": 15.56, + "p99_ms": 16.1, + "latencies_ms": [ + 15.8, + 14.471, + 13.875, + 12.293, + 15.557, + 13.625, + 16.522, + 12.655, + 14.655, + 13.005, + 13.159, + 14.24, + 14.554, + 13.617, + 12.59, + 13.755, + 14.647, + 13.503, + 13.457, + 13.602, + 15.656, + 13.216, + 14.394, + 14.635, + 12.669, + 14.747, + 13.292, + 12.767, + 15.269, + 14.182, + 13.845, + 13.713, + 13.842, + 13.146, + 13.418, + 15.203, + 14.968, + 13.493, + 12.268, + 13.328, + 13.584, + 13.92, + 14.706, + 12.099, + 14.199, + 14.897, + 13.775, + 13.478, + 14.751, + 14.305, + 13.24, + 13.457, + 14.981, + 13.895, + 13.205, + 14.434, + 12.847, + 12.566, + 11.239, + 13.039 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "semble", + "index_ms": 263.4, + "p50_ms": 0.51, + "p90_ms": 2.48, + "p95_ms": 3.27, + "p99_ms": 3.76, + "latencies_ms": [ + 0.91, + 0.535, + 0.507, + 0.503, + 0.484, + 0.512, + 0.454, + 0.447, + 0.443, + 0.448, + 0.411, + 0.399, + 0.394, + 0.402, + 0.387, + 0.419, + 0.382, + 0.467, + 0.477, + 0.424, + 0.509, + 0.477, + 0.441, + 0.436, + 0.429, + 0.422, + 0.394, + 0.381, + 0.38, + 0.39, + 0.507, + 0.461, + 0.449, + 0.441, + 0.436, + 0.431, + 0.402, + 0.402, + 0.396, + 0.396, + 0.486, + 0.506, + 0.445, + 0.431, + 0.412, + 0.534, + 0.489, + 0.478, + 0.479, + 0.49, + 0.722, + 0.68, + 0.629, + 0.626, + 0.637, + 0.736, + 0.7, + 0.687, + 0.799, + 0.759, + 0.798, + 0.757, + 0.747, + 0.735, + 0.741, + 0.572, + 0.555, + 0.542, + 0.552, + 0.562, + 0.751, + 0.715, + 0.745, + 0.739, + 0.722, + 3.878, + 3.225, + 3.242, + 3.186, + 3.169, + 2.686, + 2.181, + 2.114, + 2.14, + 2.057, + 3.776, + 3.348, + 3.284, + 3.282, + 3.309, + 0.409, + 0.392, + 0.377, + 0.365, + 0.368, + 0.343, + 0.328, + 0.32, + 0.35, + 0.354, + 0.736, + 0.673, + 0.661, + 0.674, + 0.669 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "bm25", + "index_ms": 263.4, + "p50_ms": 0.42, + "p90_ms": 2.11, + "p95_ms": 2.47, + "p99_ms": 3.27, + "latencies_ms": [ + 0.561, + 0.433, + 0.423, + 0.431, + 0.409, + 0.45, + 0.416, + 0.403, + 0.397, + 0.414, + 0.373, + 0.359, + 0.377, + 0.369, + 0.369, + 0.384, + 0.361, + 0.359, + 0.362, + 0.354, + 0.351, + 0.321, + 0.314, + 0.335, + 0.313, + 0.364, + 0.345, + 0.357, + 0.356, + 0.364, + 0.42, + 0.379, + 0.374, + 0.386, + 0.398, + 0.371, + 0.341, + 0.341, + 0.348, + 0.337, + 0.372, + 0.356, + 0.331, + 0.32, + 0.314, + 0.447, + 0.408, + 0.399, + 0.428, + 0.46, + 0.635, + 0.556, + 0.565, + 0.553, + 0.541, + 0.636, + 0.614, + 0.607, + 0.607, + 0.605, + 0.78, + 0.725, + 0.699, + 0.707, + 0.807, + 0.574, + 0.586, + 0.596, + 0.532, + 0.463, + 0.696, + 0.672, + 0.707, + 0.701, + 0.673, + 3.234, + 3.272, + 3.133, + 3.251, + 3.304, + 2.114, + 2.022, + 2.056, + 2.094, + 2.035, + 2.468, + 2.44, + 2.466, + 2.358, + 2.346, + 0.371, + 0.347, + 0.338, + 0.335, + 0.34, + 0.251, + 0.233, + 0.226, + 0.228, + 0.232, + 0.629, + 0.572, + 0.541, + 0.53, + 0.537 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "colgrep", + "index_ms": 2296.9, + "p50_ms": 115.43, + "p90_ms": 120.82, + "p95_ms": 123.97, + "p99_ms": 131.01, + "latencies_ms": [ + 132.776, + 109.99, + 111.199, + 112.698, + 112.948, + 116.94, + 116.723, + 118.96, + 110.311, + 110.216, + 110.581, + 112.865, + 116.711, + 115.433, + 118.528, + 120.817, + 123.966, + 115.823, + 115.608, + 112.947, + 110.177 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.27, + "p90_ms": 14.72, + "p95_ms": 15.08, + "p99_ms": 15.88, + "latencies_ms": [ + 14.568, + 13.059, + 12.945, + 12.85, + 13.202, + 11.62, + 12.888, + 11.099, + 13.053, + 12.02, + 12.633, + 12.657, + 12.547, + 14.405, + 16.7, + 14.565, + 14.391, + 15.112, + 14.245, + 13.484, + 14.16, + 13.034, + 12.495, + 12.919, + 14.315, + 15.38, + 15.054, + 13.815, + 13.823, + 14.702, + 13.911, + 11.523, + 12.665, + 15.08, + 12.155, + 11.434, + 11.583, + 11.983, + 10.673, + 12.698, + 12.802, + 14.224, + 11.856, + 13.031, + 14.175, + 14.119, + 13.948, + 13.894, + 14.105, + 13.269, + 13.367, + 12.15, + 13.826, + 14.159, + 13.174, + 14.723, + 14.855, + 14.11, + 13.204, + 13.285, + 12.842, + 13.274, + 12.795 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "semble", + "index_ms": 248.4, + "p50_ms": 0.61, + "p90_ms": 3.65, + "p95_ms": 5.01, + "p99_ms": 5.52, + "latencies_ms": [ + 0.993, + 3.675, + 0.619, + 3.275, + 4.076, + 0.745, + 0.771, + 3.489, + 0.651, + 0.626, + 0.484, + 0.446, + 0.435, + 0.437, + 0.429, + 0.521, + 0.483, + 0.48, + 0.477, + 0.49, + 3.577, + 3.026, + 2.938, + 3.009, + 3.031, + 6.149, + 5.426, + 5.399, + 5.42, + 5.444, + 4.531, + 3.622, + 3.647, + 3.645, + 3.577, + 0.617, + 0.577, + 0.612, + 0.607, + 0.59, + 0.569, + 0.53, + 0.544, + 0.538, + 0.532, + 0.608, + 0.57, + 0.567, + 0.578, + 0.573, + 0.537, + 0.523, + 0.529, + 0.532, + 0.565, + 0.735, + 0.648, + 0.634, + 0.602, + 0.586, + 0.596, + 0.565, + 0.573, + 0.55, + 0.547, + 0.542, + 0.507, + 0.5, + 0.496, + 0.499, + 0.595, + 0.571, + 0.56, + 0.632, + 0.583, + 0.686, + 0.651, + 0.655, + 0.65, + 0.659, + 3.36, + 2.922, + 2.843, + 2.813, + 2.807, + 3.244, + 2.739, + 2.774, + 2.817, + 2.795 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "bm25", + "index_ms": 248.4, + "p50_ms": 0.5, + "p90_ms": 3.19, + "p95_ms": 3.72, + "p99_ms": 4.19, + "latencies_ms": [ + 0.53, + 0.462, + 0.456, + 0.45, + 0.455, + 0.549, + 0.516, + 0.516, + 0.509, + 0.502, + 0.344, + 0.362, + 0.358, + 0.349, + 0.357, + 0.439, + 0.419, + 0.413, + 0.432, + 0.424, + 2.654, + 2.59, + 2.668, + 2.625, + 2.636, + 4.169, + 4.123, + 4.143, + 4.192, + 4.215, + 3.202, + 3.2, + 3.191, + 3.231, + 3.194, + 0.493, + 0.479, + 0.465, + 0.479, + 0.47, + 0.415, + 0.41, + 0.397, + 0.402, + 0.399, + 0.523, + 0.514, + 0.512, + 0.494, + 0.523, + 0.451, + 0.427, + 0.428, + 0.428, + 0.421, + 0.498, + 0.48, + 0.495, + 0.493, + 0.482, + 0.48, + 0.459, + 0.455, + 0.47, + 0.459, + 0.446, + 0.422, + 0.426, + 0.425, + 0.426, + 0.499, + 0.577, + 0.564, + 0.524, + 0.508, + 0.57, + 0.535, + 0.527, + 0.525, + 0.527, + 2.135, + 2.059, + 2.102, + 2.082, + 2.049, + 2.621, + 2.636, + 2.614, + 2.566, + 2.593 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "colgrep", + "index_ms": 2796.3, + "p50_ms": 110.34, + "p90_ms": 112.97, + "p95_ms": 118.71, + "p99_ms": 131.95, + "latencies_ms": [ + 135.255, + 111.203, + 108.939, + 109.441, + 111.654, + 110.424, + 110.218, + 109.527, + 110.253, + 115.785, + 111.767, + 109.253, + 110.955, + 109.6, + 110.161, + 110.793, + 109.931, + 110.811 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.61, + "p90_ms": 15.4, + "p95_ms": 15.82, + "p99_ms": 16.52, + "latencies_ms": [ + 13.049, + 11.998, + 11.981, + 13.955, + 12.168, + 12.039, + 13.153, + 12.598, + 11.768, + 12.513, + 11.421, + 10.92, + 14.942, + 15.052, + 13.577, + 13.613, + 15.702, + 16.936, + 14.734, + 14.367, + 13.168, + 15.534, + 14.854, + 13.2, + 13.6, + 14.227, + 14.426, + 14.62, + 14.315, + 13.294, + 16.147, + 15.739, + 14.437, + 14.158, + 14.265, + 13.083, + 13.584, + 14.768, + 15.081, + 14.262, + 14.159, + 13.984, + 13.908, + 14.363, + 13.318, + 12.211, + 12.834, + 11.797, + 12.175, + 12.61, + 13.133, + 13.46, + 15.973, + 11.666 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "semble", + "index_ms": 416.9, + "p50_ms": 0.9, + "p90_ms": 4.32, + "p95_ms": 4.86, + "p99_ms": 5.98, + "latencies_ms": [ + 1.255, + 0.944, + 0.85, + 0.795, + 0.791, + 0.897, + 0.83, + 0.87, + 0.931, + 0.885, + 6.979, + 5.964, + 5.907, + 5.851, + 5.967, + 0.843, + 0.739, + 0.715, + 0.681, + 0.682, + 0.656, + 0.627, + 0.624, + 0.62, + 0.631, + 0.751, + 0.781, + 0.72, + 0.688, + 0.677, + 4.812, + 4.375, + 4.35, + 4.319, + 4.32, + 1.054, + 1.035, + 1.056, + 1.034, + 1.046, + 0.816, + 0.785, + 0.77, + 0.765, + 0.764, + 0.818, + 0.79, + 0.783, + 0.787, + 0.782, + 0.808, + 0.807, + 0.879, + 0.83, + 0.791, + 3.365, + 3.423, + 3.468, + 3.362, + 3.323, + 4.729, + 4.373, + 4.294, + 4.175, + 4.227, + 4.232, + 3.736, + 3.751, + 3.78, + 3.706, + 0.81, + 0.761, + 0.755, + 0.75, + 0.758, + 0.903, + 0.878, + 0.884, + 0.881, + 0.895, + 0.722, + 0.657, + 0.713, + 0.697, + 0.665, + 3.455, + 3.36, + 3.327, + 3.339, + 3.444, + 3.59, + 3.491, + 3.531, + 3.596, + 3.714, + 3.726, + 3.653, + 3.713, + 3.776, + 3.677 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "bm25", + "index_ms": 416.9, + "p50_ms": 0.71, + "p90_ms": 3.37, + "p95_ms": 3.5, + "p99_ms": 4.72, + "latencies_ms": [ + 0.771, + 0.692, + 0.694, + 0.708, + 0.682, + 0.745, + 0.686, + 0.659, + 0.648, + 0.644, + 4.769, + 4.713, + 4.688, + 4.722, + 4.596, + 0.575, + 0.58, + 0.549, + 0.545, + 0.541, + 0.594, + 0.621, + 0.57, + 0.552, + 0.549, + 0.602, + 0.58, + 0.579, + 0.563, + 0.565, + 3.401, + 3.405, + 3.372, + 3.348, + 3.336, + 0.857, + 0.83, + 0.824, + 0.821, + 0.817, + 0.637, + 0.603, + 0.601, + 0.621, + 0.606, + 0.638, + 0.639, + 0.622, + 0.605, + 0.601, + 0.654, + 0.63, + 0.614, + 0.607, + 0.622, + 3.013, + 2.987, + 2.971, + 2.928, + 2.993, + 3.358, + 3.441, + 3.404, + 3.377, + 3.323, + 3.185, + 3.258, + 3.24, + 3.136, + 3.145, + 0.672, + 0.635, + 0.618, + 0.616, + 0.627, + 0.743, + 0.704, + 0.705, + 0.713, + 0.744, + 0.683, + 0.618, + 0.569, + 0.56, + 0.57, + 2.81, + 2.755, + 2.969, + 3.032, + 2.931, + 3.034, + 2.937, + 2.889, + 3.016, + 3.13, + 3.2, + 3.093, + 3.055, + 2.967, + 2.948 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "colgrep", + "index_ms": 6675.1, + "p50_ms": 120.09, + "p90_ms": 131.15, + "p95_ms": 147.36, + "p99_ms": 150.19, + "latencies_ms": [ + 150.893, + 118.182, + 119.19, + 121.911, + 147.177, + 123.625, + 122.176, + 127.106, + 122.087, + 119.912, + 118.081, + 117.859, + 120.176, + 122.234, + 117.48, + 119.63, + 120.005, + 119.889, + 119.624, + 129.373 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.01, + "p90_ms": 16.61, + "p95_ms": 16.7, + "p99_ms": 17.14, + "latencies_ms": [ + 15.965, + 16.306, + 16.467, + 16.47, + 16.435, + 14.612, + 14.292, + 14.195, + 14.132, + 12.494, + 15.589, + 14.28, + 14.331, + 14.795, + 16.604, + 15.657, + 15.903, + 14.468, + 14.795, + 14.815, + 12.058, + 11.805, + 12.0, + 14.564, + 16.399, + 13.653, + 14.997, + 13.313, + 15.651, + 13.385, + 13.634, + 13.408, + 12.917, + 14.481, + 13.577, + 13.169, + 12.868, + 14.178, + 15.363, + 16.211, + 16.704, + 15.479, + 14.778, + 15.208, + 16.652, + 15.031, + 15.566, + 17.498, + 15.684, + 15.843, + 15.097, + 16.884, + 16.709, + 15.322, + 15.822, + 16.678, + 15.12, + 14.087, + 15.553, + 13.682 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "semble", + "index_ms": 1046.8, + "p50_ms": 5.23, + "p90_ms": 7.35, + "p95_ms": 7.93, + "p99_ms": 8.76, + "latencies_ms": [ + 7.486, + 6.301, + 6.324, + 6.033, + 5.904, + 1.198, + 1.041, + 1.171, + 1.162, + 1.088, + 6.277, + 5.655, + 5.659, + 5.479, + 5.471, + 5.532, + 5.091, + 5.067, + 5.133, + 5.092, + 1.063, + 0.932, + 0.968, + 0.964, + 0.927, + 0.885, + 0.822, + 0.815, + 0.806, + 0.813, + 8.665, + 8.173, + 8.453, + 9.072, + 8.752, + 1.306, + 1.046, + 0.999, + 0.976, + 0.98, + 1.033, + 1.001, + 0.973, + 0.972, + 0.975, + 1.047, + 1.008, + 0.981, + 0.986, + 0.999, + 6.218, + 5.925, + 5.886, + 5.78, + 5.943, + 1.175, + 1.015, + 0.987, + 0.967, + 0.981, + 7.724, + 7.092, + 7.225, + 7.253, + 7.215, + 1.135, + 1.011, + 1.0, + 0.98, + 0.964, + 7.919, + 7.343, + 7.309, + 7.454, + 7.457, + 0.878, + 0.774, + 0.759, + 0.824, + 0.78, + 6.179, + 5.733, + 5.675, + 5.717, + 5.883, + 5.284, + 5.602, + 5.445, + 5.448, + 6.08, + 5.464, + 5.26, + 5.242, + 5.221, + 5.238, + 6.403, + 6.355, + 6.237, + 5.908, + 5.847 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "bm25", + "index_ms": 1046.8, + "p50_ms": 4.78, + "p90_ms": 6.73, + "p95_ms": 6.93, + "p99_ms": 7.67, + "latencies_ms": [ + 6.1, + 5.565, + 5.251, + 5.119, + 5.011, + 1.232, + 1.097, + 1.064, + 1.053, + 1.001, + 5.347, + 5.285, + 5.153, + 5.148, + 5.176, + 4.783, + 4.94, + 4.946, + 4.78, + 4.685, + 0.992, + 0.879, + 0.795, + 0.857, + 0.927, + 0.759, + 0.696, + 0.668, + 0.695, + 0.742, + 6.522, + 6.908, + 6.581, + 6.564, + 6.78, + 1.141, + 0.941, + 0.901, + 0.915, + 1.019, + 1.31, + 0.971, + 0.913, + 0.915, + 0.992, + 0.973, + 0.85, + 0.803, + 1.106, + 1.006, + 5.356, + 4.973, + 5.087, + 4.958, + 4.855, + 1.08, + 0.916, + 0.846, + 0.832, + 0.848, + 6.55, + 6.709, + 6.924, + 6.79, + 6.651, + 1.125, + 1.13, + 1.169, + 0.935, + 1.082, + 7.674, + 7.739, + 7.033, + 7.42, + 6.998, + 1.099, + 1.136, + 1.163, + 0.971, + 0.824, + 5.953, + 6.729, + 6.033, + 6.836, + 5.692, + 5.177, + 5.425, + 4.973, + 4.845, + 4.598, + 4.713, + 5.025, + 5.136, + 5.146, + 4.637, + 6.458, + 5.864, + 5.608, + 5.368, + 5.486 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "colgrep", + "index_ms": 4409.1, + "p50_ms": 123.66, + "p90_ms": 129.14, + "p95_ms": 130.29, + "p99_ms": 144.09, + "latencies_ms": [ + 147.538, + 127.631, + 126.127, + 129.381, + 129.109, + 122.441, + 124.872, + 120.318, + 119.091, + 129.04, + 124.999, + 125.325, + 127.887, + 121.213, + 121.293, + 120.013, + 120.813, + 121.194, + 121.943, + 121.01 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.85, + "p90_ms": 16.87, + "p95_ms": 17.26, + "p99_ms": 29.82, + "latencies_ms": [ + 16.696, + 15.447, + 16.351, + 16.739, + 15.707, + 47.375, + 17.579, + 15.929, + 15.787, + 14.276, + 13.135, + 15.695, + 16.807, + 14.85, + 16.21, + 16.847, + 17.247, + 16.566, + 15.635, + 15.899, + 16.07, + 14.473, + 14.649, + 16.476, + 16.116, + 14.733, + 14.746, + 13.16, + 14.367, + 14.837, + 15.546, + 14.492, + 16.664, + 16.256, + 16.204, + 15.165, + 16.013, + 17.618, + 16.08, + 15.729, + 15.283, + 15.163, + 16.703, + 17.154, + 15.793, + 16.376, + 15.312, + 15.433, + 15.197, + 14.774, + 14.482, + 15.322, + 16.411, + 16.64, + 16.583, + 17.118, + 15.437, + 15.687, + 16.694, + 16.298 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "semble", + "index_ms": 473.7, + "p50_ms": 0.96, + "p90_ms": 4.85, + "p95_ms": 6.17, + "p99_ms": 7.01, + "latencies_ms": [ + 1.339, + 0.96, + 0.924, + 0.897, + 0.895, + 1.11, + 1.019, + 0.982, + 0.982, + 1.01, + 0.813, + 0.74, + 0.738, + 0.753, + 0.737, + 0.812, + 0.784, + 0.86, + 0.815, + 0.781, + 0.843, + 0.76, + 0.752, + 0.744, + 0.734, + 0.55, + 0.522, + 0.515, + 0.521, + 0.515, + 0.621, + 0.658, + 0.656, + 0.603, + 0.581, + 0.853, + 0.786, + 0.772, + 0.778, + 0.783, + 0.851, + 0.788, + 0.774, + 0.761, + 0.764, + 0.952, + 0.84, + 0.807, + 0.793, + 0.798, + 6.137, + 5.343, + 5.346, + 5.276, + 5.245, + 0.943, + 0.907, + 0.895, + 0.887, + 0.897, + 7.344, + 6.886, + 7.001, + 7.01, + 6.795, + 4.808, + 4.34, + 4.233, + 4.214, + 4.283, + 3.832, + 3.432, + 4.548, + 3.385, + 3.368, + 3.692, + 3.206, + 3.198, + 3.246, + 3.268, + 4.418, + 3.884, + 3.924, + 3.854, + 3.873, + 1.094, + 0.987, + 1.001, + 0.965, + 0.954, + 1.049, + 1.016, + 1.0, + 0.995, + 0.985, + 4.413, + 4.001, + 3.972, + 3.926, + 3.962 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "bm25", + "index_ms": 473.7, + "p50_ms": 0.86, + "p90_ms": 3.91, + "p95_ms": 4.82, + "p99_ms": 6.53, + "latencies_ms": [ + 1.009, + 0.865, + 0.865, + 0.861, + 0.838, + 0.853, + 0.832, + 0.865, + 0.821, + 0.815, + 0.691, + 0.708, + 0.678, + 0.668, + 0.657, + 0.682, + 0.684, + 0.661, + 0.659, + 0.647, + 0.68, + 0.66, + 0.65, + 0.651, + 0.655, + 0.457, + 0.475, + 0.468, + 0.445, + 0.443, + 0.528, + 0.501, + 0.484, + 0.486, + 0.478, + 0.724, + 0.69, + 0.694, + 0.684, + 0.686, + 0.714, + 0.683, + 0.7, + 0.725, + 0.725, + 0.65, + 0.616, + 0.607, + 0.604, + 0.604, + 4.721, + 4.68, + 4.682, + 4.742, + 4.675, + 0.821, + 0.787, + 0.765, + 0.819, + 0.782, + 6.531, + 6.39, + 6.535, + 6.478, + 6.409, + 3.798, + 3.736, + 3.73, + 3.693, + 3.825, + 3.048, + 3.035, + 3.053, + 3.125, + 2.986, + 3.023, + 3.002, + 2.931, + 2.931, + 2.987, + 3.274, + 3.212, + 3.263, + 3.285, + 3.248, + 0.954, + 0.936, + 0.9, + 0.871, + 0.858, + 0.923, + 0.903, + 0.884, + 0.878, + 0.894, + 3.325, + 3.306, + 3.29, + 3.354, + 3.356 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "colgrep", + "index_ms": 4121.8, + "p50_ms": 114.61, + "p90_ms": 118.85, + "p95_ms": 125.2, + "p99_ms": 139.17, + "latencies_ms": [ + 142.657, + 116.153, + 118.248, + 116.431, + 112.462, + 124.279, + 114.63, + 113.055, + 114.067, + 114.956, + 114.625, + 114.597, + 113.853, + 115.445, + 112.548, + 113.205, + 114.717, + 113.921, + 113.508, + 114.341 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.59, + "p90_ms": 14.75, + "p95_ms": 15.48, + "p99_ms": 16.66, + "latencies_ms": [ + 16.428, + 13.912, + 13.404, + 13.92, + 14.15, + 14.441, + 14.458, + 12.839, + 12.959, + 14.17, + 13.349, + 12.809, + 13.078, + 12.806, + 12.497, + 12.607, + 11.808, + 11.691, + 12.165, + 12.153, + 14.22, + 13.227, + 13.636, + 12.395, + 12.242, + 13.913, + 13.68, + 12.714, + 12.399, + 13.033, + 14.868, + 14.723, + 13.551, + 14.735, + 13.329, + 14.276, + 13.177, + 15.456, + 14.519, + 14.163, + 13.83, + 13.609, + 13.868, + 13.998, + 13.745, + 13.611, + 12.602, + 12.073, + 12.565, + 16.988, + 16.002, + 14.11, + 14.866, + 13.275, + 13.575, + 12.53, + 11.94, + 13.737, + 13.977, + 13.245 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "semble", + "index_ms": 297.3, + "p50_ms": 2.98, + "p90_ms": 4.02, + "p95_ms": 5.26, + "p99_ms": 5.49, + "latencies_ms": [ + 5.255, + 4.251, + 4.299, + 4.225, + 3.971, + 0.984, + 0.921, + 0.83, + 0.816, + 0.795, + 1.07, + 1.057, + 0.985, + 0.977, + 0.956, + 3.902, + 3.83, + 3.712, + 3.724, + 3.772, + 1.004, + 1.001, + 1.122, + 1.05, + 0.938, + 0.967, + 0.898, + 0.873, + 0.899, + 0.936, + 0.841, + 0.79, + 0.806, + 0.8, + 0.77, + 1.09, + 1.034, + 1.051, + 1.068, + 1.044, + 3.681, + 3.69, + 3.441, + 3.438, + 3.424, + 3.83, + 3.773, + 3.665, + 3.618, + 3.639, + 3.788, + 3.81, + 3.786, + 3.707, + 3.734, + 0.929, + 0.882, + 0.865, + 0.852, + 0.875, + 3.593, + 3.083, + 3.081, + 3.101, + 3.164, + 3.319, + 2.783, + 2.845, + 2.876, + 2.833, + 3.451, + 2.933, + 2.88, + 2.899, + 3.029, + 3.121, + 2.679, + 2.764, + 2.726, + 2.674, + 3.185, + 2.711, + 2.659, + 2.563, + 2.655, + 5.483, + 5.722, + 5.445, + 5.359, + 5.338, + 3.982, + 3.966, + 3.936, + 4.011, + 4.066, + 3.616, + 3.553, + 3.545, + 3.514, + 3.638 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "bm25", + "index_ms": 297.3, + "p50_ms": 2.26, + "p90_ms": 3.0, + "p95_ms": 4.09, + "p99_ms": 4.22, + "latencies_ms": [ + 4.217, + 3.074, + 2.995, + 2.927, + 3.075, + 0.697, + 0.644, + 0.629, + 0.635, + 0.615, + 0.722, + 0.705, + 0.681, + 0.673, + 0.667, + 2.849, + 2.861, + 2.79, + 2.773, + 2.888, + 0.772, + 0.712, + 0.688, + 0.668, + 0.65, + 0.72, + 0.694, + 0.689, + 0.668, + 0.673, + 0.648, + 0.642, + 0.641, + 0.623, + 0.628, + 0.816, + 0.795, + 0.792, + 0.791, + 0.797, + 2.885, + 2.894, + 2.875, + 2.864, + 2.946, + 2.787, + 2.705, + 2.719, + 2.765, + 2.773, + 3.021, + 2.957, + 2.889, + 2.844, + 2.878, + 0.787, + 0.796, + 0.778, + 0.742, + 0.767, + 2.538, + 2.444, + 2.425, + 2.467, + 2.447, + 2.116, + 2.112, + 2.154, + 2.144, + 2.177, + 2.213, + 2.187, + 2.249, + 2.205, + 2.167, + 2.306, + 2.358, + 2.379, + 2.269, + 2.303, + 2.211, + 2.169, + 2.185, + 2.143, + 2.194, + 4.173, + 4.091, + 4.244, + 4.124, + 4.123, + 3.011, + 2.916, + 2.873, + 2.996, + 2.953, + 2.76, + 2.737, + 2.787, + 2.763, + 2.95 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "colgrep", + "index_ms": 2891.6, + "p50_ms": 124.52, + "p90_ms": 145.97, + "p95_ms": 149.81, + "p99_ms": 149.88, + "latencies_ms": [ + 149.808, + 138.934, + 145.539, + 149.898, + 126.023, + 122.706, + 121.656, + 125.212, + 121.085, + 124.226, + 138.392, + 122.765, + 120.791, + 127.761, + 122.013, + 121.913, + 124.805, + 130.6, + 122.92, + 119.621 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.81, + "p90_ms": 17.26, + "p95_ms": 17.71, + "p99_ms": 18.72, + "latencies_ms": [ + 17.111, + 14.952, + 14.313, + 15.227, + 15.168, + 13.369, + 14.323, + 13.021, + 15.508, + 13.763, + 14.868, + 14.188, + 14.922, + 13.441, + 12.893, + 12.957, + 13.539, + 13.035, + 15.94, + 15.92, + 15.873, + 16.836, + 17.693, + 16.083, + 19.202, + 18.388, + 17.498, + 16.529, + 15.765, + 15.301, + 14.666, + 15.822, + 15.287, + 16.253, + 15.928, + 14.412, + 15.074, + 16.534, + 15.958, + 17.032, + 16.451, + 17.955, + 16.501, + 17.131, + 15.723, + 17.089, + 15.266, + 17.242, + 16.771, + 17.215, + 15.701, + 15.906, + 15.639, + 16.952, + 15.542, + 15.083, + 15.92, + 17.444, + 15.795, + 16.513 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "semble", + "index_ms": 1090.8, + "p50_ms": 0.81, + "p90_ms": 1.21, + "p95_ms": 2.14, + "p99_ms": 8.39, + "latencies_ms": [ + 1.819, + 1.317, + 1.229, + 1.344, + 1.316, + 9.311, + 8.385, + 8.289, + 8.376, + 8.285, + 0.891, + 0.715, + 0.684, + 0.67, + 0.671, + 0.981, + 0.966, + 0.952, + 0.92, + 0.889, + 0.929, + 0.889, + 0.873, + 0.882, + 0.934, + 1.193, + 1.099, + 1.105, + 1.048, + 1.048, + 0.733, + 0.669, + 0.653, + 0.648, + 0.654, + 0.693, + 0.66, + 0.664, + 0.641, + 0.709, + 0.969, + 0.922, + 0.881, + 0.87, + 0.839, + 1.205, + 1.131, + 1.13, + 1.127, + 1.193, + 0.865, + 0.802, + 0.788, + 0.779, + 0.769, + 0.807, + 0.775, + 0.772, + 0.783, + 0.773, + 0.905, + 0.897, + 0.899, + 0.884, + 0.877, + 0.705, + 0.654, + 0.648, + 0.662, + 0.66, + 0.772, + 0.733, + 0.718, + 0.713, + 0.715, + 0.809, + 0.743, + 0.735, + 0.733, + 0.728, + 0.694, + 0.674, + 0.65, + 0.641, + 0.672, + 1.154, + 1.078, + 1.109, + 1.091, + 1.036, + 0.876, + 0.817, + 0.792, + 0.796, + 0.804, + 0.849, + 0.789, + 0.764, + 0.758, + 0.8 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "bm25", + "index_ms": 1090.8, + "p50_ms": 0.65, + "p90_ms": 0.99, + "p95_ms": 1.49, + "p99_ms": 7.36, + "latencies_ms": [ + 1.194, + 1.071, + 1.117, + 1.075, + 1.014, + 7.541, + 7.297, + 7.248, + 7.194, + 7.361, + 0.632, + 0.534, + 0.522, + 0.517, + 0.518, + 0.795, + 0.738, + 0.747, + 0.719, + 0.713, + 0.764, + 0.72, + 0.695, + 0.687, + 0.678, + 0.828, + 0.815, + 0.801, + 0.802, + 0.79, + 0.586, + 0.552, + 0.557, + 0.573, + 0.626, + 0.6, + 0.558, + 0.54, + 0.542, + 0.535, + 0.755, + 0.736, + 0.727, + 0.714, + 0.715, + 0.948, + 0.94, + 0.937, + 0.99, + 0.925, + 0.61, + 0.601, + 0.591, + 0.564, + 0.571, + 0.614, + 0.564, + 0.56, + 0.58, + 0.554, + 0.766, + 0.73, + 0.741, + 0.777, + 0.745, + 0.511, + 0.48, + 0.476, + 0.472, + 0.468, + 0.65, + 0.624, + 0.627, + 0.63, + 0.628, + 0.693, + 0.658, + 0.65, + 0.639, + 0.635, + 0.569, + 0.502, + 0.489, + 0.497, + 0.494, + 0.955, + 0.911, + 0.922, + 0.906, + 0.891, + 0.678, + 0.642, + 0.641, + 0.631, + 0.682, + 0.671, + 0.627, + 0.616, + 0.619, + 0.615 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "colgrep", + "index_ms": 2858.3, + "p50_ms": 115.87, + "p90_ms": 124.51, + "p95_ms": 128.21, + "p99_ms": 130.7, + "latencies_ms": [ + 131.317, + 113.898, + 116.283, + 112.908, + 115.718, + 118.703, + 116.022, + 123.396, + 119.888, + 122.58, + 115.084, + 113.078, + 112.985, + 113.271, + 112.755, + 114.284, + 123.196, + 114.532, + 124.117, + 128.047 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.1, + "p90_ms": 16.97, + "p95_ms": 17.48, + "p99_ms": 19.3, + "latencies_ms": [ + 17.805, + 16.408, + 16.907, + 16.622, + 15.043, + 17.119, + 16.962, + 13.909, + 16.443, + 15.385, + 14.694, + 15.689, + 14.947, + 14.519, + 14.98, + 14.016, + 13.84, + 13.464, + 13.169, + 14.278, + 15.762, + 17.459, + 15.783, + 17.045, + 15.546, + 15.526, + 15.498, + 16.101, + 15.953, + 14.682, + 14.454, + 14.702, + 20.257, + 16.785, + 18.643, + 15.708, + 15.948, + 15.925, + 16.204, + 15.703, + 14.092, + 14.392, + 13.838, + 14.774, + 13.188, + 12.751, + 14.336, + 13.334, + 13.404, + 14.335, + 15.148, + 14.662, + 14.275, + 13.782, + 14.848, + 15.868, + 15.207, + 14.603, + 15.976, + 15.045 + ] + } + ] +} diff --git a/benchmarks/results/speed-b468bbf0a1a5.json b/benchmarks/results/speed-b468bbf0a1a5.json new file mode 100644 index 000000000..f69af3296 --- /dev/null +++ b/benchmarks/results/speed-b468bbf0a1a5.json @@ -0,0 +1,8384 @@ +{ + "repos": [ + "nvm", + "libuv", + "nlohmann-json", + "messagepack-csharp", + "phoenix", + "gin", + "aeson", + "gson", + "axios", + "ktor", + "telescope.nvim", + "monolog", + "flask", + "rack", + "axum", + "http4s", + "alamofire", + "trpc", + "zls" + ], + "summary": { + "semble": { + "avg_index_ms": 518.2, + "avg_p50_ms": 0.91, + "avg_p90_ms": 4.88, + "avg_p95_ms": 6.26, + "avg_p99_ms": 9.08 + }, + "bm25": { + "avg_index_ms": 46.6, + "avg_p50_ms": 0.17, + "avg_p90_ms": 0.47, + "avg_p95_ms": 0.65, + "avg_p99_ms": 1.28 + }, + "colgrep": { + "avg_index_ms": 5359.2, + "avg_p50_ms": 122.42, + "avg_p90_ms": 151.84, + "avg_p95_ms": 159.08, + "avg_p99_ms": 192.73 + }, + "ripgrep": { + "avg_index_ms": null, + "avg_p50_ms": 14.46, + "avg_p90_ms": 17.22, + "avg_p95_ms": 17.88, + "avg_p99_ms": 20.14 + }, + "coderankembed": { + "avg_index_ms": 115859.8, + "avg_p50_ms": 15.56, + "avg_p90_ms": 20.33, + "avg_p95_ms": 21.92, + "avg_p99_ms": 25.25 + } + }, + "results": [ + { + "repo": "nvm", + "language": "bash", + "tool": "colgrep", + "index_ms": 2864.5, + "p50_ms": 125.32, + "p90_ms": 136.08, + "p95_ms": 137.17, + "p99_ms": 152.52, + "latencies_ms": [ + 156.356, + 124.814, + 125.824, + 128.156, + 136.071, + 119.393, + 117.59, + 117.071, + 129.408, + 136.157, + 132.054, + 120.523, + 118.213, + 119.091, + 124.128, + 121.806, + 128.689, + 121.772, + 128.562, + 127.593 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 17.01, + "p90_ms": 18.35, + "p95_ms": 19.14, + "p99_ms": 19.68, + "latencies_ms": [ + 20.133, + 18.881, + 19.293, + 17.718, + 16.033, + 17.088, + 17.672, + 17.794, + 19.364, + 18.422, + 18.338, + 15.707, + 17.025, + 16.96, + 17.43, + 17.093, + 16.641, + 17.0, + 17.249, + 17.332, + 17.762, + 17.746, + 15.755, + 16.515, + 16.552, + 17.679, + 17.31, + 17.674, + 18.195, + 16.534, + 16.737, + 16.069, + 16.381, + 16.867, + 18.239, + 16.983, + 17.39, + 16.366, + 17.438, + 17.049, + 15.709, + 16.72, + 19.133, + 17.615, + 16.411, + 17.762, + 17.677, + 15.82, + 15.909, + 15.367, + 15.458, + 16.434, + 15.731, + 17.484, + 16.647, + 16.48, + 15.775, + 16.286, + 16.003, + 16.163 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "colgrep", + "index_ms": 10659.8, + "p50_ms": 141.26, + "p90_ms": 156.42, + "p95_ms": 160.01, + "p99_ms": 185.65, + "latencies_ms": [ + 192.063, + 137.887, + 138.543, + 138.895, + 140.792, + 146.394, + 158.325, + 146.928, + 156.211, + 148.719, + 151.694, + 146.5, + 141.338, + 136.899, + 136.619, + 136.213, + 140.979, + 138.011, + 153.58, + 141.19 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.8, + "p90_ms": 19.17, + "p95_ms": 19.86, + "p99_ms": 23.55, + "latencies_ms": [ + 27.01, + 18.649, + 17.981, + 17.998, + 15.8, + 15.753, + 17.28, + 18.818, + 18.631, + 15.528, + 17.988, + 16.523, + 21.15, + 19.114, + 18.388, + 16.78, + 19.693, + 19.917, + 19.86, + 19.818, + 17.75, + 16.826, + 16.758, + 16.164, + 16.256, + 15.776, + 16.68, + 15.456, + 16.153, + 16.586, + 14.827, + 17.861, + 17.676, + 17.19, + 16.948, + 17.422, + 17.54, + 16.498, + 14.521, + 15.04, + 15.796, + 15.786, + 16.21, + 17.455, + 16.992, + 17.343, + 16.759, + 16.404, + 17.698, + 17.025, + 15.268, + 15.969, + 16.859, + 15.926, + 17.193, + 15.422, + 15.26, + 14.901, + 15.203, + 16.784 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "colgrep", + "index_ms": 5204.4, + "p50_ms": 117.89, + "p90_ms": 132.46, + "p95_ms": 143.46, + "p99_ms": 189.14, + "latencies_ms": [ + 200.565, + 120.304, + 117.892, + 114.652, + 123.228, + 140.458, + 131.567, + 120.732, + 123.26, + 113.949, + 116.615, + 117.171, + 118.32, + 117.885, + 116.577, + 115.578, + 115.023, + 117.355, + 121.68, + 116.401 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.28, + "p90_ms": 15.58, + "p95_ms": 15.89, + "p99_ms": 17.68, + "latencies_ms": [ + 18.216, + 14.956, + 15.888, + 14.639, + 15.564, + 17.303, + 15.731, + 14.495, + 13.288, + 12.382, + 13.409, + 14.862, + 13.442, + 14.313, + 14.218, + 12.715, + 12.037, + 12.931, + 15.019, + 14.809, + 13.969, + 11.693, + 13.387, + 13.548, + 14.915, + 14.97, + 13.641, + 11.947, + 11.265, + 12.512, + 12.675, + 13.061, + 12.567, + 11.23, + 10.358, + 13.613, + 12.73, + 14.652, + 15.696, + 15.9, + 13.446, + 12.371, + 12.227, + 12.205, + 12.257, + 13.036, + 13.268, + 13.508, + 11.941, + 11.403, + 12.035, + 13.543, + 11.78, + 12.939, + 11.357, + 12.651, + 14.078, + 13.006, + 11.939, + 11.49 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "colgrep", + "index_ms": 5191.0, + "p50_ms": 125.68, + "p90_ms": 142.08, + "p95_ms": 146.85, + "p99_ms": 156.91, + "latencies_ms": [ + 141.619, + 159.427, + 125.806, + 129.463, + 127.668, + 130.313, + 125.558, + 123.38, + 122.463, + 124.638, + 121.233, + 122.778, + 119.576, + 120.32, + 118.497, + 127.761, + 122.103, + 130.618, + 146.187, + 137.91 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.74, + "p90_ms": 17.76, + "p95_ms": 18.45, + "p99_ms": 20.93, + "latencies_ms": [ + 18.963, + 16.823, + 17.214, + 16.259, + 15.438, + 16.712, + 15.274, + 16.299, + 17.003, + 15.11, + 16.095, + 14.992, + 15.835, + 17.322, + 15.086, + 16.011, + 15.743, + 15.249, + 15.715, + 22.161, + 20.067, + 17.121, + 17.999, + 16.264, + 18.422, + 17.731, + 17.309, + 15.303, + 15.641, + 16.934, + 15.502, + 15.117, + 15.611, + 15.17, + 15.411, + 15.692, + 15.674, + 15.735, + 16.245, + 17.984, + 17.018, + 14.684, + 14.943, + 16.141, + 15.767, + 16.724, + 16.876, + 15.277, + 15.47, + 15.01, + 14.755, + 17.105, + 15.447, + 14.94, + 14.97, + 14.628, + 14.322, + 14.661, + 16.213, + 14.853 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "colgrep", + "index_ms": 19008.0, + "p50_ms": 157.38, + "p90_ms": 184.89, + "p95_ms": 198.93, + "p99_ms": 256.44, + "latencies_ms": [ + 270.816, + 173.743, + 183.38, + 190.945, + 177.629, + 159.606, + 155.944, + 157.38, + 159.023, + 158.657, + 156.79, + 160.48, + 155.212, + 154.499, + 154.058, + 152.181, + 155.184, + 156.612, + 156.458 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 12.91, + "p90_ms": 14.26, + "p95_ms": 14.76, + "p99_ms": 18.29, + "latencies_ms": [ + 21.511, + 13.327, + 13.77, + 13.233, + 13.114, + 12.912, + 13.352, + 13.194, + 13.059, + 12.979, + 12.122, + 12.572, + 14.615, + 13.296, + 11.59, + 15.753, + 14.709, + 13.515, + 11.463, + 12.509, + 12.895, + 11.97, + 12.904, + 14.192, + 12.694, + 12.376, + 11.399, + 12.431, + 12.679, + 12.649, + 12.945, + 13.48, + 13.436, + 11.849, + 13.125, + 12.664, + 12.71, + 11.588, + 11.233, + 13.24, + 11.474, + 14.373, + 14.03, + 14.96, + 13.468, + 11.952, + 12.719, + 12.698, + 12.499, + 12.174, + 12.473, + 13.045, + 13.309, + 12.92, + 13.192, + 11.866, + 11.219 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "colgrep", + "index_ms": 5442.0, + "p50_ms": 127.87, + "p90_ms": 150.74, + "p95_ms": 156.5, + "p99_ms": 178.09, + "latencies_ms": [ + 183.493, + 139.902, + 150.252, + 133.797, + 122.263, + 121.553, + 119.226, + 119.393, + 144.924, + 128.81, + 127.117, + 123.251, + 128.019, + 132.918, + 127.728, + 121.637, + 155.079, + 129.262, + 125.437, + 120.764 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.67, + "p90_ms": 15.34, + "p95_ms": 15.63, + "p99_ms": 15.95, + "latencies_ms": [ + 15.345, + 12.214, + 13.102, + 13.799, + 13.861, + 15.017, + 13.693, + 15.205, + 12.43, + 11.98, + 13.477, + 14.115, + 11.881, + 13.545, + 15.778, + 14.154, + 14.197, + 14.39, + 13.277, + 13.658, + 13.394, + 12.123, + 12.835, + 12.28, + 14.576, + 12.022, + 11.584, + 12.761, + 12.673, + 13.793, + 11.863, + 12.704, + 12.934, + 15.07, + 16.195, + 14.14, + 14.954, + 12.46, + 14.918, + 11.872, + 13.681, + 13.402, + 13.277, + 14.317, + 14.5, + 12.428, + 12.075, + 13.357, + 14.425, + 13.421, + 14.19, + 15.748, + 13.081, + 15.339, + 14.212, + 12.95, + 13.847, + 15.355, + 15.624, + 14.939 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "colgrep", + "index_ms": 3558.3, + "p50_ms": 123.26, + "p90_ms": 132.4, + "p95_ms": 138.06, + "p99_ms": 159.88, + "latencies_ms": [ + 165.339, + 136.619, + 130.425, + 121.88, + 124.577, + 122.537, + 122.504, + 128.48, + 126.739, + 116.912, + 117.892, + 123.986, + 124.393, + 127.468, + 120.757, + 131.934, + 119.273, + 118.897, + 115.166, + 115.937 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 12.05, + "p90_ms": 14.61, + "p95_ms": 16.54, + "p99_ms": 19.1, + "latencies_ms": [ + 13.979, + 12.14, + 12.59, + 11.546, + 11.391, + 13.283, + 12.147, + 11.632, + 11.213, + 13.442, + 18.902, + 17.056, + 16.51, + 14.249, + 14.446, + 12.288, + 11.504, + 11.362, + 10.249, + 11.337, + 11.071, + 12.124, + 13.161, + 11.752, + 11.751, + 12.562, + 11.771, + 11.506, + 11.322, + 11.28, + 10.834, + 14.162, + 19.395, + 14.376, + 15.539, + 14.511, + 15.286, + 13.914, + 13.129, + 13.742, + 14.531, + 13.583, + 10.039, + 10.83, + 11.354, + 10.642, + 13.269, + 11.942, + 11.735, + 11.98, + 12.232, + 11.188, + 10.986, + 11.794, + 11.835, + 11.222, + 13.014, + 12.969, + 11.974, + 11.342 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "colgrep", + "index_ms": 9427.5, + "p50_ms": 125.15, + "p90_ms": 134.55, + "p95_ms": 136.16, + "p99_ms": 147.03, + "latencies_ms": [ + 149.746, + 132.019, + 135.443, + 134.452, + 127.785, + 125.943, + 122.619, + 123.23, + 122.195, + 122.427, + 122.494, + 124.201, + 124.354, + 122.154, + 123.718, + 123.394, + 130.246, + 129.158, + 129.502, + 127.046 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.66, + "p90_ms": 18.23, + "p95_ms": 18.71, + "p99_ms": 20.7, + "latencies_ms": [ + 23.281, + 16.524, + 15.735, + 17.149, + 13.894, + 16.128, + 15.772, + 18.908, + 17.741, + 15.792, + 16.257, + 17.568, + 15.747, + 16.456, + 15.615, + 16.345, + 17.282, + 17.063, + 14.834, + 14.089, + 13.394, + 14.917, + 17.114, + 15.429, + 16.458, + 16.845, + 15.471, + 14.985, + 14.953, + 15.785, + 15.977, + 15.554, + 15.965, + 17.219, + 17.73, + 18.23, + 17.024, + 16.807, + 16.63, + 18.701, + 17.43, + 17.038, + 15.544, + 16.748, + 15.61, + 16.773, + 18.226, + 16.844, + 17.965, + 18.793, + 17.981, + 17.481, + 16.489, + 17.499, + 18.524, + 16.155, + 17.603, + 16.698, + 16.341, + 17.677 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "colgrep", + "index_ms": 1837.6, + "p50_ms": 112.68, + "p90_ms": 121.2, + "p95_ms": 125.98, + "p99_ms": 133.79, + "latencies_ms": [ + 135.739, + 108.823, + 110.647, + 109.423, + 112.697, + 109.626, + 117.432, + 114.531, + 119.737, + 118.358, + 120.732, + 125.465, + 116.396, + 113.607, + 110.165, + 110.96, + 109.932, + 110.34, + 112.659, + 110.925 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.83, + "p90_ms": 16.44, + "p95_ms": 16.61, + "p99_ms": 26.14, + "latencies_ms": [ + 15.173, + 15.426, + 12.159, + 13.394, + 13.374, + 12.205, + 12.352, + 14.332, + 14.163, + 16.436, + 14.431, + 14.115, + 13.829, + 14.579, + 15.322, + 14.62, + 16.292, + 14.353, + 16.617, + 14.646, + 14.772, + 14.772, + 15.568, + 15.835, + 14.772, + 15.892, + 14.209, + 13.142, + 16.29, + 15.178, + 16.608, + 16.576, + 16.255, + 16.304, + 14.335, + 14.663, + 15.73, + 15.384, + 17.494, + 14.895, + 16.231, + 16.479, + 38.579, + 15.268, + 15.285, + 15.872, + 15.172, + 15.408, + 14.576, + 15.959, + 14.587, + 15.241, + 14.16, + 15.062, + 13.492, + 12.444, + 13.658, + 12.203, + 13.046, + 14.677 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "colgrep", + "index_ms": 4203.9, + "p50_ms": 132.17, + "p90_ms": 176.25, + "p95_ms": 185.71, + "p99_ms": 193.1, + "latencies_ms": [ + 194.947, + 175.255, + 185.222, + 146.384, + 140.544, + 145.885, + 139.338, + 143.563, + 138.686, + 118.646, + 117.767, + 119.017, + 116.176, + 120.418, + 117.155, + 116.15, + 130.298, + 134.045, + 126.093, + 130.171 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.18, + "p90_ms": 16.62, + "p95_ms": 17.85, + "p99_ms": 19.86, + "latencies_ms": [ + 15.297, + 14.958, + 14.298, + 15.825, + 16.618, + 14.295, + 14.531, + 14.158, + 14.176, + 14.759, + 13.962, + 16.115, + 20.619, + 19.328, + 17.829, + 18.264, + 16.13, + 15.878, + 16.542, + 16.676, + 14.897, + 16.03, + 15.424, + 15.194, + 16.657, + 15.782, + 14.178, + 13.404, + 14.485, + 13.313, + 13.58, + 13.461, + 15.052, + 12.865, + 12.555, + 12.388, + 13.029, + 12.541, + 12.954, + 12.09, + 12.15, + 13.447, + 13.308, + 12.022, + 12.828, + 11.737, + 13.317, + 12.221, + 15.606, + 14.835, + 14.41, + 13.408, + 13.819, + 13.734, + 12.055, + 12.88, + 12.431, + 12.948, + 14.175, + 15.08 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "colgrep", + "index_ms": 4045.7, + "p50_ms": 124.38, + "p90_ms": 144.56, + "p95_ms": 146.8, + "p99_ms": 157.05, + "latencies_ms": [ + 159.617, + 128.982, + 142.265, + 120.829, + 128.587, + 128.187, + 121.303, + 146.125, + 134.464, + 144.387, + 124.982, + 114.622, + 114.102, + 123.771, + 115.702, + 112.531, + 115.236, + 123.758, + 134.306, + 123.22 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.67, + "p90_ms": 15.63, + "p95_ms": 16.39, + "p99_ms": 17.69, + "latencies_ms": [ + 15.09, + 16.776, + 15.302, + 15.253, + 14.483, + 16.366, + 13.379, + 13.013, + 13.124, + 12.289, + 12.85, + 16.111, + 13.831, + 13.507, + 13.52, + 13.644, + 14.723, + 13.972, + 15.541, + 14.324, + 13.978, + 14.059, + 15.165, + 14.598, + 13.725, + 13.716, + 13.534, + 12.699, + 17.995, + 17.479, + 15.277, + 15.42, + 15.135, + 15.062, + 13.348, + 15.612, + 15.812, + 13.87, + 13.753, + 12.975, + 12.302, + 13.704, + 12.805, + 11.589, + 11.303, + 11.146, + 11.477, + 11.409, + 10.793, + 13.223, + 12.39, + 13.72, + 12.372, + 12.245, + 12.447, + 11.773, + 11.782, + 11.519, + 11.357, + 13.237 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "colgrep", + "index_ms": 4266.4, + "p50_ms": 122.98, + "p90_ms": 149.69, + "p95_ms": 156.68, + "p99_ms": 159.62, + "latencies_ms": [ + 139.566, + 117.528, + 117.928, + 118.893, + 125.379, + 117.623, + 114.503, + 120.576, + 117.532, + 113.903, + 115.237, + 116.32, + 156.485, + 146.1, + 136.252, + 134.135, + 160.353, + 148.938, + 139.656, + 129.317 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 15.14, + "p90_ms": 17.44, + "p95_ms": 20.16, + "p99_ms": 22.58, + "latencies_ms": [ + 17.142, + 16.954, + 16.647, + 16.945, + 16.883, + 21.99, + 23.437, + 20.146, + 17.227, + 18.693, + 16.032, + 15.788, + 17.801, + 15.089, + 16.413, + 17.405, + 15.113, + 14.787, + 15.725, + 15.789, + 16.15, + 16.525, + 14.939, + 14.723, + 14.693, + 16.359, + 14.511, + 15.832, + 16.309, + 16.069, + 14.639, + 14.929, + 14.145, + 14.137, + 16.02, + 15.074, + 14.03, + 12.994, + 14.679, + 15.083, + 13.684, + 15.213, + 20.379, + 15.663, + 15.779, + 15.561, + 14.458, + 15.174, + 13.384, + 12.362, + 13.446, + 12.495, + 11.522, + 13.242, + 12.963, + 11.518, + 12.948, + 13.174, + 13.131, + 13.557 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "colgrep", + "index_ms": 2394.2, + "p50_ms": 112.08, + "p90_ms": 122.66, + "p95_ms": 125.48, + "p99_ms": 146.06, + "latencies_ms": [ + 151.205, + 111.255, + 109.941, + 109.401, + 110.037, + 109.345, + 122.66, + 116.407, + 115.891, + 120.353, + 112.105, + 108.931, + 111.086, + 125.479, + 112.263, + 111.106, + 111.84, + 110.048, + 112.258, + 113.013, + 112.077 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.06, + "p90_ms": 14.0, + "p95_ms": 14.23, + "p99_ms": 17.18, + "latencies_ms": [ + 14.214, + 13.9, + 13.55, + 11.764, + 13.065, + 12.912, + 13.986, + 13.955, + 13.587, + 12.792, + 13.276, + 13.057, + 13.244, + 13.226, + 13.103, + 13.464, + 12.966, + 11.904, + 11.618, + 12.346, + 13.483, + 13.368, + 12.702, + 11.834, + 11.632, + 12.357, + 11.849, + 11.897, + 13.147, + 12.474, + 12.51, + 12.121, + 12.752, + 12.99, + 12.827, + 14.279, + 12.609, + 13.13, + 12.257, + 13.116, + 14.073, + 14.231, + 13.83, + 13.741, + 13.998, + 12.915, + 11.682, + 13.227, + 11.315, + 12.74, + 11.302, + 12.662, + 13.493, + 12.921, + 13.213, + 13.177, + 21.919, + 13.174, + 13.12, + 12.857, + 12.841, + 12.618, + 14.256 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "colgrep", + "index_ms": 2904.7, + "p50_ms": 112.74, + "p90_ms": 122.19, + "p95_ms": 126.52, + "p99_ms": 138.12, + "latencies_ms": [ + 141.023, + 111.288, + 111.306, + 111.573, + 113.376, + 113.491, + 111.053, + 114.051, + 120.5, + 111.053, + 121.428, + 117.625, + 123.961, + 114.854, + 109.797, + 112.108, + 111.867, + 109.906 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.46, + "p90_ms": 15.04, + "p95_ms": 15.87, + "p99_ms": 16.2, + "latencies_ms": [ + 14.789, + 13.641, + 13.237, + 14.065, + 12.075, + 13.956, + 13.559, + 13.942, + 13.0, + 13.411, + 13.769, + 13.199, + 14.207, + 12.076, + 11.787, + 13.877, + 11.839, + 11.47, + 12.513, + 11.281, + 12.486, + 12.864, + 15.006, + 14.91, + 13.006, + 11.149, + 11.996, + 13.272, + 11.846, + 14.011, + 14.214, + 13.763, + 13.657, + 12.712, + 11.723, + 11.543, + 12.396, + 12.964, + 11.749, + 11.689, + 12.812, + 14.057, + 13.722, + 16.326, + 15.057, + 15.995, + 15.796, + 14.94, + 15.766, + 14.565, + 16.085, + 13.625, + 13.369, + 13.501 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "colgrep", + "index_ms": 6670.6, + "p50_ms": 122.0, + "p90_ms": 130.4, + "p95_ms": 132.53, + "p99_ms": 143.66, + "latencies_ms": [ + 146.439, + 120.878, + 118.29, + 119.267, + 130.246, + 123.088, + 123.948, + 122.381, + 131.793, + 121.014, + 120.122, + 125.845, + 124.269, + 121.611, + 123.189, + 121.147, + 120.979, + 121.057, + 125.985, + 121.454 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.83, + "p90_ms": 16.83, + "p95_ms": 17.29, + "p99_ms": 17.76, + "latencies_ms": [ + 16.861, + 17.555, + 17.65, + 17.281, + 16.801, + 15.395, + 16.364, + 15.447, + 16.075, + 16.158, + 15.068, + 16.783, + 14.72, + 14.022, + 15.52, + 14.752, + 15.851, + 15.21, + 14.008, + 16.83, + 16.087, + 17.267, + 17.916, + 16.147, + 15.082, + 14.78, + 15.215, + 14.642, + 15.277, + 12.917, + 14.212, + 14.137, + 13.676, + 14.703, + 13.561, + 13.604, + 14.208, + 15.783, + 14.524, + 14.008, + 13.929, + 15.158, + 14.367, + 14.927, + 13.375, + 14.454, + 12.863, + 13.875, + 14.822, + 12.366, + 12.864, + 14.732, + 15.246, + 12.25, + 12.414, + 14.845, + 15.239, + 14.915, + 13.283, + 13.96 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "colgrep", + "index_ms": 4386.3, + "p50_ms": 123.86, + "p90_ms": 133.23, + "p95_ms": 136.36, + "p99_ms": 163.9, + "latencies_ms": [ + 170.786, + 122.615, + 122.3, + 122.422, + 124.33, + 123.899, + 125.751, + 124.706, + 130.17, + 133.089, + 134.545, + 131.686, + 132.605, + 120.439, + 120.365, + 121.446, + 123.179, + 119.542, + 120.928, + 123.817 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 16.18, + "p90_ms": 17.33, + "p95_ms": 17.72, + "p99_ms": 18.3, + "latencies_ms": [ + 16.223, + 15.989, + 16.597, + 16.141, + 17.077, + 17.956, + 17.321, + 15.797, + 16.88, + 16.709, + 16.457, + 16.87, + 16.995, + 15.963, + 17.962, + 16.977, + 16.404, + 14.785, + 16.74, + 15.842, + 15.469, + 14.701, + 14.051, + 16.132, + 14.586, + 15.911, + 17.055, + 15.668, + 16.804, + 17.482, + 15.942, + 17.21, + 14.661, + 15.553, + 15.916, + 17.292, + 18.783, + 16.483, + 17.239, + 15.32, + 15.697, + 16.288, + 16.575, + 17.379, + 17.281, + 15.499, + 17.704, + 16.936, + 15.716, + 16.767, + 15.327, + 15.551, + 14.399, + 15.425, + 14.338, + 16.992, + 13.758, + 14.759, + 14.441, + 13.511 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "colgrep", + "index_ms": 3964.2, + "p50_ms": 116.59, + "p90_ms": 126.12, + "p95_ms": 161.57, + "p99_ms": 419.76, + "latencies_ms": [ + 144.58, + 484.304, + 117.563, + 116.703, + 120.372, + 116.479, + 123.743, + 120.036, + 124.069, + 123.128, + 113.509, + 116.749, + 115.459, + 115.93, + 114.959, + 113.668, + 112.401, + 115.047, + 113.465, + 116.117 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.41, + "p90_ms": 15.29, + "p95_ms": 15.69, + "p99_ms": 16.88, + "latencies_ms": [ + 15.274, + 15.639, + 11.758, + 13.101, + 12.937, + 14.048, + 14.119, + 15.44, + 16.778, + 16.618, + 15.57, + 13.857, + 13.122, + 12.906, + 13.399, + 13.072, + 13.412, + 14.375, + 14.241, + 14.884, + 12.068, + 12.999, + 13.154, + 13.686, + 14.162, + 13.445, + 13.578, + 15.089, + 17.024, + 15.259, + 14.241, + 13.434, + 13.176, + 14.302, + 12.828, + 13.404, + 12.706, + 13.578, + 13.982, + 13.082, + 12.748, + 13.436, + 11.107, + 12.044, + 12.271, + 13.123, + 13.267, + 12.326, + 13.862, + 12.187, + 13.724, + 13.193, + 15.22, + 12.412, + 12.12, + 10.865, + 12.116, + 11.904, + 13.384, + 13.448 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "colgrep", + "index_ms": 2900.9, + "p50_ms": 119.96, + "p90_ms": 122.2, + "p95_ms": 124.08, + "p99_ms": 143.27, + "latencies_ms": [ + 148.072, + 118.043, + 119.153, + 117.613, + 119.546, + 122.817, + 118.502, + 121.122, + 120.414, + 119.865, + 119.771, + 120.868, + 119.773, + 120.045, + 119.072, + 122.132, + 118.668, + 122.049, + 121.418, + 121.648 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 14.87, + "p90_ms": 16.43, + "p95_ms": 16.82, + "p99_ms": 17.1, + "latencies_ms": [ + 16.956, + 16.047, + 13.635, + 13.249, + 13.377, + 16.058, + 14.963, + 13.547, + 15.453, + 14.335, + 16.493, + 12.949, + 13.626, + 13.989, + 15.072, + 15.326, + 14.985, + 17.315, + 16.421, + 14.121, + 14.332, + 14.55, + 16.813, + 14.989, + 13.497, + 14.018, + 14.621, + 15.758, + 15.053, + 14.762, + 14.899, + 14.849, + 15.456, + 15.132, + 16.912, + 16.38, + 14.908, + 16.032, + 15.105, + 14.026, + 12.808, + 12.397, + 13.078, + 14.291, + 16.094, + 13.815, + 14.444, + 13.379, + 15.257, + 13.863, + 13.534, + 14.748, + 14.732, + 16.509, + 15.111, + 15.916, + 14.698, + 15.261, + 14.646, + 15.253 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "colgrep", + "index_ms": 2895.3, + "p50_ms": 115.94, + "p90_ms": 121.12, + "p95_ms": 124.8, + "p99_ms": 147.93, + "latencies_ms": [ + 153.709, + 117.919, + 114.798, + 115.74, + 116.142, + 123.274, + 120.879, + 116.171, + 117.405, + 116.233, + 114.42, + 115.493, + 118.269, + 114.297, + 117.271, + 114.364, + 114.123, + 114.315, + 115.476, + 114.455 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "ripgrep", + "index_ms": null, + "p50_ms": 13.1, + "p90_ms": 14.45, + "p95_ms": 14.78, + "p99_ms": 15.92, + "latencies_ms": [ + 17.203, + 13.248, + 12.689, + 14.315, + 14.325, + 14.155, + 13.228, + 13.006, + 12.944, + 13.725, + 14.399, + 14.095, + 13.164, + 15.035, + 12.611, + 14.192, + 13.495, + 12.928, + 13.248, + 11.406, + 12.961, + 13.268, + 14.772, + 14.19, + 13.312, + 14.703, + 12.552, + 13.569, + 14.214, + 13.038, + 12.254, + 12.405, + 12.372, + 14.433, + 13.113, + 14.123, + 12.525, + 12.04, + 12.401, + 12.004, + 14.247, + 12.19, + 12.262, + 13.096, + 13.874, + 12.298, + 14.216, + 12.225, + 11.2, + 12.111, + 12.063, + 12.278, + 12.986, + 14.615, + 14.125, + 12.645, + 12.0, + 12.031, + 12.214, + 14.907 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "semble", + "index_ms": 168.6, + "p50_ms": 0.58, + "p90_ms": 0.63, + "p95_ms": 0.66, + "p99_ms": 0.95, + "latencies_ms": [ + 0.811, + 0.551, + 0.53, + 0.532, + 0.53, + 0.553, + 0.542, + 0.524, + 0.523, + 0.516, + 0.628, + 0.6, + 0.593, + 0.588, + 0.594, + 0.677, + 0.631, + 0.584, + 0.578, + 0.575, + 0.593, + 0.583, + 0.551, + 0.582, + 0.565, + 0.524, + 0.484, + 0.477, + 0.476, + 0.477, + 0.566, + 0.557, + 0.551, + 0.615, + 0.58, + 0.62, + 0.594, + 0.588, + 0.599, + 0.557, + 0.612, + 0.583, + 0.578, + 0.592, + 0.588, + 0.524, + 0.493, + 0.491, + 0.48, + 0.486, + 0.641, + 0.628, + 0.612, + 0.622, + 0.604, + 0.56, + 0.541, + 0.542, + 0.539, + 0.537, + 0.582, + 0.583, + 0.598, + 0.599, + 0.609, + 0.604, + 0.573, + 0.589, + 0.664, + 0.643, + 0.58, + 0.539, + 0.539, + 0.541, + 0.54, + 0.614, + 0.599, + 0.601, + 0.629, + 0.948, + 0.57, + 0.49, + 0.475, + 0.477, + 0.473, + 0.614, + 0.575, + 0.575, + 0.587, + 0.576, + 0.585, + 0.962, + 0.65, + 0.809, + 0.629, + 0.602, + 0.559, + 0.542, + 0.566, + 0.554 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "semble", + "index_ms": 1128.5, + "p50_ms": 0.79, + "p90_ms": 1.17, + "p95_ms": 1.4, + "p99_ms": 1.5, + "latencies_ms": [ + 1.688, + 1.246, + 1.286, + 1.205, + 1.212, + 0.727, + 0.702, + 0.732, + 0.722, + 0.666, + 0.943, + 0.849, + 0.84, + 0.878, + 0.941, + 1.495, + 1.42, + 1.402, + 1.409, + 1.406, + 0.876, + 0.791, + 0.93, + 0.973, + 0.813, + 0.764, + 0.684, + 0.677, + 0.674, + 0.689, + 0.712, + 0.678, + 0.663, + 0.652, + 0.637, + 0.96, + 0.895, + 0.824, + 0.804, + 0.823, + 0.651, + 0.613, + 0.592, + 0.597, + 0.61, + 0.718, + 0.676, + 0.685, + 0.694, + 0.829, + 1.094, + 0.892, + 0.838, + 0.854, + 0.845, + 1.008, + 0.908, + 0.92, + 0.91, + 0.89, + 1.153, + 1.006, + 0.972, + 0.973, + 0.979, + 0.781, + 0.751, + 0.754, + 0.772, + 0.715, + 0.887, + 0.801, + 0.795, + 0.805, + 0.772, + 0.565, + 0.5, + 0.497, + 0.499, + 0.504, + 1.165, + 1.036, + 1.008, + 1.012, + 1.049, + 0.769, + 0.68, + 0.692, + 0.713, + 0.688, + 0.727, + 0.676, + 0.702, + 0.724, + 0.663, + 0.805, + 0.699, + 0.691, + 0.731, + 0.761 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "semble", + "index_ms": 706.1, + "p50_ms": 0.97, + "p90_ms": 4.4, + "p95_ms": 4.81, + "p99_ms": 5.1, + "latencies_ms": [ + 1.386, + 1.113, + 1.08, + 1.138, + 1.066, + 0.737, + 0.722, + 0.695, + 0.688, + 0.674, + 0.746, + 0.727, + 0.712, + 0.705, + 0.726, + 0.96, + 1.007, + 0.954, + 0.911, + 0.91, + 1.127, + 1.091, + 1.087, + 1.1, + 1.089, + 1.051, + 0.971, + 0.904, + 0.898, + 0.91, + 1.106, + 1.034, + 1.014, + 1.007, + 0.994, + 1.226, + 1.185, + 1.116, + 1.102, + 1.107, + 0.948, + 0.899, + 0.886, + 0.878, + 0.884, + 1.035, + 0.969, + 0.913, + 0.903, + 0.886, + 0.93, + 0.896, + 0.881, + 0.883, + 0.899, + 0.832, + 0.829, + 0.808, + 0.795, + 0.781, + 1.528, + 1.516, + 1.503, + 1.511, + 1.54, + 5.132, + 5.097, + 4.848, + 4.797, + 4.91, + 1.018, + 0.919, + 0.889, + 0.916, + 0.885, + 0.807, + 0.745, + 0.71, + 0.709, + 0.729, + 0.948, + 0.908, + 0.906, + 0.903, + 0.911, + 4.377, + 4.239, + 4.373, + 4.2, + 4.358, + 4.777, + 4.81, + 4.658, + 4.74, + 4.956, + 4.336, + 4.243, + 4.133, + 4.256, + 4.076 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "semble", + "index_ms": 856.1, + "p50_ms": 4.45, + "p90_ms": 6.3, + "p95_ms": 6.34, + "p99_ms": 6.62, + "latencies_ms": [ + 6.804, + 6.268, + 6.294, + 6.343, + 6.357, + 1.134, + 1.062, + 1.052, + 1.066, + 1.056, + 1.171, + 1.115, + 1.101, + 1.119, + 1.094, + 1.281, + 1.225, + 1.237, + 1.224, + 1.208, + 0.823, + 0.81, + 0.798, + 0.881, + 0.83, + 1.067, + 1.04, + 1.019, + 1.025, + 1.022, + 1.205, + 1.235, + 1.218, + 1.143, + 1.11, + 0.954, + 0.929, + 0.918, + 0.899, + 0.898, + 5.868, + 6.15, + 6.18, + 6.306, + 5.952, + 1.012, + 0.919, + 0.976, + 0.981, + 0.888, + 6.305, + 6.073, + 6.282, + 6.212, + 6.12, + 6.265, + 6.049, + 6.254, + 6.161, + 5.999, + 6.616, + 6.463, + 6.339, + 6.437, + 6.179, + 4.808, + 4.609, + 4.781, + 4.73, + 4.884, + 4.303, + 4.993, + 4.326, + 4.447, + 4.429, + 5.723, + 5.604, + 5.648, + 5.562, + 5.727, + 4.448, + 4.534, + 4.451, + 4.487, + 4.391, + 6.343, + 6.267, + 6.27, + 6.166, + 6.275, + 1.068, + 1.02, + 1.006, + 0.989, + 0.996, + 5.266, + 5.445, + 5.501, + 5.453, + 5.265 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "semble", + "index_ms": 469.5, + "p50_ms": 0.78, + "p90_ms": 4.61, + "p95_ms": 5.38, + "p99_ms": 6.32, + "latencies_ms": [ + 1.304, + 1.039, + 1.03, + 1.014, + 1.009, + 0.837, + 0.772, + 0.728, + 0.711, + 0.729, + 0.715, + 0.655, + 0.647, + 0.64, + 0.653, + 0.686, + 0.653, + 0.66, + 0.653, + 0.646, + 0.812, + 0.736, + 0.693, + 0.685, + 0.678, + 5.079, + 4.797, + 4.864, + 4.912, + 4.895, + 0.575, + 0.553, + 0.55, + 0.603, + 0.604, + 0.716, + 0.69, + 0.684, + 0.668, + 0.658, + 1.008, + 0.965, + 0.947, + 0.957, + 0.948, + 0.721, + 0.791, + 0.759, + 0.741, + 0.747, + 1.015, + 0.843, + 0.833, + 0.828, + 0.84, + 0.663, + 0.641, + 0.621, + 0.625, + 0.676, + 0.699, + 0.648, + 0.651, + 0.65, + 0.641, + 0.722, + 0.697, + 0.691, + 0.681, + 0.681, + 0.805, + 0.778, + 0.824, + 0.93, + 0.873, + 1.219, + 1.056, + 1.018, + 1.006, + 0.999, + 4.327, + 4.049, + 4.325, + 4.252, + 4.251, + 6.396, + 6.316, + 6.082, + 6.268, + 6.214, + 4.127, + 4.153, + 4.208, + 4.203, + 4.221 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "semble", + "index_ms": 486.7, + "p50_ms": 0.73, + "p90_ms": 3.3, + "p95_ms": 4.24, + "p99_ms": 4.57, + "latencies_ms": [ + 0.916, + 0.641, + 0.555, + 0.55, + 0.557, + 0.501, + 0.45, + 0.464, + 0.458, + 0.451, + 0.767, + 0.689, + 0.664, + 0.741, + 0.71, + 0.797, + 0.726, + 0.71, + 0.702, + 0.713, + 0.751, + 0.698, + 0.693, + 0.698, + 0.695, + 4.222, + 4.049, + 4.081, + 3.949, + 4.034, + 0.645, + 0.61, + 0.67, + 0.624, + 0.592, + 0.546, + 0.53, + 0.527, + 0.539, + 0.54, + 0.492, + 0.467, + 0.463, + 0.452, + 0.444, + 0.713, + 0.687, + 0.689, + 0.675, + 0.752, + 0.61, + 0.562, + 0.571, + 0.55, + 0.547, + 0.841, + 0.817, + 0.815, + 0.813, + 0.819, + 0.802, + 0.786, + 0.776, + 0.811, + 0.781, + 0.774, + 0.776, + 0.762, + 0.755, + 0.764, + 0.752, + 0.73, + 0.749, + 0.72, + 0.719, + 0.904, + 0.91, + 0.865, + 0.889, + 0.863, + 2.889, + 2.96, + 2.887, + 2.892, + 2.942, + 4.525, + 4.512, + 4.567, + 4.563, + 4.687, + 3.11, + 3.167, + 3.231, + 3.06, + 3.087, + 0.768, + 0.708, + 0.712, + 0.702, + 0.695 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "semble", + "index_ms": 380.3, + "p50_ms": 3.75, + "p90_ms": 6.74, + "p95_ms": 11.36, + "p99_ms": 20.4, + "latencies_ms": [ + 1.136, + 0.848, + 0.83, + 0.828, + 0.887, + 4.116, + 4.05, + 3.882, + 3.971, + 4.01, + 3.681, + 3.774, + 3.719, + 3.771, + 3.768, + 3.89, + 3.888, + 3.884, + 3.689, + 3.773, + 3.823, + 3.807, + 3.894, + 3.646, + 3.777, + 4.223, + 4.181, + 4.307, + 4.075, + 4.156, + 3.803, + 3.739, + 3.751, + 3.602, + 3.61, + 20.397, + 20.848, + 20.371, + 20.047, + 20.347, + 4.161, + 4.222, + 4.177, + 4.053, + 4.142, + 4.452, + 4.561, + 4.377, + 4.421, + 4.434, + 3.677, + 3.748, + 3.626, + 3.707, + 3.793, + 0.708, + 0.668, + 0.663, + 0.663, + 0.673, + 0.774, + 0.769, + 0.756, + 0.772, + 0.761, + 0.747, + 0.719, + 0.708, + 0.73, + 0.683, + 3.434, + 3.483, + 3.534, + 3.518, + 3.575, + 0.756, + 0.781, + 0.721, + 0.689, + 0.685, + 10.797, + 10.832, + 10.899, + 10.848, + 10.728, + 6.26, + 6.221, + 6.242, + 6.301, + 6.211, + 0.694, + 0.649, + 0.654, + 0.699, + 0.702, + 3.693, + 3.707, + 3.686, + 3.646, + 3.764 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "semble", + "index_ms": 1091.9, + "p50_ms": 1.52, + "p90_ms": 8.81, + "p95_ms": 9.63, + "p99_ms": 12.63, + "latencies_ms": [ + 13.227, + 12.419, + 12.621, + 12.39, + 12.227, + 1.705, + 1.58, + 1.598, + 1.634, + 1.632, + 1.381, + 1.363, + 1.356, + 1.324, + 1.312, + 8.395, + 8.305, + 8.092, + 8.438, + 8.46, + 6.956, + 6.896, + 6.651, + 6.945, + 6.786, + 1.519, + 1.47, + 1.524, + 1.473, + 2.122, + 1.451, + 1.418, + 1.312, + 1.349, + 1.342, + 1.264, + 1.22, + 1.223, + 1.183, + 1.227, + 1.177, + 1.101, + 1.094, + 1.133, + 1.124, + 0.91, + 0.859, + 0.851, + 0.88, + 0.889, + 1.245, + 1.194, + 1.191, + 1.208, + 1.201, + 1.26, + 1.229, + 1.235, + 1.197, + 1.179, + 1.407, + 1.367, + 1.353, + 1.38, + 1.351, + 1.448, + 1.39, + 1.429, + 1.447, + 1.477, + 1.6, + 1.599, + 1.527, + 1.496, + 1.497, + 1.781, + 1.726, + 1.716, + 1.656, + 1.77, + 8.783, + 8.505, + 8.449, + 8.163, + 8.442, + 6.772, + 6.887, + 7.042, + 6.802, + 6.852, + 9.07, + 9.49, + 9.3, + 9.236, + 9.142, + 6.976, + 7.111, + 6.965, + 6.946, + 7.407 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "semble", + "index_ms": 94.2, + "p50_ms": 0.64, + "p90_ms": 2.16, + "p95_ms": 2.26, + "p99_ms": 2.32, + "latencies_ms": [ + 0.89, + 0.647, + 0.636, + 0.621, + 0.617, + 0.572, + 0.51, + 0.48, + 0.465, + 0.461, + 0.581, + 0.56, + 0.557, + 0.572, + 0.557, + 0.635, + 0.603, + 0.601, + 0.615, + 0.6, + 2.303, + 2.102, + 2.18, + 2.143, + 2.143, + 1.707, + 1.608, + 1.585, + 1.618, + 1.686, + 2.319, + 2.262, + 2.266, + 2.318, + 2.314, + 0.545, + 0.525, + 0.594, + 0.565, + 0.507, + 0.565, + 0.548, + 0.529, + 0.533, + 0.532, + 0.67, + 0.653, + 0.638, + 0.635, + 0.645, + 0.586, + 0.566, + 0.571, + 0.569, + 0.604, + 0.585, + 0.548, + 0.551, + 0.551, + 0.559, + 0.678, + 0.672, + 0.672, + 0.672, + 0.669, + 0.624, + 0.613, + 0.602, + 0.61, + 0.601, + 0.577, + 0.552, + 0.545, + 0.558, + 0.553, + 0.691, + 0.663, + 0.678, + 0.659, + 0.646, + 0.74, + 0.715, + 0.702, + 0.702, + 0.702, + 2.157, + 2.014, + 2.07, + 2.121, + 2.115, + 2.09, + 2.001, + 2.081, + 2.085, + 2.089, + 2.143, + 2.196, + 2.187, + 2.159, + 2.201 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "semble", + "index_ms": 307.8, + "p50_ms": 1.09, + "p90_ms": 5.09, + "p95_ms": 5.96, + "p99_ms": 6.09, + "latencies_ms": [ + 6.371, + 5.946, + 6.086, + 5.754, + 6.039, + 1.073, + 1.005, + 0.993, + 1.004, + 1.002, + 0.849, + 0.833, + 0.819, + 0.82, + 0.853, + 1.036, + 0.997, + 1.024, + 1.03, + 1.016, + 0.955, + 0.933, + 0.953, + 0.939, + 0.982, + 0.921, + 0.878, + 0.864, + 0.861, + 0.872, + 0.784, + 0.782, + 0.772, + 0.766, + 0.763, + 0.663, + 0.643, + 0.716, + 0.683, + 0.661, + 3.167, + 3.18, + 3.222, + 3.115, + 3.211, + 0.883, + 0.865, + 0.882, + 0.828, + 0.823, + 0.997, + 0.962, + 0.977, + 0.962, + 0.972, + 4.931, + 4.852, + 4.937, + 5.013, + 4.923, + 4.634, + 4.891, + 4.611, + 4.745, + 4.51, + 3.436, + 3.372, + 3.313, + 3.355, + 3.393, + 4.597, + 4.744, + 4.773, + 4.679, + 4.422, + 3.371, + 3.489, + 3.53, + 3.475, + 3.535, + 5.931, + 5.96, + 5.885, + 5.959, + 6.034, + 1.285, + 1.326, + 1.181, + 1.13, + 1.101, + 0.944, + 0.934, + 0.869, + 0.862, + 0.854, + 3.61, + 3.613, + 4.208, + 3.779, + 3.72 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "semble", + "index_ms": 419.9, + "p50_ms": 0.81, + "p90_ms": 1.02, + "p95_ms": 1.59, + "p99_ms": 3.92, + "latencies_ms": [ + 1.482, + 0.97, + 0.866, + 0.858, + 0.848, + 0.886, + 0.83, + 0.772, + 0.755, + 0.736, + 0.849, + 0.796, + 0.79, + 0.784, + 0.802, + 0.859, + 0.832, + 0.871, + 0.892, + 0.872, + 1.025, + 0.895, + 0.879, + 0.883, + 0.919, + 0.967, + 1.036, + 1.019, + 0.937, + 0.833, + 0.759, + 0.69, + 0.676, + 0.688, + 0.692, + 0.852, + 0.786, + 0.777, + 0.797, + 0.832, + 0.807, + 0.856, + 0.882, + 0.802, + 0.758, + 3.972, + 3.915, + 3.762, + 3.702, + 3.733, + 0.882, + 0.806, + 0.79, + 0.772, + 0.806, + 0.808, + 0.785, + 0.782, + 0.852, + 0.789, + 0.823, + 0.788, + 0.777, + 0.779, + 0.769, + 0.946, + 0.914, + 0.911, + 0.895, + 0.901, + 0.83, + 0.795, + 0.799, + 0.787, + 0.786, + 0.778, + 0.753, + 0.74, + 0.732, + 0.744, + 0.821, + 0.946, + 1.087, + 1.023, + 0.916, + 0.847, + 0.762, + 0.742, + 0.733, + 0.752, + 0.77, + 0.712, + 0.789, + 0.778, + 0.708, + 0.847, + 0.818, + 0.813, + 0.802, + 0.817 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "semble", + "index_ms": 308.1, + "p50_ms": 0.96, + "p90_ms": 3.22, + "p95_ms": 3.32, + "p99_ms": 3.49, + "latencies_ms": [ + 1.106, + 0.82, + 0.793, + 0.79, + 0.792, + 0.97, + 0.939, + 0.927, + 0.927, + 0.907, + 0.913, + 0.826, + 0.792, + 0.779, + 0.768, + 0.734, + 0.697, + 0.687, + 0.706, + 0.689, + 3.59, + 3.322, + 3.476, + 3.493, + 3.402, + 3.09, + 3.094, + 3.004, + 3.079, + 3.089, + 3.24, + 3.208, + 3.232, + 3.223, + 3.19, + 0.962, + 0.897, + 0.893, + 0.871, + 0.961, + 1.091, + 1.031, + 1.039, + 1.057, + 1.038, + 0.976, + 0.947, + 0.931, + 0.937, + 1.014, + 1.003, + 0.918, + 0.906, + 0.907, + 0.927, + 0.935, + 0.939, + 0.91, + 0.89, + 0.901, + 0.848, + 0.77, + 0.759, + 0.766, + 0.773, + 0.849, + 0.822, + 0.818, + 0.823, + 0.809, + 0.999, + 0.953, + 0.983, + 0.942, + 0.926, + 0.982, + 0.973, + 0.959, + 0.958, + 0.972, + 1.101, + 1.081, + 1.137, + 1.135, + 1.13, + 0.992, + 0.963, + 0.956, + 0.968, + 0.972, + 3.35, + 3.173, + 3.254, + 3.282, + 3.102, + 3.005, + 3.035, + 3.065, + 3.008, + 3.117 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "semble", + "index_ms": 237.3, + "p50_ms": 0.47, + "p90_ms": 2.13, + "p95_ms": 3.23, + "p99_ms": 3.33, + "latencies_ms": [ + 0.78, + 0.505, + 0.471, + 0.488, + 0.469, + 0.498, + 0.453, + 0.443, + 0.426, + 0.435, + 0.408, + 0.373, + 0.372, + 0.371, + 0.382, + 0.422, + 0.383, + 0.394, + 0.381, + 0.384, + 0.515, + 0.422, + 0.395, + 0.405, + 0.393, + 0.39, + 0.37, + 0.374, + 0.37, + 0.363, + 0.468, + 0.44, + 0.453, + 0.435, + 0.429, + 0.416, + 0.396, + 0.391, + 0.399, + 0.398, + 0.409, + 0.403, + 0.393, + 0.447, + 0.456, + 0.511, + 0.467, + 0.457, + 0.459, + 0.459, + 0.671, + 0.637, + 0.636, + 0.639, + 0.646, + 0.751, + 0.698, + 0.685, + 0.679, + 0.68, + 0.767, + 0.727, + 0.733, + 0.742, + 0.744, + 0.572, + 0.546, + 0.546, + 0.556, + 0.542, + 0.754, + 0.702, + 0.698, + 0.699, + 0.705, + 3.163, + 3.115, + 3.182, + 3.238, + 3.057, + 2.142, + 2.059, + 2.116, + 2.085, + 2.047, + 3.333, + 3.388, + 3.258, + 3.267, + 3.267, + 0.405, + 0.377, + 0.372, + 0.384, + 0.393, + 0.357, + 0.329, + 0.341, + 0.326, + 0.316, + 0.717, + 0.662, + 0.658, + 0.676, + 0.685 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "semble", + "index_ms": 197.5, + "p50_ms": 0.59, + "p90_ms": 3.39, + "p95_ms": 4.44, + "p99_ms": 5.33, + "latencies_ms": [ + 0.797, + 0.605, + 0.579, + 0.552, + 0.53, + 0.651, + 0.607, + 0.61, + 0.609, + 0.624, + 0.466, + 0.437, + 0.43, + 0.423, + 0.43, + 0.518, + 0.483, + 0.5, + 0.486, + 0.498, + 3.116, + 2.995, + 2.935, + 2.955, + 2.889, + 5.349, + 5.199, + 5.306, + 5.17, + 5.331, + 3.388, + 3.472, + 3.55, + 3.417, + 3.515, + 0.606, + 0.566, + 0.559, + 0.603, + 0.566, + 0.541, + 0.512, + 0.503, + 0.503, + 0.515, + 0.604, + 0.576, + 0.572, + 0.574, + 0.571, + 0.534, + 0.502, + 0.498, + 0.496, + 0.502, + 0.609, + 0.636, + 0.607, + 0.597, + 0.565, + 0.559, + 0.535, + 0.532, + 0.538, + 0.548, + 0.513, + 0.506, + 0.509, + 0.498, + 0.506, + 0.591, + 0.573, + 0.552, + 0.553, + 0.566, + 0.648, + 0.622, + 0.606, + 0.608, + 0.619, + 2.795, + 2.845, + 2.764, + 2.667, + 2.759, + 2.704, + 2.664, + 2.639, + 2.608, + 2.715 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "semble", + "index_ms": 373.2, + "p50_ms": 0.88, + "p90_ms": 4.17, + "p95_ms": 4.42, + "p99_ms": 5.92, + "latencies_ms": [ + 1.071, + 0.811, + 0.794, + 0.784, + 0.787, + 0.879, + 0.816, + 0.804, + 0.888, + 0.885, + 5.976, + 5.843, + 5.919, + 5.651, + 5.715, + 0.703, + 0.661, + 0.646, + 0.653, + 0.663, + 0.658, + 0.64, + 0.613, + 0.628, + 0.615, + 0.728, + 0.683, + 0.691, + 0.68, + 0.718, + 4.163, + 4.302, + 4.189, + 4.243, + 4.255, + 1.063, + 0.994, + 1.007, + 0.989, + 0.978, + 0.803, + 0.798, + 0.769, + 0.766, + 0.772, + 0.828, + 0.792, + 0.798, + 0.782, + 0.783, + 0.82, + 0.811, + 0.796, + 0.781, + 0.775, + 3.346, + 3.323, + 3.29, + 3.251, + 3.353, + 4.357, + 4.13, + 4.149, + 4.095, + 4.171, + 3.773, + 3.641, + 3.681, + 3.71, + 3.694, + 0.822, + 0.767, + 0.752, + 0.743, + 0.746, + 0.909, + 0.871, + 0.91, + 0.861, + 0.809, + 0.638, + 0.616, + 0.617, + 0.627, + 0.624, + 3.329, + 3.283, + 3.199, + 3.243, + 3.237, + 3.332, + 3.476, + 3.38, + 3.348, + 3.429, + 3.569, + 3.548, + 3.573, + 3.448, + 3.499 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "semble", + "index_ms": 949.9, + "p50_ms": 4.65, + "p90_ms": 6.97, + "p95_ms": 7.43, + "p99_ms": 7.85, + "latencies_ms": [ + 6.482, + 5.693, + 5.801, + 5.674, + 6.293, + 1.125, + 1.035, + 1.032, + 1.055, + 1.001, + 5.28, + 5.295, + 5.427, + 5.414, + 5.257, + 4.892, + 4.736, + 4.848, + 4.684, + 4.795, + 0.962, + 0.92, + 0.977, + 0.968, + 0.939, + 0.929, + 0.902, + 0.861, + 0.831, + 0.82, + 8.031, + 7.843, + 7.749, + 7.586, + 7.518, + 1.089, + 1.113, + 1.037, + 0.985, + 1.015, + 0.966, + 0.933, + 0.925, + 0.928, + 0.942, + 1.013, + 0.97, + 0.972, + 0.97, + 1.037, + 5.388, + 5.577, + 5.401, + 5.364, + 5.553, + 1.076, + 0.986, + 1.022, + 0.933, + 0.916, + 6.944, + 7.021, + 6.895, + 6.962, + 6.741, + 0.995, + 0.984, + 0.976, + 1.09, + 1.005, + 7.428, + 7.18, + 6.935, + 7.221, + 7.091, + 0.785, + 0.755, + 0.79, + 0.756, + 0.736, + 5.632, + 5.48, + 5.447, + 5.547, + 5.639, + 4.715, + 4.879, + 4.786, + 4.771, + 4.693, + 4.618, + 4.558, + 4.588, + 4.62, + 4.497, + 5.262, + 5.145, + 5.281, + 5.328, + 5.232 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "semble", + "index_ms": 389.0, + "p50_ms": 0.94, + "p90_ms": 4.31, + "p95_ms": 5.56, + "p99_ms": 6.78, + "latencies_ms": [ + 1.244, + 0.924, + 0.926, + 0.899, + 0.895, + 0.994, + 1.019, + 0.971, + 0.947, + 0.94, + 0.788, + 0.78, + 0.771, + 0.719, + 0.72, + 0.792, + 0.751, + 0.732, + 0.773, + 0.815, + 0.793, + 0.72, + 0.715, + 0.731, + 0.748, + 0.561, + 0.523, + 0.531, + 0.51, + 0.51, + 0.616, + 0.562, + 0.559, + 0.568, + 0.607, + 0.865, + 0.797, + 0.809, + 0.807, + 0.767, + 0.815, + 0.777, + 0.78, + 0.771, + 0.762, + 0.831, + 0.784, + 0.849, + 0.811, + 0.782, + 5.367, + 5.321, + 5.414, + 5.505, + 5.499, + 1.049, + 0.889, + 0.888, + 0.888, + 0.888, + 6.836, + 6.682, + 6.629, + 6.784, + 6.59, + 4.198, + 4.027, + 4.166, + 4.054, + 4.118, + 3.334, + 3.265, + 3.318, + 3.354, + 3.215, + 3.17, + 3.197, + 3.033, + 3.121, + 3.14, + 3.752, + 3.751, + 3.759, + 3.672, + 3.785, + 1.018, + 0.99, + 0.926, + 0.948, + 0.957, + 1.038, + 1.012, + 1.003, + 1.0, + 0.993, + 3.801, + 3.802, + 3.931, + 3.792, + 3.864 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "semble", + "index_ms": 255.3, + "p50_ms": 2.74, + "p90_ms": 3.81, + "p95_ms": 4.41, + "p99_ms": 5.2, + "latencies_ms": [ + 4.239, + 3.783, + 3.842, + 3.738, + 3.824, + 0.822, + 0.787, + 0.773, + 0.853, + 0.817, + 0.942, + 0.913, + 0.914, + 0.923, + 0.945, + 3.638, + 3.468, + 3.57, + 3.621, + 3.491, + 0.972, + 0.96, + 0.918, + 0.915, + 0.908, + 0.905, + 0.918, + 0.884, + 0.847, + 0.868, + 0.768, + 0.746, + 0.74, + 0.75, + 0.734, + 1.014, + 0.982, + 0.998, + 1.023, + 0.967, + 3.382, + 3.345, + 3.326, + 3.214, + 3.27, + 3.566, + 3.634, + 3.507, + 3.428, + 3.424, + 3.672, + 3.552, + 4.379, + 3.674, + 3.484, + 0.886, + 0.871, + 0.849, + 0.852, + 0.855, + 2.995, + 2.891, + 2.993, + 3.027, + 2.877, + 2.697, + 2.724, + 2.701, + 2.65, + 2.694, + 2.899, + 2.764, + 2.811, + 2.848, + 2.867, + 2.573, + 2.611, + 2.57, + 2.58, + 2.491, + 2.545, + 2.586, + 2.51, + 2.561, + 2.517, + 5.127, + 5.034, + 5.26, + 5.198, + 5.049, + 3.775, + 3.827, + 3.749, + 3.791, + 3.812, + 3.405, + 3.492, + 3.501, + 3.404, + 3.499 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "semble", + "index_ms": 1025.2, + "p50_ms": 0.81, + "p90_ms": 1.2, + "p95_ms": 1.95, + "p99_ms": 8.17, + "latencies_ms": [ + 1.638, + 1.285, + 1.221, + 1.204, + 1.194, + 8.388, + 7.878, + 8.091, + 8.17, + 8.089, + 0.886, + 0.693, + 0.653, + 0.652, + 0.662, + 0.982, + 0.911, + 0.891, + 0.877, + 0.873, + 0.912, + 0.87, + 0.867, + 0.973, + 0.92, + 1.039, + 1.018, + 1.09, + 1.048, + 1.01, + 0.737, + 0.702, + 0.683, + 0.678, + 0.694, + 0.725, + 0.686, + 0.669, + 0.663, + 0.671, + 0.933, + 0.881, + 0.86, + 0.828, + 0.812, + 1.193, + 1.176, + 1.155, + 1.23, + 1.199, + 0.805, + 0.769, + 0.755, + 0.759, + 0.763, + 0.811, + 0.782, + 0.782, + 0.771, + 0.779, + 0.9, + 0.87, + 0.872, + 0.86, + 0.844, + 0.682, + 0.653, + 0.649, + 0.637, + 0.643, + 0.827, + 0.769, + 0.715, + 0.712, + 0.728, + 0.789, + 0.762, + 0.722, + 0.713, + 0.72, + 0.683, + 0.659, + 0.646, + 0.64, + 0.712, + 1.107, + 1.014, + 1.011, + 1.061, + 1.034, + 0.895, + 0.815, + 0.798, + 0.79, + 0.789, + 0.841, + 0.774, + 0.733, + 0.727, + 0.729 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "coderankembed", + "index_ms": 34214.6, + "p50_ms": 15.09, + "p90_ms": 15.33, + "p95_ms": 15.45, + "p99_ms": 15.71, + "latencies_ms": [ + 15.445, + 15.032, + 14.993, + 15.289, + 15.437, + 15.835, + 15.534, + 15.709, + 15.319, + 15.112, + 15.05, + 15.02, + 15.021, + 14.967, + 14.942, + 15.136, + 15.056, + 15.061, + 15.039, + 15.085, + 15.251, + 15.203, + 15.291, + 15.304, + 15.233, + 15.178, + 14.96, + 14.911, + 14.924, + 14.823, + 14.734, + 14.782, + 14.76, + 14.774, + 14.788, + 14.809, + 14.824, + 14.786, + 14.679, + 14.806, + 15.102, + 15.047, + 15.107, + 15.289, + 15.333, + 15.184, + 15.119, + 15.277, + 15.138, + 15.067, + 15.446, + 15.316, + 15.13, + 15.094, + 14.993, + 15.043, + 15.139, + 15.178, + 15.303, + 15.188, + 15.134, + 15.204, + 15.206, + 15.07, + 15.098, + 14.992, + 15.039, + 15.104, + 15.088, + 15.275, + 14.949, + 15.016, + 14.928, + 14.917, + 14.831, + 15.462, + 15.318, + 15.334, + 15.292, + 15.289, + 15.033, + 14.946, + 15.086, + 15.055, + 14.988, + 15.196, + 15.169, + 15.133, + 15.147, + 15.117, + 14.771, + 15.458, + 15.407, + 14.904, + 14.968, + 14.924, + 14.849, + 14.907, + 14.875, + 14.87 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "coderankembed", + "index_ms": 262032.5, + "p50_ms": 15.13, + "p90_ms": 15.8, + "p95_ms": 16.13, + "p99_ms": 17.13, + "latencies_ms": [ + 16.516, + 16.139, + 16.125, + 16.106, + 16.208, + 15.095, + 15.22, + 14.916, + 15.25, + 15.011, + 15.46, + 15.632, + 15.397, + 15.315, + 15.32, + 15.881, + 15.866, + 15.791, + 15.798, + 15.876, + 15.086, + 15.115, + 15.047, + 15.114, + 15.118, + 15.042, + 14.871, + 14.973, + 14.897, + 14.89, + 14.793, + 14.67, + 14.584, + 14.654, + 14.547, + 15.324, + 15.219, + 15.166, + 15.243, + 15.276, + 15.109, + 17.131, + 17.503, + 15.137, + 15.141, + 15.137, + 15.2, + 15.133, + 15.083, + 15.186, + 14.832, + 14.714, + 14.829, + 14.694, + 14.703, + 15.271, + 15.205, + 15.22, + 15.172, + 15.144, + 15.305, + 15.246, + 15.143, + 15.168, + 15.128, + 14.64, + 14.556, + 14.547, + 14.565, + 14.557, + 15.128, + 15.06, + 15.122, + 14.942, + 15.209, + 14.677, + 14.562, + 14.568, + 14.489, + 14.478, + 15.715, + 15.586, + 15.629, + 15.61, + 15.569, + 15.186, + 15.136, + 15.118, + 15.162, + 15.231, + 14.993, + 14.743, + 14.741, + 14.73, + 14.962, + 15.171, + 15.067, + 14.973, + 14.96, + 15.128 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "coderankembed", + "index_ms": 151399.1, + "p50_ms": 16.26, + "p90_ms": 20.55, + "p95_ms": 21.58, + "p99_ms": 23.79, + "latencies_ms": [ + 15.784, + 15.616, + 15.654, + 15.506, + 15.531, + 14.873, + 14.789, + 14.75, + 14.642, + 14.825, + 14.822, + 14.783, + 14.833, + 14.967, + 14.96, + 15.183, + 15.195, + 15.179, + 15.392, + 21.331, + 16.245, + 16.01, + 15.995, + 16.354, + 15.972, + 15.133, + 14.99, + 14.895, + 15.205, + 15.129, + 15.791, + 15.802, + 15.743, + 15.69, + 15.677, + 15.784, + 15.84, + 15.796, + 15.786, + 15.88, + 15.472, + 15.502, + 15.876, + 15.949, + 17.202, + 17.855, + 17.737, + 17.73, + 17.188, + 16.783, + 16.158, + 16.052, + 15.705, + 15.375, + 15.35, + 15.359, + 15.15, + 15.116, + 16.304, + 17.196, + 20.135, + 17.934, + 18.045, + 17.618, + 17.613, + 21.572, + 20.912, + 21.906, + 26.158, + 23.77, + 19.199, + 18.829, + 18.356, + 18.872, + 18.667, + 16.27, + 16.386, + 17.578, + 18.452, + 18.097, + 19.115, + 18.194, + 20.57, + 16.995, + 16.427, + 20.665, + 20.248, + 19.578, + 19.295, + 19.668, + 21.987, + 20.472, + 20.436, + 20.548, + 20.384, + 20.31, + 20.339, + 19.295, + 20.044, + 21.73 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "coderankembed", + "index_ms": 175723.8, + "p50_ms": 19.66, + "p90_ms": 22.09, + "p95_ms": 22.47, + "p99_ms": 22.58, + "latencies_ms": [ + 22.414, + 21.929, + 21.877, + 21.849, + 21.832, + 15.594, + 15.496, + 15.476, + 15.374, + 15.463, + 15.4, + 15.283, + 15.378, + 15.364, + 15.324, + 15.726, + 15.549, + 15.788, + 15.629, + 15.683, + 14.686, + 14.732, + 14.549, + 14.578, + 14.484, + 15.336, + 15.229, + 15.283, + 15.337, + 15.31, + 15.433, + 15.503, + 15.429, + 15.363, + 15.4, + 15.099, + 15.08, + 15.047, + 15.06, + 15.255, + 21.888, + 21.65, + 21.87, + 21.688, + 22.576, + 15.632, + 15.507, + 15.351, + 15.388, + 15.36, + 22.081, + 21.927, + 21.963, + 21.873, + 22.158, + 22.519, + 22.143, + 21.926, + 22.081, + 22.01, + 23.047, + 22.504, + 22.471, + 22.393, + 22.472, + 20.865, + 20.296, + 20.274, + 20.325, + 20.25, + 20.068, + 19.467, + 19.537, + 19.636, + 19.614, + 21.801, + 21.365, + 21.136, + 21.602, + 21.414, + 20.276, + 19.651, + 19.713, + 19.661, + 19.744, + 21.747, + 21.739, + 21.648, + 21.849, + 21.647, + 15.249, + 15.199, + 15.227, + 15.569, + 15.33, + 21.948, + 21.245, + 21.225, + 21.546, + 21.52 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "coderankembed", + "index_ms": 96347.8, + "p50_ms": 15.44, + "p90_ms": 19.57, + "p95_ms": 20.55, + "p99_ms": 23.27, + "latencies_ms": [ + 15.693, + 15.607, + 15.543, + 15.53, + 15.451, + 15.124, + 15.034, + 14.992, + 14.921, + 15.078, + 16.057, + 15.639, + 15.34, + 15.361, + 15.287, + 14.905, + 14.662, + 14.942, + 15.719, + 15.822, + 16.875, + 24.776, + 23.171, + 17.298, + 15.845, + 21.257, + 20.557, + 19.986, + 20.546, + 19.586, + 15.279, + 15.163, + 15.417, + 15.123, + 15.58, + 15.491, + 15.126, + 15.228, + 15.23, + 15.142, + 15.265, + 15.354, + 16.842, + 16.024, + 16.295, + 15.233, + 15.835, + 15.153, + 15.36, + 15.913, + 15.991, + 15.537, + 15.473, + 15.401, + 15.445, + 14.955, + 14.955, + 15.097, + 14.994, + 14.944, + 15.013, + 15.0, + 15.819, + 15.199, + 14.953, + 15.128, + 15.022, + 15.182, + 15.127, + 15.004, + 15.107, + 14.981, + 14.867, + 15.241, + 15.077, + 15.423, + 15.38, + 15.34, + 15.546, + 15.503, + 18.662, + 18.227, + 18.039, + 18.31, + 18.169, + 19.916, + 19.81, + 20.809, + 19.535, + 19.44, + 18.725, + 18.216, + 18.485, + 18.355, + 18.384 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "coderankembed", + "index_ms": 122587.5, + "p50_ms": 15.08, + "p90_ms": 19.64, + "p95_ms": 19.97, + "p99_ms": 20.92, + "latencies_ms": [ + 14.88, + 14.736, + 14.609, + 14.681, + 14.578, + 14.59, + 14.525, + 14.549, + 14.526, + 14.518, + 14.925, + 14.893, + 14.867, + 14.8, + 14.876, + 15.208, + 15.086, + 15.161, + 15.122, + 15.163, + 14.722, + 14.599, + 14.604, + 14.63, + 14.621, + 19.967, + 19.193, + 19.115, + 19.165, + 19.038, + 14.841, + 14.744, + 14.905, + 14.813, + 14.767, + 14.78, + 14.626, + 14.555, + 14.638, + 14.539, + 14.736, + 15.129, + 14.588, + 14.633, + 14.496, + 14.879, + 14.883, + 15.069, + 14.896, + 15.052, + 15.079, + 14.873, + 14.902, + 14.909, + 14.807, + 14.945, + 14.936, + 14.962, + 14.896, + 14.914, + 15.013, + 15.095, + 15.081, + 15.454, + 15.523, + 16.068, + 16.864, + 16.396, + 16.409, + 16.038, + 15.826, + 15.852, + 16.287, + 15.756, + 15.515, + 15.431, + 15.483, + 15.501, + 15.473, + 15.407, + 19.005, + 18.77, + 20.105, + 18.92, + 18.744, + 20.807, + 19.424, + 19.833, + 19.949, + 21.89, + 19.757, + 19.714, + 20.908, + 20.417, + 19.257, + 19.631, + 17.489, + 16.423, + 15.751, + 16.272 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "coderankembed", + "index_ms": 80621.6, + "p50_ms": 18.0, + "p90_ms": 20.82, + "p95_ms": 27.6, + "p99_ms": 34.83, + "latencies_ms": [ + 15.729, + 15.465, + 15.475, + 15.406, + 15.328, + 18.969, + 18.252, + 18.301, + 18.514, + 18.419, + 19.002, + 18.385, + 18.32, + 18.204, + 18.212, + 18.44, + 17.972, + 18.018, + 17.96, + 17.975, + 18.434, + 18.033, + 17.864, + 17.995, + 18.005, + 18.442, + 18.002, + 17.883, + 17.998, + 17.885, + 18.45, + 17.978, + 18.03, + 17.938, + 17.859, + 35.491, + 34.819, + 34.752, + 34.689, + 34.772, + 18.874, + 18.385, + 18.379, + 18.289, + 18.38, + 18.336, + 18.007, + 18.037, + 18.046, + 17.974, + 18.203, + 17.778, + 17.722, + 17.794, + 17.744, + 15.559, + 15.455, + 15.593, + 15.702, + 15.787, + 15.153, + 15.341, + 15.183, + 15.192, + 15.171, + 15.258, + 15.107, + 15.16, + 15.113, + 15.07, + 18.361, + 17.882, + 17.802, + 17.917, + 18.018, + 15.097, + 15.163, + 15.132, + 15.056, + 15.132, + 27.231, + 26.371, + 26.412, + 26.386, + 26.476, + 19.885, + 20.203, + 19.253, + 19.215, + 19.052, + 15.094, + 15.154, + 15.051, + 15.136, + 15.001, + 18.474, + 18.007, + 17.914, + 18.045, + 17.99 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "coderankembed", + "index_ms": 258344.4, + "p50_ms": 15.9, + "p90_ms": 25.11, + "p95_ms": 25.64, + "p99_ms": 27.1, + "latencies_ms": [ + 28.664, + 27.025, + 27.089, + 26.905, + 26.722, + 16.21, + 16.09, + 16.062, + 16.133, + 15.982, + 15.7, + 15.708, + 15.643, + 15.738, + 15.696, + 24.842, + 24.156, + 24.25, + 24.246, + 24.179, + 23.248, + 22.79, + 22.736, + 22.83, + 22.67, + 15.892, + 15.768, + 15.855, + 15.785, + 15.765, + 15.931, + 15.768, + 15.898, + 15.911, + 15.829, + 15.819, + 15.618, + 15.71, + 15.635, + 15.516, + 15.639, + 15.56, + 15.528, + 15.628, + 15.64, + 15.112, + 15.012, + 14.996, + 15.146, + 15.038, + 15.691, + 15.611, + 15.757, + 15.488, + 15.46, + 15.498, + 15.614, + 15.535, + 15.438, + 15.484, + 15.8, + 15.774, + 15.735, + 15.667, + 16.091, + 16.56, + 15.887, + 15.836, + 15.842, + 15.842, + 15.601, + 15.484, + 15.716, + 15.497, + 15.479, + 16.199, + 16.119, + 16.15, + 16.115, + 16.051, + 25.21, + 24.414, + 24.531, + 24.525, + 24.436, + 23.322, + 22.725, + 22.719, + 22.587, + 22.625, + 25.581, + 25.287, + 25.251, + 25.144, + 25.11, + 23.5, + 23.042, + 24.284, + 23.2, + 23.204 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "coderankembed", + "index_ms": 25676.5, + "p50_ms": 15.08, + "p90_ms": 17.74, + "p95_ms": 18.16, + "p99_ms": 19.32, + "latencies_ms": [ + 15.466, + 15.217, + 15.268, + 15.062, + 15.04, + 14.82, + 14.69, + 14.803, + 14.845, + 14.723, + 14.732, + 14.941, + 14.824, + 14.731, + 14.702, + 15.03, + 14.994, + 14.936, + 14.982, + 15.094, + 16.512, + 16.08, + 16.058, + 15.989, + 16.341, + 16.503, + 16.227, + 16.002, + 15.934, + 15.83, + 16.418, + 16.063, + 16.059, + 15.992, + 16.102, + 14.682, + 14.542, + 14.637, + 14.555, + 14.56, + 14.943, + 14.772, + 14.811, + 14.811, + 14.907, + 14.958, + 14.886, + 14.862, + 15.01, + 14.851, + 14.817, + 14.758, + 14.696, + 14.701, + 14.732, + 14.958, + 14.962, + 14.982, + 15.008, + 15.003, + 14.96, + 14.871, + 14.922, + 14.961, + 14.859, + 15.071, + 14.971, + 14.958, + 15.066, + 15.41, + 15.667, + 15.745, + 17.042, + 16.331, + 17.919, + 16.634, + 16.15, + 16.485, + 16.559, + 15.992, + 16.126, + 15.42, + 15.246, + 15.25, + 15.153, + 17.838, + 17.386, + 18.579, + 17.282, + 17.732, + 18.153, + 17.585, + 16.99, + 17.071, + 18.012, + 18.252, + 17.82, + 19.649, + 19.267, + 19.321 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "coderankembed", + "index_ms": 68379.1, + "p50_ms": 17.11, + "p90_ms": 21.03, + "p95_ms": 21.64, + "p99_ms": 22.7, + "latencies_ms": [ + 21.034, + 20.213, + 20.169, + 20.221, + 20.328, + 15.29, + 15.316, + 15.244, + 15.228, + 15.258, + 15.262, + 15.318, + 15.28, + 15.204, + 15.271, + 15.441, + 15.37, + 15.486, + 15.398, + 15.433, + 15.276, + 15.253, + 15.153, + 15.201, + 15.076, + 15.194, + 15.279, + 15.468, + 15.416, + 15.282, + 15.371, + 15.394, + 15.356, + 19.448, + 15.215, + 15.136, + 15.112, + 15.071, + 15.15, + 15.06, + 17.96, + 17.904, + 17.882, + 18.038, + 17.971, + 15.378, + 15.277, + 15.197, + 15.279, + 15.249, + 15.457, + 15.477, + 15.378, + 15.613, + 15.4, + 19.013, + 18.982, + 18.8, + 18.963, + 18.766, + 17.688, + 17.13, + 17.066, + 17.096, + 17.204, + 18.615, + 17.485, + 17.4, + 17.765, + 18.055, + 18.661, + 18.621, + 19.874, + 18.214, + 18.022, + 21.038, + 20.157, + 19.961, + 19.655, + 19.64, + 20.152, + 22.479, + 21.982, + 21.799, + 21.634, + 17.53, + 16.396, + 16.323, + 17.333, + 16.194, + 16.355, + 17.318, + 20.025, + 18.136, + 19.351, + 22.689, + 21.54, + 23.596, + 21.333, + 21.51 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "coderankembed", + "index_ms": 94982.0, + "p50_ms": 15.27, + "p90_ms": 15.55, + "p95_ms": 16.49, + "p99_ms": 18.74, + "latencies_ms": [ + 15.576, + 15.371, + 15.262, + 15.19, + 15.127, + 15.319, + 15.078, + 15.096, + 15.067, + 15.127, + 15.449, + 15.52, + 15.321, + 15.375, + 15.319, + 15.497, + 15.449, + 15.412, + 15.4, + 15.373, + 15.382, + 15.529, + 15.486, + 15.394, + 15.506, + 15.454, + 15.311, + 15.24, + 15.212, + 15.231, + 15.177, + 15.086, + 15.229, + 15.228, + 15.175, + 15.288, + 15.235, + 15.163, + 15.283, + 15.154, + 15.06, + 15.044, + 15.002, + 15.074, + 15.056, + 19.471, + 18.691, + 18.736, + 18.678, + 18.624, + 15.404, + 15.395, + 15.287, + 15.305, + 15.214, + 15.373, + 15.383, + 15.314, + 15.381, + 15.408, + 15.247, + 15.271, + 15.186, + 15.152, + 15.177, + 15.275, + 15.185, + 15.233, + 15.27, + 15.329, + 15.462, + 15.249, + 15.319, + 15.245, + 15.24, + 15.27, + 15.134, + 15.165, + 15.213, + 15.135, + 15.401, + 15.576, + 15.407, + 15.34, + 15.263, + 14.963, + 15.012, + 14.991, + 15.023, + 15.03, + 15.23, + 15.296, + 15.255, + 15.172, + 15.337, + 15.256, + 15.551, + 16.373, + 16.211, + 15.722 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "coderankembed", + "index_ms": 63844.4, + "p50_ms": 15.37, + "p90_ms": 17.57, + "p95_ms": 17.81, + "p99_ms": 18.05, + "latencies_ms": [ + 16.863, + 16.265, + 17.22, + 15.988, + 16.11, + 17.8, + 18.047, + 17.914, + 17.362, + 17.214, + 16.578, + 16.55, + 16.965, + 15.397, + 15.344, + 15.286, + 15.081, + 14.983, + 15.186, + 15.193, + 18.642, + 17.708, + 17.712, + 17.56, + 17.625, + 17.453, + 16.997, + 16.976, + 16.849, + 17.032, + 17.551, + 17.148, + 17.324, + 17.061, + 17.051, + 15.328, + 15.363, + 15.314, + 15.197, + 15.319, + 15.232, + 15.161, + 15.313, + 15.203, + 15.225, + 15.31, + 15.223, + 15.226, + 15.303, + 15.203, + 15.426, + 15.205, + 15.162, + 15.189, + 15.137, + 15.306, + 15.254, + 15.299, + 15.303, + 15.305, + 15.106, + 15.203, + 15.137, + 15.081, + 15.135, + 15.277, + 15.286, + 15.287, + 15.372, + 15.416, + 15.267, + 15.112, + 15.235, + 15.135, + 15.127, + 15.137, + 15.138, + 15.167, + 15.853, + 15.463, + 15.518, + 15.381, + 15.493, + 15.648, + 16.802, + 15.519, + 15.358, + 15.209, + 15.347, + 15.337, + 17.912, + 17.236, + 17.212, + 17.259, + 17.135, + 18.007, + 17.613, + 17.403, + 17.498, + 17.332 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "coderankembed", + "index_ms": 46563.6, + "p50_ms": 15.17, + "p90_ms": 18.26, + "p95_ms": 19.44, + "p99_ms": 20.28, + "latencies_ms": [ + 14.971, + 14.754, + 14.682, + 14.748, + 14.668, + 14.491, + 14.499, + 14.523, + 14.458, + 14.884, + 14.568, + 14.729, + 14.765, + 14.857, + 15.076, + 15.296, + 15.145, + 15.018, + 14.945, + 14.579, + 14.546, + 14.488, + 14.481, + 14.761, + 14.606, + 14.519, + 14.809, + 14.44, + 14.345, + 14.387, + 14.588, + 14.613, + 14.568, + 14.438, + 14.392, + 14.452, + 14.393, + 14.422, + 14.42, + 14.43, + 14.354, + 14.297, + 14.208, + 14.248, + 14.202, + 14.702, + 14.726, + 14.691, + 14.608, + 14.649, + 15.16, + 15.249, + 15.236, + 15.187, + 15.118, + 15.295, + 15.225, + 15.327, + 15.222, + 15.174, + 15.312, + 15.701, + 15.482, + 15.923, + 16.663, + 16.528, + 16.856, + 15.682, + 15.802, + 16.094, + 16.209, + 15.884, + 15.481, + 15.425, + 15.45, + 17.748, + 17.373, + 19.449, + 18.669, + 19.012, + 17.775, + 17.881, + 17.61, + 17.4, + 17.247, + 19.509, + 18.613, + 19.403, + 19.525, + 21.993, + 16.603, + 16.366, + 17.16, + 17.403, + 16.455, + 15.169, + 15.172, + 16.194, + 16.68, + 16.87, + 20.303, + 18.25, + 19.85, + 18.272, + 17.456 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "coderankembed", + "index_ms": 41054.0, + "p50_ms": 15.12, + "p90_ms": 17.85, + "p95_ms": 18.66, + "p99_ms": 18.86, + "latencies_ms": [ + 15.381, + 15.158, + 15.09, + 15.16, + 15.232, + 15.276, + 15.189, + 15.208, + 15.488, + 15.288, + 14.864, + 14.824, + 14.74, + 14.686, + 14.597, + 14.877, + 14.763, + 14.776, + 14.902, + 14.822, + 17.421, + 17.027, + 16.907, + 16.847, + 16.884, + 19.336, + 18.626, + 18.73, + 18.77, + 18.693, + 18.796, + 18.07, + 18.05, + 17.89, + 17.846, + 15.081, + 15.014, + 15.003, + 14.985, + 14.949, + 14.768, + 14.658, + 14.654, + 14.637, + 14.726, + 15.19, + 15.157, + 15.062, + 15.012, + 14.997, + 14.832, + 14.765, + 14.772, + 14.883, + 14.845, + 15.084, + 15.063, + 14.993, + 15.086, + 15.084, + 14.96, + 14.838, + 14.801, + 14.86, + 14.829, + 16.007, + 15.672, + 14.997, + 14.888, + 14.797, + 15.12, + 15.14, + 15.267, + 14.995, + 15.141, + 15.18, + 15.122, + 15.202, + 15.076, + 15.239, + 17.575, + 17.043, + 16.992, + 17.054, + 17.059, + 17.773, + 17.334, + 17.31, + 17.254, + 17.303 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "coderankembed", + "index_ms": 100960.6, + "p50_ms": 15.45, + "p90_ms": 18.47, + "p95_ms": 18.98, + "p99_ms": 19.61, + "latencies_ms": [ + 15.725, + 15.498, + 15.507, + 15.648, + 15.704, + 15.372, + 15.257, + 15.339, + 15.365, + 15.175, + 20.608, + 19.566, + 19.547, + 19.599, + 19.513, + 15.018, + 15.114, + 15.043, + 14.929, + 14.885, + 15.047, + 14.991, + 14.966, + 14.98, + 14.97, + 15.581, + 15.507, + 15.357, + 15.411, + 15.377, + 18.862, + 18.526, + 18.438, + 18.459, + 18.412, + 15.451, + 15.401, + 15.355, + 15.448, + 15.425, + 14.961, + 15.195, + 14.997, + 14.923, + 14.882, + 15.177, + 15.068, + 15.094, + 15.043, + 15.913, + 15.523, + 15.188, + 15.164, + 15.047, + 15.108, + 17.937, + 17.749, + 17.851, + 17.825, + 17.898, + 18.948, + 18.301, + 18.318, + 18.306, + 18.836, + 18.682, + 18.246, + 18.145, + 18.038, + 18.045, + 15.017, + 15.046, + 14.891, + 14.93, + 15.014, + 15.185, + 15.165, + 15.269, + 15.143, + 15.23, + 14.974, + 14.882, + 14.952, + 14.892, + 14.861, + 17.349, + 17.052, + 17.156, + 17.157, + 17.097, + 17.811, + 17.637, + 17.778, + 17.705, + 17.664, + 17.46, + 17.406, + 17.423, + 17.305, + 17.586 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "coderankembed", + "index_ms": 178931.3, + "p50_ms": 19.83, + "p90_ms": 22.41, + "p95_ms": 22.69, + "p99_ms": 23.0, + "latencies_ms": [ + 22.144, + 20.981, + 20.936, + 20.805, + 20.768, + 15.394, + 15.43, + 15.448, + 15.279, + 15.366, + 22.469, + 20.737, + 20.799, + 20.935, + 20.647, + 21.051, + 20.45, + 20.53, + 20.397, + 20.624, + 14.93, + 14.728, + 14.743, + 14.735, + 14.88, + 14.926, + 14.634, + 14.862, + 14.755, + 14.685, + 22.999, + 22.205, + 22.406, + 22.218, + 22.41, + 15.054, + 15.479, + 15.08, + 14.941, + 15.085, + 15.05, + 15.075, + 14.901, + 14.857, + 14.876, + 15.617, + 16.243, + 15.515, + 15.345, + 15.398, + 21.117, + 20.376, + 21.389, + 20.516, + 20.698, + 14.846, + 14.884, + 14.778, + 14.747, + 14.802, + 22.908, + 22.489, + 22.381, + 22.408, + 22.455, + 15.276, + 15.162, + 15.187, + 15.145, + 15.082, + 23.347, + 22.891, + 22.821, + 22.687, + 22.674, + 14.985, + 15.234, + 14.959, + 14.94, + 14.915, + 21.29, + 21.197, + 20.87, + 21.114, + 20.84, + 19.883, + 19.715, + 21.113, + 19.653, + 19.836, + 19.823, + 19.896, + 19.752, + 19.803, + 19.865, + 21.246, + 20.694, + 20.755, + 20.701, + 20.687 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "coderankembed", + "index_ms": 103921.5, + "p50_ms": 17.37, + "p90_ms": 20.48, + "p95_ms": 21.19, + "p99_ms": 21.6, + "latencies_ms": [ + 18.56, + 17.984, + 18.151, + 19.13, + 18.613, + 17.349, + 16.807, + 18.543, + 17.122, + 16.871, + 16.47, + 16.212, + 16.025, + 16.07, + 21.002, + 21.228, + 21.603, + 18.439, + 18.156, + 17.44, + 16.661, + 16.208, + 16.933, + 15.796, + 15.839, + 16.519, + 15.555, + 15.58, + 15.834, + 16.751, + 15.714, + 15.466, + 16.444, + 15.747, + 15.57, + 15.907, + 17.592, + 18.123, + 17.395, + 17.103, + 16.933, + 16.822, + 17.145, + 15.995, + 15.708, + 15.386, + 15.147, + 15.175, + 15.072, + 15.026, + 21.037, + 20.468, + 20.607, + 20.413, + 20.393, + 15.292, + 15.325, + 15.2, + 15.209, + 15.245, + 21.765, + 21.318, + 21.134, + 21.278, + 21.184, + 19.218, + 18.807, + 18.985, + 18.878, + 18.788, + 18.74, + 18.317, + 18.435, + 18.197, + 18.302, + 18.408, + 18.075, + 18.064, + 18.031, + 17.981, + 18.374, + 17.878, + 17.83, + 17.899, + 17.81, + 15.265, + 15.24, + 15.296, + 15.313, + 15.221, + 15.289, + 15.36, + 15.334, + 15.412, + 15.43, + 18.718, + 18.302, + 18.322, + 18.285, + 18.326 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "coderankembed", + "index_ms": 61338.3, + "p50_ms": 16.79, + "p90_ms": 17.91, + "p95_ms": 18.73, + "p99_ms": 19.01, + "latencies_ms": [ + 18.717, + 17.909, + 17.828, + 17.733, + 17.78, + 14.667, + 14.714, + 14.638, + 14.544, + 14.744, + 14.882, + 14.813, + 14.882, + 14.723, + 14.831, + 17.399, + 17.268, + 17.408, + 17.306, + 17.35, + 15.678, + 14.803, + 14.774, + 14.705, + 14.7, + 15.258, + 15.156, + 15.162, + 15.142, + 15.122, + 14.801, + 14.749, + 14.779, + 14.859, + 14.817, + 14.929, + 14.971, + 14.912, + 14.815, + 15.315, + 18.2, + 18.211, + 18.126, + 17.908, + 17.836, + 17.459, + 17.63, + 17.502, + 17.495, + 17.428, + 17.781, + 17.84, + 17.787, + 17.901, + 17.785, + 15.493, + 15.371, + 15.368, + 15.594, + 15.526, + 17.503, + 16.981, + 16.851, + 16.823, + 16.933, + 16.987, + 16.526, + 16.461, + 16.474, + 16.461, + 17.084, + 16.619, + 16.634, + 16.684, + 16.673, + 17.325, + 16.883, + 16.711, + 16.749, + 16.721, + 17.109, + 16.652, + 16.6, + 16.586, + 16.611, + 19.002, + 18.975, + 18.971, + 19.726, + 18.998, + 17.775, + 17.728, + 17.636, + 17.594, + 17.57, + 17.254, + 17.327, + 17.424, + 17.431, + 17.381 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "coderankembed", + "index_ms": 234412.8, + "p50_ms": 16.66, + "p90_ms": 19.16, + "p95_ms": 22.02, + "p99_ms": 25.08, + "latencies_ms": [ + 16.546, + 16.336, + 16.279, + 16.393, + 16.422, + 25.691, + 25.077, + 25.054, + 24.733, + 24.789, + 15.586, + 15.638, + 15.52, + 15.546, + 15.565, + 15.761, + 15.744, + 15.864, + 15.854, + 15.851, + 15.814, + 15.807, + 15.842, + 15.731, + 15.777, + 15.834, + 15.768, + 15.869, + 15.784, + 15.808, + 16.116, + 16.487, + 16.87, + 19.131, + 18.311, + 17.504, + 17.147, + 17.017, + 16.79, + 16.744, + 16.55, + 16.073, + 16.469, + 15.981, + 15.917, + 16.455, + 18.218, + 18.045, + 18.27, + 17.312, + 17.031, + 16.992, + 16.346, + 16.386, + 16.544, + 17.466, + 19.394, + 18.283, + 18.527, + 18.557, + 17.339, + 17.339, + 17.353, + 19.098, + 17.398, + 16.228, + 16.402, + 17.766, + 19.403, + 21.881, + 20.018, + 19.918, + 17.832, + 18.727, + 17.27, + 16.654, + 16.408, + 16.537, + 16.387, + 16.585, + 17.79, + 16.009, + 15.98, + 15.886, + 16.341, + 16.656, + 16.843, + 17.209, + 16.935, + 17.025, + 17.769, + 18.646, + 17.901, + 18.06, + 18.146, + 17.335, + 18.372, + 16.431, + 16.071, + 15.981 + ] + }, + { + "repo": "nvm", + "language": "bash", + "tool": "bm25", + "index_ms": 10.6, + "p50_ms": 0.14, + "p90_ms": 0.21, + "p95_ms": 0.24, + "p99_ms": 0.32, + "latencies_ms": [ + 0.252, + 0.178, + 0.149, + 0.233, + 0.146, + 0.139, + 0.134, + 0.132, + 0.132, + 0.221, + 0.242, + 0.195, + 0.173, + 0.174, + 0.214, + 0.201, + 0.174, + 0.166, + 0.166, + 0.164, + 0.098, + 0.093, + 0.092, + 0.094, + 0.093, + 0.1, + 0.096, + 0.095, + 0.095, + 0.138, + 0.135, + 0.121, + 0.119, + 0.119, + 0.117, + 0.152, + 0.15, + 0.152, + 0.169, + 0.156, + 0.146, + 0.141, + 0.142, + 0.14, + 0.143, + 0.104, + 0.1, + 0.1, + 0.101, + 0.101, + 0.185, + 0.185, + 0.179, + 0.181, + 0.182, + 0.125, + 0.315, + 0.254, + 0.216, + 0.143, + 0.183, + 0.177, + 0.178, + 0.211, + 0.38, + 0.128, + 0.113, + 0.111, + 0.174, + 0.15, + 0.15, + 0.14, + 0.14, + 0.139, + 0.141, + 0.152, + 0.15, + 0.16, + 0.15, + 0.149, + 0.074, + 0.069, + 0.068, + 0.068, + 0.068, + 0.183, + 0.207, + 0.174, + 0.247, + 0.234, + 0.111, + 0.1, + 0.099, + 0.102, + 0.104, + 0.084, + 0.08, + 0.079, + 0.079, + 0.082 + ] + }, + { + "repo": "libuv", + "language": "c", + "tool": "bm25", + "index_ms": 90.5, + "p50_ms": 0.21, + "p90_ms": 0.36, + "p95_ms": 0.51, + "p99_ms": 0.73, + "latencies_ms": [ + 0.497, + 0.459, + 0.451, + 0.443, + 0.444, + 0.107, + 0.104, + 0.1, + 0.1, + 0.1, + 0.3, + 0.288, + 0.284, + 0.284, + 0.287, + 0.743, + 0.727, + 0.719, + 0.724, + 0.735, + 0.266, + 0.25, + 0.245, + 0.242, + 0.242, + 0.198, + 0.244, + 0.214, + 0.194, + 0.193, + 0.188, + 0.178, + 0.172, + 0.173, + 0.173, + 0.239, + 0.211, + 0.208, + 0.211, + 0.209, + 0.077, + 0.068, + 0.068, + 0.067, + 0.066, + 0.168, + 0.157, + 0.153, + 0.154, + 0.171, + 0.328, + 0.287, + 0.286, + 0.29, + 0.277, + 0.327, + 0.306, + 0.356, + 0.306, + 0.29, + 0.35, + 0.315, + 0.308, + 0.307, + 0.306, + 0.08, + 0.07, + 0.068, + 0.069, + 0.067, + 0.102, + 0.096, + 0.092, + 0.092, + 0.091, + 0.053, + 0.044, + 0.043, + 0.043, + 0.043, + 0.356, + 0.352, + 0.339, + 0.339, + 0.339, + 0.154, + 0.143, + 0.14, + 0.138, + 0.138, + 0.088, + 0.074, + 0.071, + 0.071, + 0.071, + 0.271, + 0.246, + 0.252, + 0.251, + 0.247 + ] + }, + { + "repo": "nlohmann-json", + "language": "cpp", + "tool": "bm25", + "index_ms": 47.1, + "p50_ms": 0.32, + "p90_ms": 0.53, + "p95_ms": 0.65, + "p99_ms": 0.87, + "latencies_ms": [ + 0.446, + 0.412, + 0.403, + 0.396, + 0.396, + 0.266, + 0.257, + 0.256, + 0.259, + 0.263, + 0.295, + 0.293, + 0.289, + 0.291, + 0.294, + 0.434, + 0.421, + 0.423, + 0.419, + 0.423, + 0.489, + 0.476, + 0.474, + 0.472, + 0.472, + 0.33, + 0.32, + 0.316, + 0.32, + 0.316, + 0.522, + 0.507, + 0.507, + 0.512, + 0.521, + 0.584, + 0.584, + 0.573, + 0.64, + 0.59, + 0.329, + 0.325, + 0.32, + 0.32, + 0.318, + 0.311, + 0.306, + 0.303, + 0.307, + 0.305, + 0.296, + 0.29, + 0.288, + 0.283, + 0.285, + 0.263, + 0.258, + 0.257, + 0.254, + 0.254, + 0.856, + 0.837, + 0.844, + 0.903, + 0.866, + 0.277, + 0.265, + 0.267, + 0.267, + 0.265, + 0.373, + 0.361, + 0.364, + 0.367, + 0.362, + 0.272, + 0.265, + 0.265, + 0.26, + 0.263, + 0.379, + 0.367, + 0.367, + 0.365, + 0.365, + 0.344, + 0.337, + 0.336, + 0.344, + 0.342, + 0.265, + 0.258, + 0.258, + 0.261, + 0.262, + 0.029, + 0.026, + 0.026, + 0.026, + 0.025 + ] + }, + { + "repo": "messagepack-csharp", + "language": "csharp", + "tool": "bm25", + "index_ms": 72.1, + "p50_ms": 0.49, + "p90_ms": 0.93, + "p95_ms": 1.08, + "p99_ms": 1.38, + "latencies_ms": [ + 0.762, + 0.69, + 0.673, + 0.683, + 0.678, + 0.199, + 0.191, + 0.188, + 0.189, + 0.187, + 0.286, + 0.274, + 0.272, + 0.274, + 0.27, + 0.572, + 0.51, + 0.475, + 0.472, + 0.645, + 0.168, + 0.145, + 0.104, + 0.104, + 0.098, + 0.24, + 0.21, + 0.236, + 0.241, + 0.22, + 0.405, + 0.383, + 0.375, + 0.358, + 0.362, + 0.159, + 0.146, + 0.146, + 0.145, + 0.146, + 0.712, + 0.685, + 0.685, + 0.687, + 0.686, + 0.282, + 0.265, + 0.263, + 0.263, + 0.262, + 0.834, + 0.807, + 0.889, + 0.864, + 0.976, + 0.803, + 0.723, + 0.668, + 0.73, + 0.634, + 0.669, + 0.652, + 0.662, + 0.658, + 0.658, + 0.865, + 0.897, + 0.842, + 0.812, + 0.805, + 0.138, + 0.111, + 0.108, + 0.106, + 0.106, + 0.84, + 1.024, + 1.378, + 1.186, + 1.078, + 0.293, + 0.141, + 0.128, + 0.125, + 0.123, + 1.18, + 1.003, + 1.292, + 0.982, + 1.534, + 0.519, + 0.391, + 0.418, + 0.47, + 0.589, + 0.657, + 0.924, + 0.708, + 0.889, + 0.726 + ] + }, + { + "repo": "phoenix", + "language": "elixir", + "tool": "bm25", + "index_ms": 46.7, + "p50_ms": 0.13, + "p90_ms": 0.4, + "p95_ms": 0.42, + "p99_ms": 0.46, + "latencies_ms": [ + 0.429, + 0.395, + 0.4, + 0.4, + 0.398, + 0.118, + 0.113, + 0.112, + 0.11, + 0.109, + 0.133, + 0.127, + 0.125, + 0.125, + 0.124, + 0.145, + 0.142, + 0.14, + 0.141, + 0.141, + 0.165, + 0.162, + 0.162, + 0.195, + 0.374, + 0.423, + 0.329, + 0.213, + 0.161, + 0.156, + 0.082, + 0.075, + 0.073, + 0.074, + 0.074, + 0.202, + 0.226, + 0.163, + 0.138, + 0.136, + 0.274, + 0.441, + 0.321, + 0.301, + 0.283, + 0.103, + 0.094, + 0.093, + 0.092, + 0.092, + 0.324, + 0.326, + 0.302, + 0.293, + 0.351, + 0.109, + 0.092, + 0.092, + 0.092, + 0.091, + 0.097, + 0.086, + 0.085, + 0.085, + 0.085, + 0.112, + 0.098, + 0.097, + 0.097, + 0.096, + 0.143, + 0.139, + 0.137, + 0.167, + 0.145, + 0.42, + 0.454, + 0.488, + 0.406, + 0.381, + 0.042, + 0.039, + 0.038, + 0.038, + 0.038, + 0.035, + 0.033, + 0.032, + 0.032, + 0.032, + 0.031, + 0.029, + 0.028, + 0.028, + 0.028 + ] + }, + { + "repo": "gin", + "language": "go", + "tool": "bm25", + "index_ms": 73.9, + "p50_ms": 0.15, + "p90_ms": 0.38, + "p95_ms": 0.41, + "p99_ms": 0.61, + "latencies_ms": [ + 0.117, + 0.068, + 0.063, + 0.084, + 0.121, + 0.082, + 0.073, + 0.071, + 0.071, + 0.152, + 0.221, + 0.174, + 0.167, + 0.166, + 0.167, + 0.164, + 0.154, + 0.213, + 0.197, + 0.203, + 0.207, + 0.218, + 0.193, + 0.2, + 0.185, + 0.389, + 0.377, + 0.446, + 0.405, + 0.355, + 0.104, + 0.077, + 0.067, + 0.066, + 0.065, + 0.088, + 0.18, + 0.144, + 0.18, + 0.225, + 0.14, + 0.117, + 0.079, + 0.039, + 0.031, + 0.147, + 0.108, + 0.106, + 0.105, + 0.106, + 0.127, + 0.115, + 0.111, + 0.3, + 0.276, + 0.375, + 0.333, + 0.396, + 0.483, + 0.382, + 0.398, + 0.276, + 0.679, + 0.429, + 0.212, + 0.215, + 0.217, + 0.218, + 0.218, + 0.199, + 0.182, + 0.193, + 0.262, + 0.258, + 0.26, + 0.606, + 0.308, + 0.243, + 0.275, + 0.316, + 0.054, + 0.069, + 0.032, + 0.05, + 0.041, + 0.096, + 0.077, + 0.093, + 0.078, + 0.094, + 0.057, + 0.051, + 0.051, + 0.05, + 0.05, + 0.103, + 0.086, + 0.085, + 0.087, + 0.147 + ] + }, + { + "repo": "aeson", + "language": "haskell", + "tool": "bm25", + "index_ms": 41.3, + "p50_ms": 0.36, + "p90_ms": 0.59, + "p95_ms": 0.66, + "p99_ms": 1.28, + "latencies_ms": [ + 0.646, + 0.561, + 0.399, + 0.38, + 0.379, + 0.377, + 1.294, + 1.259, + 1.284, + 0.556, + 0.418, + 0.274, + 0.259, + 0.393, + 0.433, + 0.436, + 0.309, + 0.282, + 0.277, + 0.27, + 0.278, + 0.406, + 0.339, + 0.4, + 0.299, + 0.442, + 0.437, + 0.428, + 0.43, + 0.433, + 0.379, + 0.538, + 0.361, + 0.357, + 0.358, + 0.449, + 0.441, + 0.542, + 0.726, + 0.588, + 0.416, + 0.574, + 0.426, + 0.865, + 0.427, + 0.369, + 0.363, + 0.363, + 0.36, + 0.362, + 0.326, + 0.45, + 0.316, + 0.448, + 0.224, + 0.24, + 0.232, + 0.235, + 0.231, + 0.231, + 0.323, + 0.316, + 0.654, + 0.433, + 0.336, + 0.245, + 0.229, + 0.227, + 0.224, + 0.223, + 0.324, + 0.305, + 0.308, + 0.373, + 0.383, + 0.233, + 0.222, + 0.222, + 0.221, + 0.229, + 0.612, + 0.588, + 0.585, + 0.569, + 0.579, + 0.144, + 0.141, + 0.137, + 0.135, + 0.132, + 0.241, + 0.229, + 0.231, + 0.23, + 0.231, + 0.535, + 0.42, + 0.347, + 0.497, + 0.573 + ] + }, + { + "repo": "gson", + "language": "java", + "tool": "bm25", + "index_ms": 121.8, + "p50_ms": 0.54, + "p90_ms": 0.84, + "p95_ms": 1.21, + "p99_ms": 1.69, + "latencies_ms": [ + 1.689, + 1.613, + 1.614, + 1.817, + 1.659, + 0.733, + 0.669, + 0.65, + 0.638, + 0.642, + 0.332, + 0.331, + 0.388, + 0.432, + 0.497, + 0.434, + 0.41, + 0.404, + 0.449, + 0.421, + 0.373, + 0.355, + 0.351, + 0.35, + 0.35, + 0.649, + 0.633, + 0.629, + 0.63, + 0.632, + 0.609, + 0.672, + 0.621, + 0.598, + 0.598, + 0.578, + 0.562, + 0.603, + 0.558, + 0.553, + 0.423, + 0.401, + 0.397, + 0.394, + 0.395, + 0.244, + 0.229, + 0.228, + 0.227, + 0.243, + 0.214, + 0.191, + 0.191, + 0.189, + 0.187, + 0.277, + 0.256, + 0.251, + 0.251, + 0.251, + 0.56, + 0.54, + 0.536, + 0.537, + 0.538, + 0.678, + 0.86, + 0.673, + 0.626, + 0.624, + 0.67, + 0.646, + 0.635, + 0.644, + 0.796, + 1.185, + 0.855, + 0.84, + 0.845, + 0.833, + 0.875, + 0.653, + 0.629, + 0.622, + 0.623, + 0.382, + 0.372, + 0.368, + 0.366, + 0.368, + 0.525, + 0.508, + 0.504, + 0.54, + 0.501, + 0.506, + 0.718, + 0.503, + 0.487, + 0.484 + ] + }, + { + "repo": "axios", + "language": "javascript", + "tool": "bm25", + "index_ms": 8.7, + "p50_ms": 0.03, + "p90_ms": 0.07, + "p95_ms": 0.08, + "p99_ms": 0.1, + "latencies_ms": [ + 0.062, + 0.039, + 0.037, + 0.036, + 0.036, + 0.033, + 0.031, + 0.03, + 0.031, + 0.031, + 0.021, + 0.02, + 0.02, + 0.02, + 0.02, + 0.082, + 0.084, + 0.076, + 0.078, + 0.077, + 0.023, + 0.022, + 0.022, + 0.023, + 0.023, + 0.008, + 0.008, + 0.007, + 0.007, + 0.007, + 0.024, + 0.022, + 0.022, + 0.023, + 0.023, + 0.014, + 0.013, + 0.013, + 0.013, + 0.013, + 0.032, + 0.031, + 0.031, + 0.03, + 0.029, + 0.021, + 0.021, + 0.02, + 0.021, + 0.021, + 0.04, + 0.039, + 0.041, + 0.037, + 0.038, + 0.067, + 0.066, + 0.065, + 0.066, + 0.066, + 0.054, + 0.052, + 0.052, + 0.052, + 0.051, + 0.022, + 0.02, + 0.02, + 0.02, + 0.02, + 0.047, + 0.045, + 0.045, + 0.044, + 0.044, + 0.095, + 0.094, + 0.094, + 0.097, + 0.094, + 0.035, + 0.034, + 0.034, + 0.034, + 0.034, + 0.016, + 0.014, + 0.014, + 0.014, + 0.014, + 0.044, + 0.043, + 0.044, + 0.044, + 0.043, + 0.044, + 0.042, + 0.042, + 0.042, + 0.042 + ] + }, + { + "repo": "ktor", + "language": "kotlin", + "tool": "bm25", + "index_ms": 31.3, + "p50_ms": 0.19, + "p90_ms": 0.39, + "p95_ms": 0.44, + "p99_ms": 0.63, + "latencies_ms": [ + 0.629, + 0.582, + 0.561, + 0.563, + 0.683, + 0.184, + 0.171, + 0.168, + 0.176, + 0.169, + 0.165, + 0.161, + 0.159, + 0.16, + 0.162, + 0.218, + 0.188, + 0.189, + 0.188, + 0.188, + 0.184, + 0.179, + 0.176, + 0.174, + 0.175, + 0.129, + 0.125, + 0.124, + 0.266, + 0.145, + 0.142, + 0.134, + 0.133, + 0.139, + 0.132, + 0.06, + 0.054, + 0.056, + 0.053, + 0.053, + 0.094, + 0.088, + 0.089, + 0.089, + 0.089, + 0.257, + 0.314, + 0.267, + 0.248, + 0.265, + 0.262, + 0.253, + 0.251, + 0.251, + 0.251, + 0.264, + 0.258, + 0.255, + 0.26, + 0.264, + 0.122, + 0.117, + 0.119, + 0.118, + 0.123, + 0.112, + 0.11, + 0.11, + 0.109, + 0.117, + 0.124, + 0.114, + 0.111, + 0.111, + 0.114, + 0.245, + 0.244, + 0.24, + 0.289, + 0.269, + 0.432, + 0.419, + 0.424, + 0.414, + 0.42, + 0.288, + 0.281, + 0.287, + 0.282, + 0.277, + 0.296, + 0.292, + 0.389, + 0.296, + 0.29, + 0.281, + 0.335, + 0.292, + 0.269, + 0.272 + ] + }, + { + "repo": "telescope.nvim", + "language": "lua", + "tool": "bm25", + "index_ms": 42.1, + "p50_ms": 0.23, + "p90_ms": 0.34, + "p95_ms": 0.39, + "p99_ms": 0.58, + "latencies_ms": [ + 0.261, + 0.389, + 0.364, + 0.334, + 0.306, + 0.29, + 0.345, + 0.208, + 0.192, + 0.189, + 0.212, + 0.197, + 0.196, + 0.194, + 0.202, + 0.246, + 0.239, + 0.234, + 0.288, + 0.336, + 0.34, + 0.726, + 0.575, + 0.421, + 0.338, + 0.337, + 0.314, + 0.314, + 0.312, + 0.306, + 0.178, + 0.169, + 0.169, + 0.169, + 0.168, + 0.232, + 0.223, + 0.223, + 0.222, + 0.22, + 0.201, + 0.195, + 0.195, + 0.193, + 0.194, + 0.295, + 0.286, + 0.41, + 0.322, + 0.334, + 0.208, + 0.207, + 0.22, + 0.243, + 0.221, + 0.231, + 0.211, + 0.21, + 0.209, + 0.214, + 0.3, + 0.282, + 0.28, + 0.283, + 0.509, + 0.346, + 0.292, + 0.29, + 0.286, + 0.287, + 0.245, + 0.301, + 0.345, + 0.248, + 0.238, + 0.193, + 0.182, + 0.181, + 0.275, + 0.241, + 0.197, + 0.224, + 0.318, + 0.267, + 0.198, + 0.165, + 0.286, + 0.236, + 0.163, + 0.233, + 0.204, + 0.169, + 0.168, + 0.286, + 0.213, + 0.197, + 0.169, + 0.17, + 0.166, + 0.166 + ] + }, + { + "repo": "monolog", + "language": "php", + "tool": "bm25", + "index_ms": 23.6, + "p50_ms": 0.14, + "p90_ms": 0.24, + "p95_ms": 0.26, + "p99_ms": 0.29, + "latencies_ms": [ + 0.192, + 0.174, + 0.164, + 0.162, + 0.165, + 0.264, + 0.259, + 0.294, + 0.306, + 0.265, + 0.064, + 0.064, + 0.057, + 0.055, + 0.053, + 0.082, + 0.078, + 0.076, + 0.077, + 0.078, + 0.026, + 0.023, + 0.023, + 0.022, + 0.022, + 0.107, + 0.101, + 0.104, + 0.102, + 0.103, + 0.062, + 0.059, + 0.06, + 0.059, + 0.059, + 0.16, + 0.161, + 0.145, + 0.148, + 0.15, + 0.245, + 0.246, + 0.238, + 0.24, + 0.24, + 0.192, + 0.185, + 0.23, + 0.193, + 0.181, + 0.229, + 0.22, + 0.226, + 0.229, + 0.237, + 0.237, + 0.24, + 0.233, + 0.23, + 0.229, + 0.132, + 0.109, + 0.109, + 0.109, + 0.108, + 0.16, + 0.156, + 0.153, + 0.152, + 0.154, + 0.145, + 0.14, + 0.139, + 0.139, + 0.142, + 0.141, + 0.136, + 0.134, + 0.141, + 0.135, + 0.264, + 0.25, + 0.254, + 0.242, + 0.243, + 0.099, + 0.098, + 0.091, + 0.092, + 0.091, + 0.129, + 0.123, + 0.121, + 0.122, + 0.122, + 0.117, + 0.111, + 0.108, + 0.109, + 0.11 + ] + }, + { + "repo": "flask", + "language": "python", + "tool": "bm25", + "index_ms": 19.3, + "p50_ms": 0.03, + "p90_ms": 0.2, + "p95_ms": 0.22, + "p99_ms": 0.23, + "latencies_ms": [ + 0.048, + 0.032, + 0.029, + 0.029, + 0.028, + 0.059, + 0.056, + 0.055, + 0.055, + 0.055, + 0.016, + 0.015, + 0.015, + 0.014, + 0.014, + 0.022, + 0.021, + 0.027, + 0.019, + 0.02, + 0.013, + 0.011, + 0.011, + 0.011, + 0.011, + 0.019, + 0.018, + 0.019, + 0.018, + 0.018, + 0.035, + 0.032, + 0.031, + 0.031, + 0.031, + 0.023, + 0.022, + 0.022, + 0.022, + 0.022, + 0.014, + 0.013, + 0.013, + 0.013, + 0.013, + 0.041, + 0.039, + 0.039, + 0.04, + 0.039, + 0.149, + 0.145, + 0.144, + 0.148, + 0.143, + 0.197, + 0.198, + 0.199, + 0.233, + 0.213, + 0.235, + 0.225, + 0.221, + 0.22, + 0.218, + 0.059, + 0.05, + 0.05, + 0.05, + 0.05, + 0.203, + 0.203, + 0.189, + 0.191, + 0.19, + 0.019, + 0.018, + 0.017, + 0.017, + 0.017, + 0.022, + 0.02, + 0.02, + 0.019, + 0.018, + 0.032, + 0.031, + 0.031, + 0.031, + 0.031, + 0.014, + 0.014, + 0.013, + 0.013, + 0.013, + 0.008, + 0.008, + 0.008, + 0.008, + 0.008, + 0.121, + 0.124, + 0.115, + 0.116, + 0.117 + ] + }, + { + "repo": "rack", + "language": "ruby", + "tool": "bm25", + "index_ms": 15.6, + "p50_ms": 0.05, + "p90_ms": 0.1, + "p95_ms": 0.11, + "p99_ms": 0.11, + "latencies_ms": [ + 0.113, + 0.091, + 0.09, + 0.088, + 0.088, + 0.112, + 0.11, + 0.11, + 0.11, + 0.11, + 0.014, + 0.012, + 0.012, + 0.012, + 0.012, + 0.029, + 0.027, + 0.027, + 0.027, + 0.027, + 0.031, + 0.029, + 0.029, + 0.028, + 0.028, + 0.055, + 0.052, + 0.052, + 0.051, + 0.052, + 0.038, + 0.037, + 0.037, + 0.044, + 0.037, + 0.072, + 0.078, + 0.069, + 0.078, + 0.075, + 0.025, + 0.023, + 0.023, + 0.023, + 0.023, + 0.095, + 0.085, + 0.084, + 0.085, + 0.084, + 0.051, + 0.049, + 0.049, + 0.049, + 0.049, + 0.07, + 0.069, + 0.069, + 0.068, + 0.105, + 0.044, + 0.036, + 0.035, + 0.034, + 0.034, + 0.043, + 0.038, + 0.037, + 0.037, + 0.037, + 0.074, + 0.068, + 0.067, + 0.068, + 0.069, + 0.106, + 0.102, + 0.1, + 0.104, + 0.1, + 0.02, + 0.019, + 0.018, + 0.018, + 0.018, + 0.016, + 0.015, + 0.015, + 0.015, + 0.015 + ] + }, + { + "repo": "axum", + "language": "rust", + "tool": "bm25", + "index_ms": 32.7, + "p50_ms": 0.15, + "p90_ms": 0.25, + "p95_ms": 0.36, + "p99_ms": 0.45, + "latencies_ms": [ + 0.222, + 0.2, + 0.2, + 0.2, + 0.199, + 0.219, + 0.212, + 0.212, + 0.218, + 0.215, + 0.167, + 0.157, + 0.172, + 0.162, + 0.158, + 0.129, + 0.127, + 0.126, + 0.128, + 0.125, + 0.113, + 0.109, + 0.108, + 0.111, + 0.108, + 0.165, + 0.161, + 0.159, + 0.158, + 0.161, + 0.261, + 0.253, + 0.486, + 0.3, + 0.259, + 0.433, + 0.449, + 0.386, + 0.355, + 0.398, + 0.158, + 0.134, + 0.132, + 0.13, + 0.142, + 0.135, + 0.128, + 0.126, + 0.139, + 0.136, + 0.155, + 0.147, + 0.147, + 0.15, + 0.146, + 0.088, + 0.082, + 0.082, + 0.085, + 0.081, + 0.26, + 0.253, + 0.25, + 0.25, + 0.248, + 0.189, + 0.181, + 0.185, + 0.179, + 0.181, + 0.116, + 0.119, + 0.119, + 0.106, + 0.13, + 0.211, + 0.19, + 0.19, + 0.189, + 0.19, + 0.126, + 0.11, + 0.114, + 0.111, + 0.11, + 0.081, + 0.077, + 0.084, + 0.089, + 0.076, + 0.132, + 0.126, + 0.125, + 0.127, + 0.135, + 0.11, + 0.106, + 0.106, + 0.105, + 0.105 + ] + }, + { + "repo": "http4s", + "language": "scala", + "tool": "bm25", + "index_ms": 70.3, + "p50_ms": 0.22, + "p90_ms": 0.63, + "p95_ms": 1.3, + "p99_ms": 1.56, + "latencies_ms": [ + 0.383, + 0.343, + 0.34, + 0.34, + 0.382, + 0.432, + 0.414, + 0.419, + 0.43, + 0.438, + 0.166, + 0.16, + 0.161, + 0.17, + 0.163, + 0.136, + 0.133, + 0.132, + 0.133, + 0.132, + 0.197, + 0.191, + 0.192, + 0.2, + 0.192, + 0.123, + 0.119, + 0.119, + 0.119, + 0.12, + 0.554, + 0.553, + 0.55, + 0.548, + 0.557, + 0.401, + 0.396, + 0.399, + 0.389, + 0.427, + 0.329, + 0.299, + 0.309, + 0.309, + 0.299, + 0.233, + 0.23, + 0.231, + 0.235, + 0.246, + 0.127, + 0.123, + 0.125, + 0.125, + 0.124, + 0.297, + 0.291, + 0.289, + 0.29, + 0.294, + 1.576, + 1.549, + 1.557, + 1.523, + 1.536, + 0.155, + 0.149, + 0.149, + 0.15, + 0.147, + 1.287, + 1.284, + 1.288, + 1.291, + 1.291, + 0.143, + 0.115, + 0.113, + 0.113, + 0.111, + 0.154, + 0.157, + 0.147, + 0.15, + 0.153, + 0.301, + 0.292, + 0.309, + 0.302, + 0.31, + 0.051, + 0.056, + 0.049, + 0.049, + 0.046, + 0.037, + 0.037, + 0.037, + 0.036, + 0.036 + ] + }, + { + "repo": "alamofire", + "language": "swift", + "tool": "bm25", + "index_ms": 40.7, + "p50_ms": 0.19, + "p90_ms": 0.4, + "p95_ms": 0.41, + "p99_ms": 0.44, + "latencies_ms": [ + 0.406, + 0.387, + 0.39, + 0.391, + 0.387, + 0.36, + 0.357, + 0.354, + 0.357, + 0.356, + 0.238, + 0.21, + 0.201, + 0.197, + 0.196, + 0.179, + 0.179, + 0.179, + 0.178, + 0.193, + 0.218, + 0.213, + 0.215, + 0.215, + 0.22, + 0.031, + 0.031, + 0.03, + 0.03, + 0.029, + 0.07, + 0.071, + 0.072, + 0.072, + 0.073, + 0.253, + 0.25, + 0.242, + 0.241, + 0.243, + 0.226, + 0.213, + 0.211, + 0.216, + 0.214, + 0.191, + 0.176, + 0.174, + 0.174, + 0.178, + 0.384, + 0.383, + 0.379, + 0.374, + 0.381, + 0.308, + 0.306, + 0.304, + 0.317, + 0.348, + 0.073, + 0.067, + 0.064, + 0.06, + 0.064, + 0.154, + 0.143, + 0.142, + 0.144, + 0.143, + 0.048, + 0.042, + 0.041, + 0.041, + 0.043, + 0.048, + 0.047, + 0.061, + 0.047, + 0.046, + 0.086, + 0.069, + 0.068, + 0.083, + 0.069, + 0.401, + 0.406, + 0.402, + 0.404, + 0.401, + 0.449, + 0.441, + 0.441, + 0.437, + 0.433, + 0.129, + 0.124, + 0.122, + 0.124, + 0.126 + ] + }, + { + "repo": "trpc", + "language": "typescript", + "tool": "bm25", + "index_ms": 21.6, + "p50_ms": 0.12, + "p90_ms": 0.2, + "p95_ms": 0.23, + "p99_ms": 0.28, + "latencies_ms": [ + 0.212, + 0.196, + 0.211, + 0.201, + 0.203, + 0.034, + 0.032, + 0.033, + 0.034, + 0.033, + 0.067, + 0.062, + 0.063, + 0.063, + 0.063, + 0.19, + 0.186, + 0.184, + 0.185, + 0.188, + 0.074, + 0.071, + 0.07, + 0.07, + 0.07, + 0.14, + 0.137, + 0.134, + 0.136, + 0.134, + 0.104, + 0.101, + 0.1, + 0.1, + 0.099, + 0.163, + 0.164, + 0.159, + 0.161, + 0.159, + 0.29, + 0.283, + 0.281, + 0.279, + 0.278, + 0.047, + 0.044, + 0.043, + 0.043, + 0.043, + 0.203, + 0.199, + 0.199, + 0.197, + 0.23, + 0.176, + 0.162, + 0.154, + 0.154, + 0.155, + 0.106, + 0.102, + 0.101, + 0.101, + 0.103, + 0.069, + 0.066, + 0.065, + 0.068, + 0.064, + 0.024, + 0.022, + 0.023, + 0.024, + 0.023, + 0.063, + 0.059, + 0.063, + 0.063, + 0.062, + 0.047, + 0.047, + 0.047, + 0.048, + 0.047, + 0.142, + 0.141, + 0.142, + 0.141, + 0.14, + 0.173, + 0.172, + 0.17, + 0.169, + 0.171, + 0.133, + 0.128, + 0.128, + 0.128, + 0.127 + ] + }, + { + "repo": "zls", + "language": "zig", + "tool": "bm25", + "index_ms": 76.0, + "p50_ms": 0.2, + "p90_ms": 0.43, + "p95_ms": 0.46, + "p99_ms": 0.49, + "latencies_ms": [ + 0.513, + 0.489, + 0.464, + 0.457, + 0.46, + 0.383, + 0.374, + 0.374, + 0.405, + 0.376, + 0.129, + 0.122, + 0.123, + 0.121, + 0.121, + 0.272, + 0.262, + 0.261, + 0.26, + 0.26, + 0.194, + 0.185, + 0.227, + 0.195, + 0.182, + 0.327, + 0.306, + 0.305, + 0.304, + 0.311, + 0.129, + 0.125, + 0.122, + 0.127, + 0.123, + 0.108, + 0.106, + 0.108, + 0.106, + 0.107, + 0.285, + 0.274, + 0.274, + 0.271, + 0.272, + 0.466, + 0.445, + 0.447, + 0.448, + 0.439, + 0.162, + 0.152, + 0.151, + 0.15, + 0.15, + 0.16, + 0.156, + 0.155, + 0.155, + 0.155, + 0.255, + 0.246, + 0.244, + 0.246, + 0.243, + 0.061, + 0.056, + 0.055, + 0.055, + 0.055, + 0.258, + 0.218, + 0.203, + 0.2, + 0.201, + 0.219, + 0.212, + 0.222, + 0.214, + 0.218, + 0.066, + 0.062, + 0.061, + 0.061, + 0.061, + 0.423, + 0.426, + 0.411, + 0.409, + 0.41, + 0.205, + 0.198, + 0.194, + 0.198, + 0.195, + 0.157, + 0.151, + 0.148, + 0.149, + 0.15 + ] + } + ] +} diff --git a/benchmarks/results/token-efficiency-0e2d85215c7e.json b/benchmarks/results/token-efficiency-0e2d85215c7e.json new file mode 100644 index 000000000..3449cb40f --- /dev/null +++ b/benchmarks/results/token-efficiency-0e2d85215c7e.json @@ -0,0 +1,323 @@ +{ + "tool": "token-efficiency", + "tokenizer": "cl100k_base", + "budgets": [ + 500, + 1000, + 2000, + 4000, + 8000, + 16000, + 32000 + ], + "n_queries": 1251, + "recall_at_budget": { + "semble": { + "500": 0.8416, + "1000": 0.9233, + "2000": 0.9668, + "4000": 0.988, + "8000": 0.9948, + "16000": 0.9952, + "32000": 0.9952 + }, + "grep+read": { + "500": 0.0, + "1000": 0.0072, + "2000": 0.0228, + "4000": 0.042, + "8000": 0.0755, + "16000": 0.0987, + "32000": 0.1251 + }, + "grep-kw+read": { + "500": 0.0008, + "1000": 0.008, + "2000": 0.0372, + "4000": 0.0859, + "8000": 0.2074, + "16000": 0.3736, + "32000": 0.5833 + } + }, + "first_hit_reduction": { + "grep+read": { + "n_paired": 189.0, + "median_semble_tokens": 136.0, + "median_other_tokens": 7864.0, + "median_reduction": 0.9842, + "mean_reduction": 0.9659, + "semble_better_pct": 1.0, + "expected_cost_at_cap": 29683.4972, + "expected_cost_cap": 32000.0, + "expected_cost_ratio_vs_semble": 85.3374 + }, + "grep-kw+read": { + "n_paired": 1214.0, + "median_semble_tokens": 167.0, + "median_other_tokens": 19428.5, + "median_reduction": 0.9906, + "mean_reduction": 0.9766, + "semble_better_pct": 1.0, + "expected_cost_at_cap": 45587.0328, + "expected_cost_cap": 32000.0, + "expected_cost_ratio_vs_semble": 131.0586 + }, + "semble": { + "expected_cost_at_cap": 347.8369, + "expected_cost_cap": 32000.0 + } + }, + "plot": { + "budgets": [ + 100, + 114, + 130, + 149, + 170, + 194, + 222, + 253, + 289, + 331, + 378, + 431, + 493, + 563, + 643, + 735, + 839, + 959, + 1095, + 1251, + 1429, + 1633, + 1865, + 2131, + 2434, + 2780, + 3176, + 3628, + 4144, + 4734, + 5407, + 6176, + 7055, + 8059, + 9205, + 10515, + 12011, + 13720, + 15672, + 17902, + 20448, + 23358, + 26681, + 30476, + 34812, + 39765, + 45422, + 51884, + 59265, + 67697, + 77328, + 88329, + 100895, + 115249, + 131645, + 150374, + 171766, + 196203, + 224116, + 255999 + ], + "recall": { + "semble": [ + 0.1516, + 0.1868, + 0.2419, + 0.3355, + 0.4712, + 0.5895, + 0.6551, + 0.6859, + 0.7166, + 0.7473, + 0.7896, + 0.82, + 0.8396, + 0.8608, + 0.8797, + 0.8925, + 0.9077, + 0.9177, + 0.9313, + 0.9416, + 0.9484, + 0.9539, + 0.9622, + 0.9676, + 0.9724, + 0.9752, + 0.9808, + 0.984, + 0.9888, + 0.9908, + 0.994, + 0.994, + 0.9944, + 0.9948, + 0.9948, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952, + 0.9952 + ], + "grep+read": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0008, + 0.0032, + 0.0056, + 0.0064, + 0.0072, + 0.008, + 0.0124, + 0.014, + 0.0172, + 0.022, + 0.026, + 0.0288, + 0.0328, + 0.0336, + 0.038, + 0.0428, + 0.0476, + 0.0516, + 0.0568, + 0.0651, + 0.0763, + 0.0787, + 0.0811, + 0.0879, + 0.0943, + 0.0971, + 0.1055, + 0.1123, + 0.1187, + 0.1211, + 0.1251, + 0.1259, + 0.1311, + 0.1371, + 0.1379, + 0.1411, + 0.1427, + 0.1435, + 0.1459, + 0.1471, + 0.1483, + 0.1483, + 0.1491, + 0.1491, + 0.1491, + 0.1491, + 0.1491 + ], + "grep-kw+read": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0008, + 0.0008, + 0.0008, + 0.0008, + 0.0008, + 0.0008, + 0.0016, + 0.0032, + 0.0052, + 0.0068, + 0.008, + 0.01, + 0.016, + 0.0176, + 0.0252, + 0.0348, + 0.0432, + 0.0512, + 0.0608, + 0.0635, + 0.0783, + 0.0903, + 0.1135, + 0.1307, + 0.1527, + 0.1795, + 0.2086, + 0.2341, + 0.272, + 0.3008, + 0.332, + 0.3636, + 0.4271, + 0.4599, + 0.5043, + 0.5441, + 0.5757, + 0.6013, + 0.6343, + 0.6751, + 0.7264, + 0.7563, + 0.7803, + 0.8104, + 0.832, + 0.85, + 0.8754, + 0.8901, + 0.9029, + 0.9145, + 0.9237, + 0.9305, + 0.9353 + ] + } + } +} diff --git a/benchmarks/speed_benchmark.py b/benchmarks/speed_benchmark.py index 1eee147ad..0e56be67b 100644 --- a/benchmarks/speed_benchmark.py +++ b/benchmarks/speed_benchmark.py @@ -5,15 +5,22 @@ from dataclasses import dataclass import numpy as np -from model2vec import StaticModel from sentence_transformers import SentenceTransformer from benchmarks.data import RepoSpec, Task, available_repo_specs, load_tasks, save_results from benchmarks.tools import run_colgrep_files, run_ripgrep_count from semble import SembleIndex -from semble.types import EmbeddingMatrix +from semble.index.bm25 import BM25 +from semble.index.create import create_index_from_path +from semble.index.dense import load_model +from semble.index.sparse import enrich_for_bm25 +from semble.index.types import make_chunk_id +from semble.tokens import tokenize +from semble.types import ContentType from semble.utils import DEFAULT_MODEL_NAME +_CRE_MODEL_NAME = "nomic-ai/CodeRankEmbed" + # One representative repo per language (medium size, healthy NDCG on the main benchmark). _REPOS: list[str] = [ "nvm", # bash @@ -72,59 +79,103 @@ def p99_ms(self) -> float: return float(np.percentile(self.latencies_ms, 99)) -class _CREWrapper: - """Wrap SentenceTransformer with asymmetric query/document prompts.""" +_UNSET = object() + + +class _AsymmetricWrapper: + """Wrap SentenceTransformer with asymmetric query/document prompts. + + semble only passes use_multiprocessing during index-time calls, never at query time, so its + presence is a reliable query/document discriminator (batch size is not: single-chunk files are + real one-element document batches). + """ def __init__(self, model: SentenceTransformer, max_seq_length: int = 512) -> None: - """Initialise wrapper and cap sequence length to avoid OOM on CPU.""" self._model = model self._model.max_seq_length = max_seq_length - def encode(self, texts: Sequence[str], /) -> EmbeddingMatrix: - """Encode with query prompt for single items, document prompt for batches.""" + def encode(self, texts: Sequence[str], use_multiprocessing: object = _UNSET) -> np.ndarray: + """Encode with the query prompt only when use_multiprocessing wasn't passed.""" text_list = list(texts) - if len(text_list) == 1: + if use_multiprocessing is _UNSET: return self._model.encode(text_list, prompt_name="query", batch_size=1) # type: ignore[return-value] return self._model.encode(text_list, batch_size=1) # type: ignore[return-value] -def _bench_semble( - spec: RepoSpec, tasks: list[Task], model: StaticModel | None -) -> tuple[float, SembleIndex, tuple[float, ...]]: - """Index a repo with semble and measure query latency; return (index_ms, index, latencies_ms).""" +def _bench_coderankembed( + spec: RepoSpec, tasks: list[Task], model: _AsymmetricWrapper +) -> tuple[float, tuple[float, ...]]: + """Index a repo with CodeRankEmbed via semble and measure query latency; return (index_ms, latencies_ms).""" started = time.perf_counter() - index = SembleIndex.from_path(spec.benchmark_dir, model=model) + bm25_index, semantic_index, chunks, _manifest = create_index_from_path( + spec.benchmark_dir, + model=model, # type: ignore[arg-type] + content=(ContentType.CODE,), # type: ignore[arg-type] + ) + index = SembleIndex( + model=model, # type: ignore[arg-type] + bm25_index=bm25_index, + semantic_index=semantic_index, + chunks=chunks, + model_path=_CRE_MODEL_NAME, + root=spec.benchmark_dir, + ) index_ms = (time.perf_counter() - started) * 1000 latencies: list[float] = [] for task in tasks: for _ in range(5): started = time.perf_counter() - index.search(task.query, top_k=_TOP_K, mode="hybrid") + index.search(task.query, top_k=_TOP_K, alpha=1.0) latencies.append((time.perf_counter() - started) * 1000) - return index_ms, index, tuple(latencies) + return index_ms, tuple(latencies) -def _bench_bm25(index: SembleIndex, index_ms: float, tasks: list[Task]) -> tuple[float, tuple[float, ...]]: - """Measure BM25-only query latency on a pre-built semble index; return (index_ms, latencies_ms).""" +def _bench_semble(spec: RepoSpec, tasks: list[Task]) -> tuple[float, SembleIndex, tuple[float, ...]]: + """Index a repo with semble and measure query latency; return (index_ms, index, latencies_ms).""" + started = time.perf_counter() + index = SembleIndex.from_path(spec.benchmark_dir, model_path=DEFAULT_MODEL_NAME) + index_ms = (time.perf_counter() - started) * 1000 latencies: list[float] = [] for task in tasks: for _ in range(5): started = time.perf_counter() - index.search(task.query, top_k=_TOP_K, mode="bm25") + index.search(task.query, top_k=_TOP_K) latencies.append((time.perf_counter() - started) * 1000) - return index_ms, tuple(latencies) + return index_ms, index, tuple(latencies) -def _bench_coderankembed(spec: RepoSpec, tasks: list[Task], model: _CREWrapper) -> tuple[float, tuple[float, ...]]: - """Index a repo with CodeRankEmbed via semble and measure query latency; return (index_ms, latencies_ms).""" +def _bench_bm25(index: SembleIndex, tasks: list[Task]) -> tuple[float, tuple[float, ...]]: + """Build a standalone BM25 index from already-chunked content and measure query latency. + + Reuses semble's chunks (no re-parsing) but times only the BM25-specific work — building a + combined semble index also pays for dense embedding, which dominates and would make this + number meaningless as a "BM25 index time". + """ started = time.perf_counter() - index = SembleIndex.from_path(spec.benchmark_dir, model=model) + bm25_index = BM25() + doc_ids: list[str] = [] + slot = 0 + prev_path = None + for chunk in index.chunks: + slot = slot + 1 if chunk.file_path == prev_path else 0 + prev_path = chunk.file_path + chunk_id = make_chunk_id(chunk.file_path, slot) + bm25_index.add_document(chunk_id, tokenize(enrich_for_bm25(chunk))) + doc_ids.append(chunk_id) + bm25_index.set_doc_order(doc_ids) index_ms = (time.perf_counter() - started) * 1000 + + # Query bm25_index directly — index.search(alpha=0.0) still runs dense encoding and reranking. latencies: list[float] = [] for task in tasks: for _ in range(5): started = time.perf_counter() - index.search(task.query, top_k=_TOP_K, mode="semantic") + tokens = tokenize(task.query) + scores = bm25_index.get_scores(tokens) + if scores.size: + k = min(_TOP_K, scores.size) + partitioned = np.argpartition(-scores, kth=k - 1)[:k] + np.argsort(-scores[partitioned]) latencies.append((time.perf_counter() - started) * 1000) return index_ms, tuple(latencies) @@ -192,12 +243,12 @@ def main() -> None: print("Loading semble model...", file=sys.stderr) started = time.perf_counter() - semble_model = StaticModel.from_pretrained(DEFAULT_MODEL_NAME) + load_model(DEFAULT_MODEL_NAME) # warms semble's internal model cache so repo #1 isn't penalized print(f" loaded in {(time.perf_counter() - started) * 1000:.0f}ms", file=sys.stderr) print("Loading CodeRankEmbed...", file=sys.stderr) started = time.perf_counter() - cre_model = _CREWrapper(SentenceTransformer("nomic-ai/CodeRankEmbed", trust_remote_code=True, device="cpu")) + cre_model = _AsymmetricWrapper(SentenceTransformer(_CRE_MODEL_NAME, trust_remote_code=True, device="cpu")) print(f" loaded in {(time.perf_counter() - started) * 1000:.0f}ms", file=sys.stderr) print(file=sys.stderr) @@ -215,27 +266,27 @@ def main() -> None: spec = specs[repo] tasks = repo_tasks[repo] - index_ms, semble_index, latencies_ms = _bench_semble(spec, tasks, semble_model) + index_ms, semble_index, latencies_ms = _bench_semble(spec, tasks) result = ToolResult( repo=repo, language=spec.language, tool="semble", index_ms=index_ms, latencies_ms=latencies_ms ) all_results.append(result) print(f"{repo:<22} {spec.language:<14} {'semble':<16} {index_ms:>8.0f}ms {_fmt_stats(result)}", file=sys.stderr) - bm25_index_ms, latencies_ms = _bench_bm25(semble_index, index_ms, tasks) + bm25_index_ms, latencies_ms = _bench_bm25(semble_index, tasks) result = ToolResult( repo=repo, language=spec.language, tool="bm25", index_ms=bm25_index_ms, latencies_ms=latencies_ms ) all_results.append(result) print(f"{'':22} {spec.language:<14} {'bm25':<16} {bm25_index_ms:>8.0f}ms {_fmt_stats(result)}", file=sys.stderr) - index_ms, latencies_ms = _bench_coderankembed(spec, tasks, cre_model) + cre_index_ms, latencies_ms = _bench_coderankembed(spec, tasks, cre_model) result = ToolResult( - repo=repo, language=spec.language, tool="coderankembed", index_ms=index_ms, latencies_ms=latencies_ms + repo=repo, language=spec.language, tool="coderankembed", index_ms=cre_index_ms, latencies_ms=latencies_ms ) all_results.append(result) print( - f"{'':22} {spec.language:<14} {'coderankembed':<16} {index_ms:>8.0f}ms {_fmt_stats(result)}", + f"{'':22} {spec.language:<14} {'coderankembed':<16} {cre_index_ms:>8.0f}ms {_fmt_stats(result)}", file=sys.stderr, ) diff --git a/benchmarks/token_efficiency.py b/benchmarks/token_efficiency.py index 77f10edfc..0c24db038 100644 --- a/benchmarks/token_efficiency.py +++ b/benchmarks/token_efficiency.py @@ -12,7 +12,6 @@ import matplotlib.ticker as ticker import numpy as np import tiktoken -from model2vec import StaticModel from benchmarks.data import ( Target, @@ -24,14 +23,14 @@ target_matches_location, ) from semble import SembleIndex -from semble.index.file_walker import DEFAULT_IGNORED_DIRS, FILE_TYPES, FileCategory +from semble.index.dense import load_model +from semble.index.file_walker import _DEFAULT_IGNORED_DIRS as DEFAULT_IGNORED_DIRS +from semble.index.files import get_extensions from semble.ranking.boosting import _STOPWORDS as _SEMBLE_STOPWORDS -from semble.types import Chunk +from semble.types import Chunk, ContentType from semble.utils import DEFAULT_MODEL_NAME -_RG_INCLUDE_GLOBS: tuple[str, ...] = tuple( - f"*{ext}" for ext, spec in FILE_TYPES.items() if spec.category == FileCategory.CODE -) +_RG_INCLUDE_GLOBS: tuple[str, ...] = tuple(f"*{ext}" for ext in get_extensions([ContentType.CODE])) _RG_EXCLUDE_GLOBS: tuple[str, ...] = tuple(f"!{d}" for d in DEFAULT_IGNORED_DIRS) _BUDGETS = (500, 1000, 2000, 4000, 8000, 16000, 32000) @@ -378,7 +377,7 @@ def run_recall(args: argparse.Namespace) -> None: print("Loading tokenizer + model...", file=sys.stderr) enc = tiktoken.get_encoding(_TOKENIZER_NAME) - model = StaticModel.from_pretrained(DEFAULT_MODEL_NAME) + load_model(DEFAULT_MODEL_NAME) # warms semble's internal model cache method_curves: dict[str, MethodCurves] = defaultdict(list) print(f"\n{'Repo':<22} {'Language':<12} {'Tasks':>6} {'Time':>8}", file=sys.stderr) @@ -386,7 +385,7 @@ def run_recall(args: argparse.Namespace) -> None: for repo, repo_task_list in sorted(grouped_tasks(tasks).items()): spec = repo_specs[repo] started = time.perf_counter() - index = SembleIndex.from_path(spec.benchmark_dir, model=model) + index = SembleIndex.from_path(spec.benchmark_dir, model_path=DEFAULT_MODEL_NAME) per_method = _evaluate_repo_recall(index, repo_task_list, spec.benchmark_dir, enc) for m, lst in per_method.items(): method_curves[m].extend(lst) diff --git a/benchmarks/tools.py b/benchmarks/tools.py index 73fc86e00..45e1f97a2 100644 --- a/benchmarks/tools.py +++ b/benchmarks/tools.py @@ -38,6 +38,35 @@ def run_ripgrep_count( return [path for path, _ in entries[:top_k]] +def run_cs( + query: str, + benchmark_dir: Path, + *, + top_k: int, + timeout: int = 30, +) -> list[str]: + """Return file paths from cs (Code Spelunker) JSON output, ranked by its structural BM25 score.""" + cmd = ["cs", query, "--format", "json", "--dir", str(benchmark_dir), "--result-limit", str(top_k)] + try: + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + except subprocess.TimeoutExpired: + return [] + if proc.returncode != 0: + return [] + try: + data = json.loads(proc.stdout) + except json.JSONDecodeError: + return [] + if not data: + return [] + seen: dict[str, None] = {} + for item in data: + location = item.get("location", "") + if location: + seen[location] = None + return list(seen)[:top_k] + + def run_colgrep_files( query: str, benchmark_dir: Path,