dq-toolkit

Local-first documentation and code search for Claude Code. Ask "signature of model_validate in pydantic" or "where is index_repo defined" and the agent answers from a local SQLite index of curated library docs and your own repos — offline, cited, in milliseconds.

Install

One line (macOS/Linux; needs claude, cargo, git):

curl -fsSL https://raw.githubusercontent.com/Magic-Man-us/dq-toolkit/main/install.sh | bash

Or run the two steps install.sh wraps:

claude plugin marketplace add Magic-Man-us/dq-toolkit
claude plugin install dq-toolkit@dq-toolkit

Then:

1. Start Claude Code in any repo. The first session builds the binaries in the background (~1 min; log at ~/.cache/dq/build.log).
2. Run /reload-plugins when the build finishes.
3. Ask a doc or code question — the skills trigger automatically.

Update with claude plugin update dq-toolkit@dq-toolkit; the next session rebuilds if the source changed.

Use

The plugin adds three skills that route your question to the index:

Skill	Ask it things like
doc-search	"signature of field_validator in pydantic", "what options does reqwest::ClientBuilder take"
code-search	"where is open defined", "who calls fetch_url", "what changed in this repo last week"
resolve-doc-source	/resolve-doc-source <library-or-url> — add a new doc set to the index

Quick lookups run inline (~6 ms); heavy multi-step research forks a research/code-research subagent that returns a tight cited answer. A SessionStart hook keeps the active repo's working tree indexed and injects the code/doc catalog into context.

Refresh the code index any time with dq-scan refresh && dq-sql index.

How It Works

Two write paths fill one store; one read path serves the agent.

flowchart LR
    subgraph Code["Code feed"]
        GH["GitHub repos"]
        SCAN["dq-scan refresh"]
        CACHE["~/.cache/dq/repos"]
        IDX["dq-sql index"]
    end

    subgraph Docs["Doc feed"]
        ING["dq-ingest"]
        JSONL[("data/*.jsonl")]
        CFG[("config/doc_sets.toml")]
        RB["dq-sql rebuild"]
    end

    DB[("~/.local/share/dq/code.db")]
    ENGINE["dq-sql<br/>(engine)"]
    MCP["dq MCP<br/>(agent surface)"]
    AGENT["Claude Code agent"]

    GH --> SCAN --> CACHE --> IDX --> DB
    ING --> JSONL
    JSONL --> RB
    CFG --> RB
    RB --> DB

    AGENT -->|"tool call"| MCP
    MCP -->|"dq-sql … --json"| ENGINE
    ENGINE -->|"SQL + FTS5"| DB

    style Code fill:#4a90d9,color:#fff
    style Docs fill:#e67e22,color:#fff
    style DB fill:#e74c3c,color:#fff,stroke:#e74c3c
    style MCP fill:#2ecc71,color:#fff

Loading

Code: dq-scan refresh clones/pulls your GitHub repos into ~/.cache/dq/repos. dq-sql index walks that cache with tree-sitter, extracts symbols, and writes them into code.db as kind='code' rows. The SessionStart hook also indexes the active repo's working tree at its live state.

Docs: dq-ingest fetches and normalizes documentation into data/*.jsonl. config/doc_sets.toml groups those files into named sets. dq-sql rebuild loads them into code.db as kind='doc' rows.

Query: the agent calls the dq MCP tools (search_index, get_record, goto_definition, find_references, explore_symbol, grep_source, ast_query, …); each maps to a dq-sql subcommand and returns its JSON. Everything is read-only except fetch_doc_page, which live-fetches a doc page — but only a url already present in the index. Search is offline; only ingest/refresh and fetch_doc_page touch the network.

The binaries

Binary	Role
dq-mcp	The dq MCP server (stdio JSON-RPC) — the agent's query surface; shells to dq-sql
dq-sql	Engine + indexer for the SQLite store: index, rebuild, search, get, def, refs, grep, ast, sources, schema, export/import
dq-scan	Syncs the GitHub clone cache: dq-scan refresh [--full]
dq-ingest	Ingests docs (url / git / local / pkg) → JSONL, then normalize / validate / quality

Standalone CLI (without the plugin)

Build from source (needs a recent stable Rust toolchain):

git clone https://github.com/Magic-Man-us/dq-toolkit.git && cd dq-toolkit
make plugin                 # build dq-scan, dq-sql, dq-mcp, dq-ingest into bin/
export PATH="$PWD/bin:$PATH"
dq-sql rebuild              # populate the index from config/doc_sets.toml + data/

Then query directly:

dq-sql search "BaseModel validation" --kind doc --json -k 5
dq-sql search "field_validator mode before" --site pydantic --json -k 5
dq-sql def index_repo --json
dq-sql refs DocRecord --site dq-toolkit --json
dq-sql grep "struct .*Config" --lang rust --site dq-toolkit --json
dq-sql sources --kind doc   # list installed doc sets

dq-sql finds the doc root from --docs-root, then $DQ_DOCS_ROOT, then $CLAUDE_PLUGIN_ROOT, then the current directory.

The Store

~/.local/share/dq/code.db (path from XDG_DATA_HOME) is one SQLite + FTS5 store holding both code symbols (kind='code') and documentation records (kind='doc'). Indexing keeps the full structure, not just text: symbol_kind, container/scope, line range, a meta JSON of params/returns/decorators/async, and a relationship graph in edges (inherits, contains). Private source bodies live under ~/.local/share/dq, never in this repo.

Search uses FTS5 with porter stemming (so "validators" matches "validate"), ranks title/signature matches above body mentions, and falls back from strict AND to OR when a multi-word query would otherwise return nothing.

dq-sql schema --json inspects the layout:

Table	Purpose
sources	Registry of every doc set and code repo (kind = doc or code)
symbols	One row per code symbol or doc section (FTS-searchable; structured columns for code)
symbols_fts	FTS5 over signature and body (porter-stemmed)
edges	Symbol relationships: inherits, contains
defs / refs	Definition and reference locations

The DB is a derived cache and self-heals to the current schema: if only derived objects drift (FTS table, indexes, views, triggers), dq-sql recreates just those in place — no re-clone, no re-index. Only a base-table change (or a foreign DB) triggers a full wipe-and-repopulate. Build/refresh:

dq-sql rebuild                    # repopulate both: code (from clones) and docs (from data/*.jsonl)
dq-sql index                      # code only, incremental, HEAD-gated (frequent path)
dq-sql index --repo .             # index one working-tree repo, bypassing HEAD gate
dq-scan refresh && dq-sql index   # full code refresh (clone cache + index)

Trust Model

Ingestion treats recipes (config/doc_sets.toml) and the sources they point at as untrusted. Recipe fields are validated at the parse boundary (crates/dq-engine/src/validate.rs): git targets are scheme-allowlisted and rejected if option-like, docs_path can't traverse, package names are charset-constrained, and every git/tool subprocess runs as an argv array with a -- end-of-options marker. HTTP fetches are http(s)-only and blocked from loopback/internal addresses, on the initial URL and every redirect. The query path is read-only: the MCP shells to dq-sql with bound SQL parameters, and indexed text reaches the agent as data.

Development

make plugin    # build dq-scan, dq-sql, dq-mcp, dq-ingest into bin/
make deploy    # build, then sync binaries/skills/commands/hooks to the plugin cache
make test      # cargo test --workspace
make lint      # cargo fmt + clippy -D warnings
make audit     # cargo-deny: advisories, sources, bans, licenses

After make deploy, run /reload-plugins in active sessions.

CI (.github/workflows/) runs the same gates on every push and PR: ci.yml is build + clippy -D warnings + test; security.yml is OSV-Scanner over Cargo.lock plus cargo-deny, with a weekly re-scan. Dependabot opens grouped weekly cargo/actions updates.

Add a new doc set

/resolve-doc-source <library-or-url> automates this end to end. By hand:

# 1. Ingest
cargo run -p dq-ingest -- git https://github.com/user/library.git --name mylib

# 2. Normalize, validate, quality-check
cargo run -p dq-ingest -- normalize data/mylib.jsonl -i
cargo run -p dq-ingest -- validate data/mylib.jsonl
cargo run -p dq-ingest -- quality data/mylib.jsonl --threshold 50 --fix

# 3. Register in config/doc_sets.toml, then rebuild
dq-sql rebuild

Project Layout

dq-toolkit/
├── crates/
│   ├── dq-engine/     # Shared types: DocRecord, doc_sets.toml parser, recipe validation
│   ├── dq-scan/       # tree-sitter extraction lib + dq-scan clone-cache binary
│   ├── dq-sql/        # SQLite index + query engine (dq-sql CLI, behind the MCP)
│   ├── dq-ingest/     # Ingestion pipeline (fetch, parse, normalize)
│   └── dq-mcp/        # dq MCP server (stdio JSON-RPC), the agent's typed query surface
├── config/
│   └── doc_sets.toml          # Doc set registry (name, JSONL files, metadata)
├── data/
│   └── *.jsonl                # Curated documentation records
├── bin/
│   └── dq-{scan,sql,mcp,ingest}  # Compiled binaries (gitignored; built on install)
├── hooks/
│   └── hooks.json             # SessionStart (build, index, catalog) + usage tracking
├── scripts/                   # Hook scripts (ensure-binaries, launch-mcp, code-index, doc-catalog, usage-*) + dev/CI helpers
├── skills/
│   ├── doc-search/            # Library/API doc lookup
│   ├── code-search/           # Repo-wide code navigation
│   └── resolve-doc-source/    # Add a doc set by measured ingest
├── agents/
│   ├── dq-research.md             # research: cited doc/code retrieval from the dq index
│   ├── dq-code-research.md        # code-research: investigate a repo (commits, bugs, upstream/downstream)
│   └── dq-doc-source-resolver.md  # doc-source-resolver: resolve a library to its doc source
├── .github/
│   ├── workflows/             # ci.yml (build/clippy/test) + security.yml (OSV + cargo-deny)
│   └── dependabot.yml         # Grouped weekly cargo + actions updates
├── deny.toml                  # cargo-deny policy (advisories, sources, bans, licenses)
└── Makefile                   # Build, deploy, test, lint, audit

License

MIT © Magic-Man