Summary
search_graph(query=...) returns unranked noise for any query written in a non-Latin script, and no amount of documentation helps, because docstrings are not part of the full-text index either. On a codebase documented in Russian (or Greek, or Japanese) the BM25 mode is effectively unusable.
Two independent causes, both small:
1. The query tokenizer drops every non-ASCII byte. bm25_build_match() (src/mcp/mcp.c) accepts only a-z, A-Z, 0-9 and _ as token bytes; everything else is a separator. A Cyrillic query is therefore tokenized to nothing, reaches FTS5 as an empty MATCH, and the caller gets whatever the fallback path returns.
2. Docstrings are not indexed. nodes_fts is declared as fts5(name, qualified_name, label, file_path, ...) in src/store/store.c. Identifiers in real projects are overwhelmingly English even when the team writes prose in another language, so the only text that could match a native-language query is exactly the text that never enters the index.
Reproduction
Index any project that has a non-English docstring, e.g.
def _validate_args(name: str, arguments: dict) -> str | None:
"""Провалидировать аргументы по схеме инструмента."""codebase-memory-mcp cli search_graph --project <p> --query 'провалидировать аргументы схеме'
Expected: the documented function.
Actual (v0.9.0): unrelated nodes — in my case .env.example, a Coordinates class and a README section; the target function is nowhere in the results.
The English control query behaves as expected, which is what makes the failure easy to miss.
Suggested fix
- Treat bytes
>= 0x80 as token bytes in bm25_build_match(). No UTF-8 decoding is required — the point is only to keep multi-byte sequences intact and let the FTS5 unicode61 tokenizer apply the real word-boundary rules, which it already does correctly on both sides.
- Add a
docstring column to nodes_fts and populate it from json_extract(properties, '$.docstring') in both backfill paths (pipeline.c, pipeline_incremental.c). Existing databases carry the 4-column table, so the schema probe has to drop and recreate it; the table is contentless, so nothing is lost — the next backfill repopulates it from nodes.
Result after the fix
Same query, same project:
total: 1
search_mode: bm25
...server._validate_args Function src/yandex_biz_mcp/server.py 78-96 -18.89
The English query improves as well — validate arguments schema now ranks _validate_args first, where previously it returned noise: docstrings carry intent that identifiers alone do not.
I have a patch with a regression test ready and will open a PR referencing this issue.
Environment
- v0.9.0 (release build, UI variant), macOS 15.5 arm64
- Reproduced on several projects; the smallest is an 8-tool Python MCP server, 137 nodes
Summary
search_graph(query=...)returns unranked noise for any query written in a non-Latin script, and no amount of documentation helps, because docstrings are not part of the full-text index either. On a codebase documented in Russian (or Greek, or Japanese) the BM25 mode is effectively unusable.Two independent causes, both small:
1. The query tokenizer drops every non-ASCII byte.
bm25_build_match()(src/mcp/mcp.c) accepts onlya-z,A-Z,0-9and_as token bytes; everything else is a separator. A Cyrillic query is therefore tokenized to nothing, reaches FTS5 as an empty MATCH, and the caller gets whatever the fallback path returns.2. Docstrings are not indexed.
nodes_ftsis declared asfts5(name, qualified_name, label, file_path, ...)insrc/store/store.c. Identifiers in real projects are overwhelmingly English even when the team writes prose in another language, so the only text that could match a native-language query is exactly the text that never enters the index.Reproduction
Index any project that has a non-English docstring, e.g.
Expected: the documented function.
Actual (v0.9.0): unrelated nodes — in my case
.env.example, aCoordinatesclass and a README section; the target function is nowhere in the results.The English control query behaves as expected, which is what makes the failure easy to miss.
Suggested fix
>= 0x80as token bytes inbm25_build_match(). No UTF-8 decoding is required — the point is only to keep multi-byte sequences intact and let the FTS5unicode61tokenizer apply the real word-boundary rules, which it already does correctly on both sides.docstringcolumn tonodes_ftsand populate it fromjson_extract(properties, '$.docstring')in both backfill paths (pipeline.c,pipeline_incremental.c). Existing databases carry the 4-column table, so the schema probe has to drop and recreate it; the table is contentless, so nothing is lost — the next backfill repopulates it fromnodes.Result after the fix
Same query, same project:
The English query improves as well —
validate arguments schemanow ranks_validate_argsfirst, where previously it returned noise: docstrings carry intent that identifiers alone do not.I have a patch with a regression test ready and will open a PR referencing this issue.
Environment