feat: Implement hybrid cache that use in-memory index and milvus based doc store

[rootfs]

What type of PR is this?

This hybrid cache is combination of in-memory index and milvus persistent storage, allowing doc search in router's memory and only retrieving cached response if the similar doc is found, avoiding round-trip to milvus on every chat request. This is especially helpful when semantically similar contents are low.

graph TB
    subgraph "Client Layer"
        Client[LLM Client]
    end
    
    subgraph "Cache Layer"
        HC[Hybrid Cache]
        subgraph "In-Memory Components"
            HNSW[HNSW Graph Index]
            EMB[Embedding Cache]
            IDMAP[ID Mapping]
        end
        subgraph "External Storage"
            MILVUS[(Milvus Vector DB)]
        end
    end
    
    subgraph "LLM Layer"
        LLM[Large Language Model]
    end
    
    Client -->|1. Query| HC
    HC -->|2. Generate Embedding| EMB
    EMB -->|3. Search| HNSW
    HNSW -->|4. Top-K IDs| IDMAP
    IDMAP -->|5. Milvus IDs| MILVUS
    MILVUS -->|6. Documents| HC
    HC -->|7. Cache Hit| Client
    HC -->|8. Cache Miss| LLM
    LLM -->|9. Response| HC
    HC -->|10. Store| MILVUS
    HC -->|11. Update Index| HNSW
    
    style HNSW fill:#90EE90
    style EMB fill:#90EE90
    style IDMAP fill:#90EE90
    style MILVUS fill:#87CEEB

Loading

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Release Notes: Yes/No

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	b929ec5
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/68f93219fdb8bc00097c8e09
😎 Deploy Preview	https://deploy-preview-504--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 config

Owners: @rootfs
Files changed:

• config/config.hybrid.yaml
• config/config.development.yaml
• config/config.yaml

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

• src/semantic-router/pkg/cache/comprehensive_benchmark_test.go
• src/semantic-router/pkg/cache/hybrid_cache.go
• src/semantic-router/pkg/cache/hybrid_cache_test.go
• src/semantic-router/pkg/cache/hybrid_vs_milvus_benchmark_test.go
• src/semantic-router/pkg/cache/large_scale_benchmark_test.go
• src/semantic-router/pkg/cache/simd_benchmark_test.go
• src/semantic-router/pkg/cache/simd_distance_amd64.go
• src/semantic-router/pkg/cache/simd_distance_amd64.s
• src/semantic-router/pkg/cache/simd_distance_generic.go
• src/semantic-router/go.mod
• src/semantic-router/go.sum
• src/semantic-router/pkg/cache/cache_factory.go
• src/semantic-router/pkg/cache/cache_interface.go
• src/semantic-router/pkg/cache/inmemory_cache.go
• src/semantic-router/pkg/cache/inmemory_cache_integration_test.go
• src/semantic-router/pkg/cache/milvus_cache.go

📁 website

Owners: @Xunzhuo, @rootfs, @yuluo-yx
Files changed:

• website/docs/tutorials/semantic-cache/hybrid-cache.md
• website/docs/tutorials/semantic-cache/in-memory-cache.md

📁 Root Directory

Owners: @rootfs, @Xunzhuo
Files changed:

• .github/workflows/pre-commit.yml
• .github/workflows/publish-crate.yml
• .github/workflows/test-and-build.yml
• .pre-commit-config.yaml
• Dockerfile.extproc
• Dockerfile.extproc.cross

📁 candle-binding

Owners: @rootfs
Files changed:

• candle-binding/Cargo.lock
• candle-binding/Cargo.toml

📁 tools

Owners: @yuluo-yx, @rootfs, @Xunzhuo
Files changed:

• tools/make/build-run-test.mk
• tools/make/milvus.mk
• tools/make/rust.mk

---

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

[Copilot]

Pull Request Overview

This PR implements a hybrid cache architecture that combines an in-memory HNSW (Hierarchical Navigable Small World) index for fast similarity search with Milvus vector database for scalable, persistent storage. The hybrid approach provides O(log n) search performance while supporting millions of entries without storing full documents in memory.

Key changes:

• Introduces hybrid cache backend combining HNSW index with Milvus storage
• Adds SIMD-optimized (AVX2/AVX-512) dot product operations for vector similarity calculations
• Implements comprehensive benchmarking suite comparing hybrid vs pure Milvus performance

Reviewed Changes

Copilot reviewed 30 out of 32 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/semantic-router/pkg/cache/hybrid_cache.go	Core hybrid cache implementation with HNSW index and Milvus integration
src/semantic-router/pkg/cache/inmemory_cache.go	Extended in-memory cache with HNSW index support
src/semantic-router/pkg/cache/simd_distance_amd64.go	SIMD-optimized vector operations for x86-64
src/semantic-router/pkg/cache/milvus_cache.go	Added batch operations and GetByID for hybrid cache
website/docs/tutorials/semantic-cache/hybrid-cache.md	Documentation for hybrid cache architecture
tools/make/rust.mk	Added rust-ci target for CI/CD without CUDA
config/config.yaml	Updated configuration with HNSW parameters

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

fmt.Appendf does not exist in the fmt package. This should be fmt.Sprintf to format the string before passing to md5.Sum, or use a bytes.Buffer with fmt.Fprintf.

Suggested change

	id := fmt.Sprintf("%x", md5.Sum(fmt.Appendf(nil, "%s_%s_%d", entry.Model, entry.Query, time.Now().UnixNano())))
	id := fmt.Sprintf("%x", md5.Sum([]byte(fmt.Sprintf("%s_%s_%d", entry.Model, entry.Query, time.Now().UnixNano()))))

[Copilot]

This comment and blank identifier assignment serves no purpose since ctx is a local variable that would be garbage collected anyway. Remove these lines.

Suggested change

	// Suppress context error to avoid noise
	_ = ctx

[Copilot]

The comment references 'Knuth's multiplicative hash' but the constants used (2654435761 and 1013904223) are not from Knuth's algorithm. Knuth's multiplicative hash uses 2654435761 alone with modulo 2^32. Clarify the algorithm or correct the implementation.

Suggested change

	hash = hash*2654435761 + 1013904223 // Knuth's multiplicative hash
	hash = hash * 2654435761 // Knuth's multiplicative hash (multiply only, no addition)

[Copilot]

Hard-coded absolute path is environment-specific and will fail in other environments. Use relative path resolution or require PROJECT_ROOT environment variable to be set.

Suggested change

	projectRoot := "/home/ubuntu/rootfs/back/semantic-router.bak"
	if envRoot := os.Getenv("PROJECT_ROOT"); envRoot != "" {
	projectRoot = envRoot
	projectRoot := os.Getenv("PROJECT_ROOT")
	if projectRoot == "" {
	b.Fatalf("PROJECT_ROOT environment variable must be set for benchmark output directory")

[Copilot]

Hard-coded absolute path is environment-specific and will fail in other environments. Use relative path resolution or require an environment variable.

Suggested change

	// Try absolute path first (for direct test execution)
	configPath := "/home/ubuntu/rootfs/back/semantic-router.bak/config/cache/milvus.yaml"
	if _, err := os.Stat(configPath); err == nil {
	return configPath
	}
	// Try relative from project root (when run via make)
	configPath = "config/cache/milvus.yaml"
	// Check for environment variable first
	if envPath := os.Getenv("MILVUS_CONFIG_PATH"); envPath != "" {
	if _, err := os.Stat(envPath); err == nil {
	return envPath
	}
	}
	// Try relative from project root (when run via make)
	configPath := "config/cache/milvus.yaml"

[Copilot]

Magic numbers (1000, 200, 200) should be defined as named constants to clarify their purpose and make them easier to tune.

[Copilot]

Magic number 16 (max layers) should be defined as a constant to make it configurable and document the architectural limitation.

[Copilot]

Pull Request Overview

Copilot reviewed 30 out of 32 changed files in this pull request and generated 4 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Converting an integer to a rune and then to string produces incorrect benchmark names (will show unicode characters instead of numbers). Use fmt.Sprintf(\"SIMD/%d\", size) instead.

[Copilot]

Same issue as SIMD benchmark: converting integer to rune produces incorrect benchmark names. Use fmt.Sprintf(\"Scalar/%d\", size) instead.

[Copilot]

This workaround silently overrides configuration settings, which could lead to unexpected behavior in production. The YAML parsing issue should be fixed at its root cause rather than working around it. If a workaround is necessary, add environment variable guards to ensure it only applies in test/benchmark scenarios.

Suggested change

	if !config.Development.AutoCreateCollection && !config.Development.DropCollectionOnStartup {
	fmt.Printf("[WARN] Development settings parsed as false, forcing to true for benchmarks\n")
	// Only apply this workaround if SR_BENCHMARK_MODE or SR_TEST_MODE is set
	benchmarkMode := os.Getenv("SR_BENCHMARK_MODE")
	testMode := os.Getenv("SR_TEST_MODE")
	if (benchmarkMode == "1" || benchmarkMode == "true" || testMode == "1" || testMode == "true") &&
	!config.Development.AutoCreateCollection && !config.Development.DropCollectionOnStartup {
	fmt.Printf("[WARN] Development settings parsed as false, forcing to true for benchmarks/tests\n")

[Copilot]

The context is intentionally ignored with a blank assignment. This appears to be dead code to suppress unused variable warnings. Consider removing the context variable or using it properly for timeout/cancellation.

Suggested change

	// Suppress context error to avoid noise
	_ = ctx
	// No match found above threshold, return miss