feat(ten-moss): Moss ↔ TEN Framework integration by HarshaNalluru · Pull Request #396 · usemoss/moss

HarshaNalluru · 2026-07-15T07:44:21Z

PR 1 of 2 — `ten-moss` session manager

Deep integration between Moss (sub-10ms on-device semantic search) and the TEN Framework (real-time multimodal conversational AI). This first PR lands the reusable, offline-testable core; the runnable, Moss-wired TEN voice-assistant example follows in PR 2.

What's in this PR

packages/ten-moss/ — a Moss session manager for TEN, built on the Moss Sessions API:

MossSessionManager — wraps a Moss session (await client.session(index_name=...), a local in-process index) and manages a voice agent's session-scoped grounding. Its surface mirrors the Sessions SDK:
- await open() — open the session (create-or-resume)
- await query_context(text) -> str — injection-ready grounding for this turn; "" on blank/no-hit/error (never raises into the caller, so the voice loop never stalls)
- await add_docs(docs) / await get_docs() / await delete_docs(ids) — session document ops
- await push_index() — persist the session to the cloud for durability / cross-agent handoff
- doc_count; disabled via enable_moss=false (constructs no client, all methods no-op)
MossSessionConfig — pydantic model standardizing the moss_* property names (moss_project_id, moss_index_name, moss_model_id, moss_top_k, moss_alpha, moss_context_header, enable_moss) that the TEN app in PR 2 consumes from property.json.
examples/create_index.py — builds a demo index.
Packaging + docs: pyproject.toml (uv), README.md, CHANGELOG.md, CONTRIBUTING.md, .env.example, LICENSE; AGENTS.md entries.

The callable surface is deliberately session-oriented — a control extension asks for grounding, it does not issue retrieval calls.

How it will be used (preview of PR 2)

In the TEN control extension, on each final ASR transcript:

ctx = await session.query_context(text)
await agent.queue_llm_input(f"{ctx}\n\n[Current User Question]\n{text}" if ctx else text)

This mirrors TEN Agent's own memory/RAG integrations (voice-assistant-with-memU).

Testing

15 offline unit tests (uv run pytest tests/), mocking the Moss client + session — no credentials, no network. Covers config defaults, start/context_for/remember/persist/doc_count, context formatting, and the degrade-to-"" paths (blank input, no hits, exception, timeout).
ruff check + ruff format --check clean.

Notes for reviewers

Tests use the repo's mocked-client convention (as in examples/cookbook/langchain/test_integration.py) — deterministic, always CI-green.
The root README Integrations table row is intentionally deferred to PR 2, where it can point at the runnable apps/ten-moss/ demo instead of a helper-only package.

The 2-PR plan

packages/ten-moss/ (this PR) — reusable session-manager core + tests.
apps/ten-moss/ — a full runnable TEN voice assistant with MossSessionManager wired into the control extension (vendored from TEN's voice-assistant example + the Moss delta), a sample knowledge corpus, and user-facing docs.

…entry

Copilot

Pull request overview

Introduces packages/ten-moss/, a reusable Python helper package that wraps the Moss Python SDK for TEN Framework “ambient” retrieval, along with offline unit tests and supporting docs/specs to enable follow-on TEN app wiring in later PRs.

Changes:

Added MossRetrievalStore (load once, per-turn hybrid retrieve, context formatting, config-based construction).
Added MossRetrievalConfig to standardize TEN moss_* property names.
Added offline tests, a demo index creation script, and integration/spec documentation updates.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/ten-moss/uv.lock	Adds a uv lockfile capturing pinned dependencies for the new package.
packages/ten-moss/tests/test_retrieval_store.py	Offline unit tests for config defaults, context formatting, retrieval behavior, and the demo builder.
packages/ten-moss/src/ten_moss/moss_retrieval_store.py	Implements the retrieval store: `load()`, `retrieve()`, formatting, and `from_config()`.
packages/ten-moss/src/ten_moss/config.py	Adds `MossRetrievalConfig` pydantic model for standardized `moss_*` properties.
packages/ten-moss/src/ten_moss/init.py	Package exports for the store/config and selected Moss SDK types.
packages/ten-moss/README.md	Package README describing install/usage/configuration and the demo script.
packages/ten-moss/pyproject.toml	Python packaging metadata, dependencies, and ruff configuration.
packages/ten-moss/LICENSE	BSD-2-Clause license for the new package.
packages/ten-moss/examples/create_index.py	Demo script to create and populate a Moss index (plus `build_documents()` for testability).
packages/ten-moss/CONTRIBUTING.md	Local dev workflow for uv + tests + ruff.
packages/ten-moss/CHANGELOG.md	Initial changelog entry for `0.0.1`.
packages/ten-moss/.gitignore	Package-local ignores for venv/build/test artifacts.
packages/ten-moss/.env.example	Example env vars for the demo index script.
docs/superpowers/specs/2026-07-14-ten-framework-moss-integration-design.md	Design spec for the Moss ↔ TEN integration (PR 1–3 plan).
docs/superpowers/plans/2026-07-14-ten-moss-package.md	Detailed implementation plan for PR 1 (the helper package).
AGENTS.md	Adds `ten-moss/` to the repo’s packages list/table.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-07-15T07:48:54Z

Codex review

The PR adds a small TEN wrapper, but it also regresses Pipecat package metadata and dependency resolution. The new wrapper has one public API mismatch around push_index.

Findings not on changed lines:

packages/pipecat-moss/pyproject.toml:3 BLOCKING This reverts the package metadata from the existing 0.0.5 state back to 0.0.4 and also lowers the Pipecat floor below the version this package had already been bumped to support:

version = "0.0.4"
...
"pipecat-ai>=1.1.0"

That can make a release fail as a duplicate/older version and lets consumers resolve older Pipecat APIs than the package was most recently validated against. Restore the prior metadata, or bump forward instead:

version = "0.0.5"  # or 0.0.6
"pipecat-ai>=1.5.0"
"pipecat-ai[runner]>=1.5.0"

apps/pipecat-moss/ollama-local/pyproject.toml:7 BLOCKING Dropping the lower bound makes the committed lock resolve this example to pipecat-ai==0.0.107 instead of the previous 1.5.x line:

"pipecat-ai[webrtc,daily,silero,deepgram,openai,cartesia,runner]",

That leaves the app running against pre-1.x Pipecat packages while the code imports modern runner/transport modules. Restore the lower bound here and in the other Pipecat app manifests that made the same change, then regenerate the locks:

"pipecat-ai[webrtc,daily,silero,deepgram,openai,cartesia,runner]>=1.5.0"

…s API - Rename MossRetrievalStore -> MossSessionManager, MossRetrievalConfig -> MossSessionConfig - Build on client.session() (local in-process index) instead of load_index/query - Public API: start / context_for / remember / persist / doc_count; query is inlined - Add moss_model_id config; update README, CHANGELOG, AGENTS.md

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

- Rename methods to mirror the SDK: start->open, context_for->query_context, remember->add_docs, persist->push_index; add get_docs/delete_docs - Respect enable_moss: disabled managers construct no client and no-op - Redact user query text from failure logs (log length + exception type) - Bump moss>=1.7.1 (Sessions API floor) and refresh uv.lock - Update README to new names; note spec/plan supersessions

HarshaNalluru · 2026-07-15T19:56:31Z

Pushed review fixes (also folded in a maintainer request to align names with the Moss Sessions SDK).

Fixed

moss floor too loose → bumped to moss>=1.7.1 (first floor with the Sessions API) and refreshed uv.lock.
from_config ignored enable_moss → a disabled config now constructs no client and every method is a safe no-op (new test test_disabled_config_no_client_and_noop).
Logging the raw user query (PII) → failure logs now record only query_len + exception type, never the utterance.
README dead link to apps/ten-moss/ → reworded to reference the companion app PR rather than a path absent in this PR.
Stale spec/plan docs (live-index testing, "no write-back", old class names) → added a "supersedes" revision note: tests are offline/mocked (no creds), write-back is now in scope via the Sessions API, and MossRetrievalStore→MossSessionManager.
API naming → mirrored the Sessions SDK: open(), add_docs(), get_docs(), delete_docs(), push_index(), doc_count, plus query_context(text) -> str for injection-ready grounding.

Not changed (with reasoning)

except Exception swallowing CancelledError — verified: since Py 3.8 asyncio.CancelledError derives from BaseException, so except Exception does not catch it. Our floor is 3.10, so cancellation already propagates; no change needed.
self._loaded unused — obsolete after the session-manager reframe; that field no longer exists.

Out of scope

The Pipecat version findings (pipecat-moss 0.0.4, ollama-local floor) aren't part of this PR — no Pipecat files are in the diff.

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

- CHANGELOG: list the actual API (open/query_context/add_docs/.../push_index) - examples/create_index.py: make python-dotenv import optional (dev-only dep) - query_context: re-raise asyncio.CancelledError explicitly - spec: reconcile Non-Goals (write-back) and Testing (offline/mocked) body text

HarshaNalluru · 2026-07-15T22:35:27Z

Follow-up round — the remaining threads (incl. the fresh re-review) are addressed; resolving them:

CHANGELOG now lists the real surface (open/query_context/add_docs/get_docs/delete_docs/push_index/doc_count).
examples/create_index.py — python-dotenv import is now optional, so the script runs after a plain pip install ten-moss (dotenv is dev-only).
query_context now explicitly raises asyncio.CancelledError before the catch-all (it was already BaseException-derived and uncaught on 3.10+, but this makes intent unambiguous).
Spec body — reconciled the Non-Goals ("write-back" is now available/unused-in-demo) and Testing (offline/mocked, no creds) sections, not just the top revision note.
PR description updated to the current API surface.

Earlier round: moss>=1.7.1, enable_moss respected (no client + no-ops when disabled), PII-safe failure logs, README dead link, class rename. 17 offline tests green, ruff clean.

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.

Per review — internal design/plan process docs don't belong in the repo.

cubic-dev-ai

All reported issues were addressed across 14 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

- Model project key as SecretStr (masked in reprs/logs); unwrap in from_config - Validate moss_top_k (>0), moss_alpha (0-1), moss_max_context_chars (>=0) - Unspecified model_id now adopts the stored index's model (was forced moss-minilm) - Reject model_id='custom' (needs a query embedding this manager can't supply) - Cap injected grounding via moss_max_context_chars (default 2000) - push_index() returns the SDK result instead of discarding it - README: python-dotenv note for the demo; document new config semantics

HarshaNalluru · 2026-07-18T08:04:34Z

Addressed the latest review (cubic + codex). All fixed:

P1 — open() forced moss-minilm, breaking resume of a differently-modeled index: model_id is now optional; unset ("") omits it so Moss adopts the stored index's model (SDK default for a fresh index).
P2 — project key exposed in reprs/logs: moss_project_key is now a SecretStr, unwrapped only in from_config().
P2 — no validation of top_k/alpha: pydantic Field constraints (gt=0, 0.0–1.0); invalid config raises at construction.
P2 — model_id="custom" can never ground: open() now rejects it with a clear error (it needs a query embedding this text-only manager can't supply).
P2 — large docs blow the context/latency budget: new moss_max_context_chars (default 2000) caps the injected block.
CONSIDER — push_index() discarded its result: now returns the SDK job/result; annotation loosened to Any.
P3 — README demo .env wouldn't load without python-dotenv: README now says to pip install python-dotenv (or export the vars).

23 offline tests green (added coverage for the model-id omission, custom rejection, range validation, SecretStr unwrap, push_index return, and the char budget); ruff clean. Resolving the threads.

cubic-dev-ai

All reported issues were addressed across 5 files (changes from recent commits).

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

Previously the budget counted only document entries, so the header + separators (and an arbitrarily long custom header) could push the injected block over moss_max_context_chars. Now the cap bounds the entire formatted string.

HarshaNalluru · 2026-07-20T18:30:05Z

Fixed — moss_max_context_chars now bounds the entire formatted block, header and separators included (a long custom header is truncated too rather than bypassing the cap). Added a test asserting the whole block ≤ cap and a second asserting a 500-char header still can't exceed a 50-char budget. 24 offline tests green, ruff clean. Resolving.

Follow-up to #396: make `ten-moss` **releasable** to PyPI. Release pipeline + package changes only — the usage/latency example is split into #454. ## What - **`.github/workflows/publish-ten-moss.yml`** — `workflow_dispatch` release, mirrors `publish-pipecat-moss.yml`: reads version from pyproject, builds, smoke-tests `import ten_moss`, publishes to PyPI (`PYPI_API_TOKEN`), tags `ten-moss-v<version>`. - **`MossSessionManager.last_time_taken_ms`** — exposes the engine-measured retrieval time from `SearchResult.time_taken_ms` (set on each `query_context`; `None` before first query / on error). Additive; all 24 offline tests pass. - Version **0.0.1 → 0.1.0**; CHANGELOG + README (API row). ## Release steps (after merge) 1. Merge this PR. 2. Run the **Publish ten-moss** workflow (`workflow_dispatch`). 3. Verify `pip install ten-moss==0.1.0` and `import ten_moss`. Depends on `moss>=1.7.1` (1.7.2 published), `pydantic>=2`, `loguru>=0.7`. --- Example / latency breakdown: **#454** (stacked on this PR).

…ant app (#399) ## PR 2 of 2 — `apps/ten-moss/`: TEN voice assistant with Moss session grounding The runnable half of the Moss ↔ [TEN Framework](https://github.com/ten-framework/ten-framework) integration. A real-time voice agent that, on each final ASR transcript, asks a **Moss session** for session-scoped grounding (~1–10ms, in-process) and injects it into the LLM prompt before the model responds. > **Depends on #396 — merge that first.** This branch's `main_python` imports > `ten_moss` (`MossSessionManager`), which ships in #396. On `main` (both merged) > the import resolves; in isolation this PR won't import `ten_moss`. ### The Moss delta (the part to review) The `tenapp/` baseline is vendored verbatim from TEN's `voice-assistant` example; the Moss integration is a small, localized delta in the `main_python` control extension: - **`config.py`** — `MainControlConfig(MossSessionConfig)` picks up the `moss_*` properties. - **`extension.py` `on_init`** — opens the session, best-effort: ```python if self.config.enable_moss and self.config.moss_index_name: self.moss = MossSessionManager.from_config(self.config) await self.moss.open() ``` - **`extension.py` `_on_asr_result`** (final turn) — grounds the turn (guarded so a grounding failure can't drop the turn): ```python ctx = await self.moss.query_context(event.text) llm_input = f"{ctx}\n\n[Current User Question]\n{event.text}" if ctx else event.text await self.agent.queue_llm_input(llm_input) ``` - **`manifest.json`** declares the `moss_*` properties; **`requirements.txt`** documents the `ten-moss` install (not a hard PyPI pin, since it isn't published yet). - **`tenapp/property.json`** sets `moss_*` on the `main_control` node (env-substituted). Plus a sample corpus (`data/knowledge.jsonl`) and `create_index.py` to build the demo index. ### Graph `agora_rtc → streamid_adapter → stt(deepgram) → main_control(main_python) → llm(openai) → tts(elevenlabs) → agora_rtc`, with `message_collector2` for transcripts. Identical to TEN's `voice-assistant` graph minus the weather tool. ### Provenance / license `tenapp/` (graph, `main_python`, agent runtime, scripts) is vendored from ten-framework at commit [`c385d27`](https://github.com/ten-framework/ten-framework/tree/c385d2724a1f3e6ac4ee0b81fcc7dada8346c0e0/ai_agents/agents/examples/voice-assistant) under **Apache-2.0** (headers preserved). Only the Moss delta above — plus two small correctness patches (`decorators.py` annotation, defensive `session_id` parse) — is applied on top. The repo-level TEN monorepo harness (Taskfile/Dockerfile/playground/server) is intentionally **not** vendored; this example is designed to run inside a TEN Framework checkout (it references shared `ten_packages` via `../../../`). See the app README. ### Validation Static checks all pass: JSON parse (4 files), JSONL corpus (10 docs), `py_compile` (all 11 modules incl. the vendored runtime), import graph resolves, `create_index.load_documents()` returns 10 docs. **Not** run end-to-end in CI — it needs the TEN toolchain (tman/Docker) plus Agora/Deepgram/OpenAI/ElevenLabs keys. Documented for manual runs in the app README. The `ten-moss` logic itself is covered by the offline unit tests in #396. ### Running (summary) ```bash cp .env.example .env # Moss + provider keys python create_index.py # build the demo index (from this dir; needs only the Moss SDK) ``` Then drop `tenapp/` into a TEN Framework checkout and run it with TEN's own tooling (after `uv pip install -e /path/to/moss/packages/ten-moss`). Full steps in the app README.

HarshaNalluru added 6 commits July 14, 2026 19:19

docs: design spec for Moss <-> TEN Framework integration

0807040

docs: implementation plan for ten-moss package (PR 1)

ae4b774

feat(ten-moss): scaffold package and MossRetrievalConfig

5c975a3

feat(ten-moss): add MossRetrievalStore with format_context

e6c0762

feat(ten-moss): add async load, retrieve, and from_config

6cbd4dc

docs(ten-moss): add create-index example, docs, and repo integration …

d39fe67

…entry

Copilot AI review requested due to automatic review settings July 15, 2026 07:44

HarshaNalluru requested a review from r4ghu as a code owner July 15, 2026 07:44

Copilot started reviewing on behalf of HarshaNalluru July 15, 2026 07:44 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

HarshaNalluru added 2 commits July 15, 2026 12:08

docs: note session-manager reframe in design spec

fb66fca

Copilot AI review requested due to automatic review settings July 15, 2026 19:09

HarshaNalluru changed the title ~~feat(ten-moss): Moss ↔ TEN Framework integration — PR 1: reusable retrieval package~~ feat(ten-moss): Moss ↔ TEN Framework integration — PR 1: session manager package Jul 15, 2026

Copilot started reviewing on behalf of HarshaNalluru July 15, 2026 19:09 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

HarshaNalluru changed the title ~~feat(ten-moss): Moss ↔ TEN Framework integration — PR 1: session manager package~~ feat(ten-moss): Moss ↔ TEN Framework integration Jul 15, 2026

HarshaNalluru mentioned this pull request Jul 15, 2026

feat(ten-moss): Moss ↔ TEN Framework integration — PR 2: voice-assistant app #399

Merged

Copilot AI review requested due to automatic review settings July 15, 2026 19:53

Copilot started reviewing on behalf of HarshaNalluru July 15, 2026 19:54 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread packages/ten-moss/CHANGELOG.md

Comment thread packages/ten-moss/examples/create_index.py Outdated

Comment thread packages/ten-moss/src/ten_moss/moss_session_manager.py

Comment thread packages/ten-moss/README.md

Copilot AI review requested due to automatic review settings July 15, 2026 22:34

Copilot started reviewing on behalf of HarshaNalluru July 15, 2026 22:34 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

docs(ten-moss): reconcile spec/plan body with shipped API + 2-PR plan

2bc5ee6

chore(ten-moss): remove superpowers spec/plan artifacts

7561135

Per review — internal design/plan process docs don't belong in the repo.

github-actions Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread packages/ten-moss/src/ten_moss/moss_session_manager.py Outdated

cubic-dev-ai Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread packages/ten-moss/src/ten_moss/config.py

Comment thread packages/ten-moss/src/ten_moss/moss_session_manager.py Outdated

ashvathsureshkumar approved these changes Jul 20, 2026

View reviewed changes

ashvathsureshkumar merged commit d1f1a27 into main Jul 21, 2026
22 of 24 checks passed

HarshaNalluru mentioned this pull request Jul 21, 2026

chore(ten-moss): release v0.1.0 — publish workflow + package #453

Merged

Uh oh!

Conversation

HarshaNalluru commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR 1 of 2 — ten-moss session manager

What's in this PR

How it will be used (preview of PR 2)

Testing

Notes for reviewers

The 2-PR plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HarshaNalluru commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HarshaNalluru commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HarshaNalluru commented Jul 18, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HarshaNalluru commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

HarshaNalluru commented Jul 15, 2026 •

edited

Loading

PR 1 of 2 — `ten-moss` session manager

github-actions Bot commented Jul 15, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading