Headless enhancement: add stream-json as input-format/output-format to support programmatically use

[x22x22]

TLDR

This PR introduces a new architecture for stream-json implementation in packages/cli/src/services/, establishing a controller-based pattern alongside the existing packages/cli/src/streamJson/ implementation.

Known Issue

• Unexpected tool use failure messages
• Subagent fails to execute
• No subagent execution message emitted

Todo Actions

• Type System Alignment: Refine types in types/protocol.ts and streamJson/types.ts, align them with claude-code-like types to gain compatibility

• Control Request Refactoring: Refactor current control request handling code to corresponding controllers (gradual rollout)

  • systemController.ts - System operations refactoring
  • permissionController.ts - Permission request handling refactoring
  • mcpController.ts - MCP server lifecycle refactoring
  • hookController.ts - Extension hooks refactoring

• Entry Point Refactoring: Refactor gemini.tsx and nonInteractiveCli.ts for better support of the arguments matrix:

  • [input-format, output-format] × [text, json, stream-json]
  • Gracefully shut down

• Test Coverage: Ensure E2E tests and unit tests remain green with every task completion

  • Run tests after type system changes
  • Run tests after each controller refactoring
  • Run tests after entry point refactoring

• SDK Alignment: Update SDK to align with CLI changes

  • Protocol message types
  • Control request/response handling
  • Stream event handling
  • Review and expand supported configuration options in SDK

• Comprehensive Testing: Test coverage across all scenarios

  • All input/output format combinations
  • Control request flows
  • Error handling paths
  • MCP integration
  • Permission handling

• Documentation & Examples: Create docs and examples for normal use cases

  • Basic query/response flows
  • Control request usage
  • MCP integration examples
  • Permission handling examples

• Behavior Comparison: Let users know the behavior differences with *

Dive Deeper

Background: Existing Implementation

The previous commits (task1-2-2 through task1-2-5) implemented stream-json support in packages/cli/src/streamJson/:

• types.ts: Message envelopes and type definitions
• writer.ts: StreamJsonWriter for output serialization
• controller.ts: StreamJsonController for control request handling
• input.ts: Input parsing and validation
• session.ts: Session management (runStreamJsonSession)
• io.ts: I/O utilities (extractUserMessageText, writeStreamJsonEnvelope)

What's New: Services Architecture

This PR adds a new architecture in packages/cli/src/services/:

Core Service Layer (packages/cli/src/services/)

• StreamJson (StreamJson.ts): JSON Lines protocol handler

  • Bidirectional communication over Readable/Writable streams
  • Message serialization/deserialization
  • Converts Gemini stream events into CLI protocol messages
  • Handles control requests from external processes

• MessageRouter (MessageRouter.ts): Routes control requests to handlers

  • Dispatches requests by type (system, permission, MCP, hooks)
  • Manages request/response lifecycle
  • Error handling and validation

• Control Framework (services/control/):

  • ControlDispatcher: Orchestrates control request handling
    • Manages controller registration
    • Coordinates request routing
    • Error handling
  • ControlContext: Shared context for controllers
    • Access to config, permissions, MCP servers, hooks
    • State management
  • Controllers (extending BaseController):
    • systemController: System operations (config access, abort, lifecycle)
    • permissionController: Permission request handling
    • mcpController: MCP server lifecycle and tool invocations
    • hookController: Extension hooks

Protocol Definition (packages/cli/src/types/protocol.ts)

• Message type system for CLI ↔ external process communication
• Control request/response types
• Stream event types
• Type guards and validation utilities
• Entry point: nonInteractiveStreamJson.ts for stream-json mode

Configuration Changes

• packages/cli/src/config/config.ts: Stream-json format configuration
• packages/core/src/config/config.ts: Configuration options
• packages/core/src/output/types.ts: Stream-json output types

Testing Infrastructure

Note: The packages/sdk/typescript/ directory contains a minimal prototype SDK used for development and E2E testing of the CLI's stream-json functionality. This SDK is not intended for release and serves only as a test harness to validate the CLI's bidirectional communication capabilities.

The SDK prototype includes tests that exercise the stream-json features:

• Basic query/response flows
• Multi-turn conversation handling
• Control request/response cycles
• Abort and lifecycle management
• MCP server integration
• Permission handling

Reviewer Test Plan

The review should focus on changes made to the CLI and Core. The SDK implementation is currently only used to evaluate the effectiveness of CLI-side changes.

Running Tests

# Run SDK E2E tests
cd packages/sdk/typescript
npm test

# Run with coverage
npm run test:coverage

# Run specific test suites
# Warning: batch runs may instantly hit the rate limit with 429 error
npm test -- abort-and-lifecycle
npm test -- multi-turn

Manual Testing

import { query } from '@qwen-code/sdk-typescript';

const q = query({
  prompt: 'What files are in this directory?',
  options: { cwd: process.cwd() },
});

for await (const message of q) {
  console.log('Received from CLI:', message);
}

await q.close();

Key Areas to Review

New Architecture Implementation:

• services/StreamJson.ts: Protocol implementation, message handling, stream management
  • Compare with streamJson/writer.ts and streamJson/input.ts
• services/MessageRouter.ts: Request routing and validation
  • Compare with streamJson/controller.ts
• services/control/ framework:
  • ControlDispatcher: Request orchestration
  • ControlContext: Shared state
  • Individual controllers: system, permission, MCP, hooks
• types/protocol.ts: Protocol message types
  • Compare with streamJson/types.ts

Integration Points:

• New entry point: nonInteractiveStreamJson.ts
• Integration with existing CLI flow (gemini.tsx)
• Coexistence with existing streamJson/ directory

Testing:

• SDK tests validating the new architecture
• Test coverage across all components

Testing Matrix

	🍏	🪟	🐧
npm run	✅	❓	❓
npx	❓	❓	❓
Docker	❓	❓	❓
Podman	❓	-	-
Seatbelt	❓	-	-

Note: Primary testing done on macOS with npm run.

Linked issues / bugs

This PR introduces a new architecture for stream-json implementation.

Development Context:

Phase 0 (commits task1-2-2 through task1-2-5): Initial stream-json implementation

• Stream-json format support in packages/cli/src/streamJson/
• StreamJsonWriter and controller
• Session management and control requests

Phase 1 (this PR): New services architecture

• New services/ directory with controller pattern
• StreamJson, MessageRouter, ControlDispatcher components
• Protocol type definitions in types/protocol.ts
• Minimal SDK prototype for testing (development only, not for release)

Future Phases:

• Migration path from streamJson/ to services/
• Production-ready SDK development
• Integration into main CLI workflows
• Documentation and examples