# Talk Streaming Transport Design (Agent <-> talk client)

Status: Approved for implementation
Date: 2026-03-06
Step: `talk-streaming.design-streaming-transport-between-agent-and-talk-client`

## Goals

- Stream partial assistant output to `anima talk` in near real time.
- Preserve existing `anima.interact` behavior for state, logs, and compatibility.
- Preserve tool-call semantics and budget enforcement from current interact loop.
- Keep old agents usable via automatic non-streaming fallback.

## Scope and Existing Surface

- Agent side references:
  - `src/anima/shell/agent.py` (`AgentRunner._process_interact_message`)
  - `src/anima/shell/mcp_agent.py` (`AgentMCPServer`, MCP tools)
- Client side reference:
  - `src/anima/shell/cli_commands/talk_cmd.py` (`run_talk_session`)

## Decision: Tool-Call Tradeoff (A/B/C)

Chosen: **Option A**

- A: Keep `agent.iter()` and stream deltas by observing response growth during the interact loop.
- B: Use `agent.run_stream()` for interact turns (drops tool-call support in interact).
- C: Hybrid choose B when possible, fallback to A/iter otherwise.

Rationale:

1. Current interact path already handles `CallToolsNode`, token usage updates, budget checks, `/stop`, and lifecycle behavior; Option A preserves this with minimal semantic risk.
2. Option B is incompatible with tool-using interact turns, which is a behavior regression for `anima talk`.
3. Option C adds policy/heuristic complexity (tool availability vs actual invocation) and inconsistent UX across turns.

Consequence:

- Streaming unit is text delta from accumulated assistant output (not guaranteed one-token granularity).
- No feature regression for tools, budget, or existing MCP behavior.

## Transport and Protocol

### Endpoint

- Agent exposes SSE route on per-agent MCP server: `GET /stream` (same UDS endpoint currently used for MCP HTTP).
- Route added via FastMCP `@mcp.custom_route("/stream", methods=["GET"])`.

### Correlation Model

- Stream is bound to a single active interact turn.
- `turn` in payload equals interact turn number used by `anima.interact` response.
- Supports concurrent consumer fan-out (up to 8 consumers per agent instance);
  the 9th consumer receives `409` with a small JSON error body.

### Event Schema

SSE `event` values and JSON `data` payloads:

1. `event: chunk`

```json
{
  "turn": 12,
  "seq": 41,
  "chunk": "partial text"
}
```

2. `event: done`

```json
{
  "turn": 12,
  "seq": 42,
  "done": true,
  "stop_reason": "complete",
  "final_response_chars": 1834,
  "usage": {
    "prompt_tokens": 123,
    "completion_tokens": 456,
    "total_tokens": 579
  }
}
```

3. `event: error`

```json
{
  "turn": 12,
  "seq": 42,
  "error": "human readable message",
  "code": "INTERNAL"
}
```

Rules:

- `seq` is strictly increasing within a turn.
- Exactly one terminal event per stream: `done` or `error`.
- `chunk` events are best-effort deltas and may coalesce multiple model tokens.

## Backpressure and Queue Sizing

### Queue design

- Add stream queue on `AgentMCPServer`:
  - type: `asyncio.Queue[StreamEvent]` per consumer
  - `maxsize = 32` per consumer queue
- Event producer: `AgentRunner._process_interact_message`.
- Event consumer: SSE generators bound to `GET /stream` (fan-out per subscriber).

### Why per-consumer 32

- Bounded memory for each attached consumer.
- Large enough to absorb brief terminal/render stalls.
- Keeps tail latency bounded when one consumer slows down.

### Backpressure policy

- Non-terminal events (`chunk`, `tool_call`) are best-effort and dropped when a
  subscriber queue is full.
- If client disconnects, SSE consumer sets a stream-cancel flag; producer stops enqueue attempts and continues normal interact completion.
- Terminal events (`done`/`error`) are never dropped for connected consumers;
  producer retries enqueue until accepted or stream canceled.

## Stop Mid-Stream Semantics

- `/stop` continues to call `anima.stop_turn`.
- Agent consumes stop signal at existing node boundaries (`_consume_stop_turn_request`).
- On stop detection:
  1. stop generating new chunks,
  2. keep already emitted chunks visible,
  3. set final textual response to cancellation marker (existing behavior),
  4. emit terminal `done` with `stop_reason: "stopped"`.

Notes:

- Stop is cooperative, not immediate preemption of model internals; boundary-based cancellation remains unchanged.
- `anima.interact` still returns final response object (possibly cancellation marker), so non-streaming clients remain correct.

## Non-Streaming Fallback

Client behavior in `talk_cmd.py`:

1. Attempt SSE support detection once per session (first turn): open `GET /stream`.
2. If HTTP 404, 405, connection failure, or SSE protocol parse failure, mark `streaming_supported = false`.
3. For `streaming_supported = false`, use current blocking `anima.interact` display path for all subsequent turns.

Rationale:

- One-time probe avoids repeated latency and noisy logs.
- Old agent binaries keep working with no user flags.

## Compatibility: Tokens, Budget, Tools, and Logs

### Tokens and budget

- Keep current token/budget accounting inside `agent.iter()` loop.
- Streaming does not alter budgeting decisions; it only mirrors output progressively.
- `done` event includes usage snapshot for UI/diagnostics; source of truth remains state/log updates.

### Tool calls

- Tool calls continue to run through `CallToolsNode` as today (Option A).
- No suppression of tools in interact turns.

### `set_response` and non-streaming consumers

- Continue calling `set_response(full_text, turn_number)` exactly once at turn completion.
- Existing `anima.interact` response contract remains unchanged.

### Logging

- Existing `response` log entry remains required and unchanged in meaning.
- Optional: add lightweight stream lifecycle log events (`stream_start`, `stream_done`, `stream_error`) for debugging only.

## Error Handling

- Stream transport errors do not fail the interact turn by themselves.
- If SSE path fails mid-turn:
  - continue interact execution,
  - still produce final `anima.interact` response,
  - emit `error` event only if connection is active.
- If interact turn fails internally:
  - emit `error` event when possible,
  - ensure terminal signal to unblock client waiters.

## Implementation Notes (for next steps)

1. `src/anima/shell/mcp_agent.py`
   - add stream queue + stream session state,
   - add `GET /stream` custom route,
   - add event enqueue helpers and terminal event guarantees.

2. `src/anima/shell/agent.py`
   - in `_process_interact_message`, compute text deltas and enqueue `chunk` events,
   - keep existing node traversal, token tracking, budget checks, and `set_response`.

3. `src/anima/shell/cli_commands/talk_cmd.py`
   - run interact call and SSE consumer concurrently,
   - print `chunk` events progressively,
   - keep fallback to blocking response path when streaming unsupported.

## Acceptance Criteria for This Design

- SSE schema fixed (`chunk`, `done`, `error`) with explicit fields above.
- Backpressure policy fixed (bounded per-consumer queues, best-effort non-terminal fan-out, terminal guarantee).
- `/stop` behavior fixed (partial output retained, terminal `done` with `stopped`).
- Fallback behavior fixed (one-time probe, persistent non-streaming mode for older agents).
- Tool-call tradeoff resolved explicitly to Option A.
