# Subagent Coordination Tools

Status: DRAFT v3
Date: 2025-03-15

## Problem

Parent agents can spawn subagents via `ctl_spawn_subagent` but have no way to:
- Wait for a child to complete
- Retrieve a child's output
- Check a child's status
- Cancel a running child

The parent fabricates "child results" since it cannot read them. Subagent dispatch is currently useless.

## Design Principles

1. **Sync by default**: `ctl_delegate` is the primary tool — one call, one result. Minimizes LLM cognitive overhead and fabrication risk.
2. **Async for fan-out**: `ctl_spawn` + `ctl_await` for parallel work. Requires explicit await before results are available.
3. **Runtime guards over prompt engineering**: Validate child_ids, block finish with un-awaited children. Prompting alone cannot prevent fabrication.
4. **No timeout by default**: Subagents may run for minutes. Parent's own budget/turn limits are the implicit upper bound.

## Tool-Factory Integration Contract

All new ctl tools follow the same pattern as existing tools in `create_ctl_tools()` (`ctl_tools.py`):

1. **Implementation function** is `async`, accepts `instance_id: str` as first arg (the calling agent's own ID), plus tool-specific params. Returns `Result[str, AnimaError]`.
2. **`instance_id` is captured via closure** in `create_ctl_tools` — it is NOT exposed in the LLM-facing tool schema. The LLM never sees or passes `instance_id`.
3. **Return value**: The impl function returns `Success(json.dumps({...}))` — a JSON string. On error, returns `Failure(AnimaError)`. The tool factory wrapper in `ctl_tools.py` stringifies failures.
4. **All return envelopes** follow a consistent shape (see Error Envelope below).

## Return Envelope

All tools return JSON strings with a consistent shape:

**Success envelope** (tool-specific fields vary):
```json
{
  "child_id": "inst_abc123",
  "status": "completed|running|failed|killed|cancelled|pending|timeout",
  "summary": "Human-readable one-liner.",
  ...tool-specific fields...
}
```

**Error envelope** (for guard violations, validation errors):
```json
{
  "status": "error",
  "error": "Descriptive error message.",
  "summary": "Same as error field."
}
```

The `summary` field is always present and self-contained. LLMs that fail to parse JSON fields can fall back to reading `summary`.

## Tool Specifications

### ctl_delegate (blocking, sync)

Spawn an ephemeral child, wait for completion, return result. The common path (~80% of subagent use).

```
ctl_delegate(
    task: str,                    # Required. Self-contained instruction for child.
    persona_ref: str | None,      # Child persona file. None = inherit parent's persona_ref.
    context_mode: str | None,     # "isolated" (default) or "shared".
    max_turns: int | None,        # Child turn limit.
    max_tokens: int | None,       # Child token budget.
    timeout: float | None,        # Parent wait timeout in seconds. None = no timeout.
    max_response_chars: int | None, # Truncate response_text to this length. Default: 4000.
) -> str
```

**Behavior**:
1. Spawn ephemeral child (internally calls existing spawn machinery with `lifecycle=ephemeral`, `auto_rm=False`).
2. Poll child's `state.json` every 0.5s until terminal state.
3. Check child PID liveness every 5th poll as crash fallback.
4. On terminal: read `result.json`, return `response_text`.
5. On timeout: return status with `"status": "timeout"`.

**Constraints**:
- Always forces `lifecycle=ephemeral`. Daemon children with ctl_delegate would deadlock (daemon agents do not self-terminate).
- The `lifecycle` parameter is NOT exposed in the tool schema.

**Blocking semantics**: This tool blocks the PydanticAI agent loop via `asyncio.sleep()` in a poll loop, identical to the existing `ctl_suspend_until` pattern (`ctl_tools.py` lines 265-316). The agent cannot process other events during the wait. This is intentional — the parent is semantically "waiting for the result" and should not proceed until it has one.

**Return value** (JSON string):
```json
{
  "child_id": "inst_abc123",
  "status": "completed",
  "response_text": "The analysis found 3 issues...",
  "error": null,
  "summary": "Child inst_abc123 completed successfully."
}
```

**Response truncation**: When `response_text` exceeds `max_response_chars` (default 4000), it is truncated with a note:
```
...[truncated. Full response in ~/.anima/agents/{child_id}/result.json]
```
The full `response_text` is always available in the child's `result.json` file. Set `max_response_chars=None` to disable truncation.

On failure:
```json
{
  "child_id": "inst_abc123",
  "status": "failed",
  "response_text": "Last response before failure if any...",
  "error": "max_turns exceeded",
  "current_turn": 5,
  "debug_hint": "Full logs: anima logs inst_abc123",
  "summary": "Child inst_abc123 failed at turn 5: max_turns exceeded"
}
```

`response_text` on failure = the child's last successful model response before it failed (may be `null` if the child never produced a response). `current_turn` shows how far the child got. `debug_hint` tells the parent (and user) where to find full logs.

### ctl_spawn (non-blocking, async)

Start a child agent in the background. Returns immediately with child_id. Renamed from `ctl_spawn_subagent`.

```
ctl_spawn(
    task: str,                    # Required.
    persona_ref: str | None,      # None = inherit parent's persona_ref.
    context_mode: str | None,
    lifecycle: str | None,        # None = inherit parent's lifecycle (preserves existing default).
    max_turns: int | None,
    max_tokens: int | None,
) -> str
```

**Note**: No `timeout` parameter on spawn. Child execution deadline is set via `JobSpec.timeout` (existing mechanism). The `timeout` concept in this spec always means "how long the parent waits", not "child execution deadline".

**Lifecycle default**: `None` inherits the parent's lifecycle, matching existing `ctl_spawn_subagent` behavior. This is NOT changed. When used with `ctl_await`, callers should explicitly pass `lifecycle="ephemeral"` to ensure the child self-terminates.

**Return value** (JSON string):
```json
{
  "child_id": "inst_abc123",
  "status": "running",
  "summary": "Spawned child inst_abc123 (running). Call ctl_await to get results."
}
```

### ctl_await (blocking, batch wait)

Wait for ALL specified children to reach terminal state. Supports concurrent polling of multiple children.

```
ctl_await(
    child_ids: list[str],         # Required. List of child_ids from ctl_spawn.
    timeout: float | None,        # Parent wait timeout in seconds. None = no timeout.
    max_response_chars: int | None, # Truncate each child's response_text. Default: 4000.
) -> str
```

**Behavior**:
1. Validate all child_ids are in parent's `children` list.
2. Concurrently poll each child's `state.json` every 0.5s.
3. PID liveness check every 5th poll per child.
4. Wait until ALL children reach terminal state (`completed`, `failed`, `killed` — the set defined by `core.lifecycle.is_terminal()`).
5. Read `result.json` for each terminal child.
6. On timeout: return results for completed children + `"status": "pending"` for others.

**Blocking semantics**: Same as `ctl_delegate` — blocks the PydanticAI agent loop via `asyncio.sleep()` poll loop. This is intentional.

**Partial failure**: If child 2 of 3 fails, await continues waiting for the remaining children. All results (including failures) are returned together. The parent LLM decides how to handle partial failure.

**response_text race mitigation**: See "Phase 1: Prerequisite — response_text wiring" below. After the prerequisite is implemented, `response_text` is written atomically before state transitions to terminal, eliminating the race.

**Response truncation**: Same as `ctl_delegate` — each child's `response_text` is truncated per `max_response_chars`.

**Return value** (JSON string):
```json
{
  "results": [
    {"child_id": "inst_abc", "status": "completed", "response_text": "...", "error": null},
    {"child_id": "inst_def", "status": "failed", "response_text": "partial...", "error": "max_turns exceeded", "current_turn": 5, "debug_hint": "Full logs: anima logs inst_def"},
    {"child_id": "inst_ghi", "status": "completed", "response_text": "...", "error": null}
  ],
  "summary": "3 children awaited: 2 completed, 1 failed."
}
```

On timeout:
```json
{
  "results": [
    {"child_id": "inst_abc", "status": "completed", "response_text": "...", "error": null},
    {"child_id": "inst_def", "status": "pending", "response_text": null, "error": null}
  ],
  "summary": "Timeout after 60s. 1 completed, 1 still pending."
}
```

### ctl_poll (non-blocking, status check)

Check a child's current status without waiting. Does NOT return response_text (use `ctl_await` for that).

```
ctl_poll(
    child_id: str,                # Required.
) -> str
```

**Return value** (JSON string):
```json
{
  "child_id": "inst_abc123",
  "status": "running",
  "current_turn": 3,
  "error": null,
  "summary": "Child inst_abc123: running, turn 3."
}
```

### ctl_child_logs (non-blocking, opt-in debug)

Read a child agent's conversation log for debugging. NOT a default tool — parent calls this explicitly when a child's result is unexpected or it needs to understand the child's reasoning.

```
ctl_child_logs(
    child_id: str,                # Required.
    last_n: int | None,           # Return last N conversation events. Default: 10.
) -> str
```

**Behavior**:
1. Validate child_id is in parent's children list.
2. Read child's `log.jsonl`.
3. Filter to conversation events: `user_message`, `response`, `tool_call`, `error`. Exclude infrastructure noise (`created`, `state_transition`, etc.).
4. Return the last `last_n` events.

**Context budget control**: `last_n` defaults to 10 to limit context consumption. Setting `last_n=None` returns all events (use with caution).

**Return value** (JSON string):
```json
{
  "child_id": "inst_abc123",
  "total_events": 47,
  "returned_events": 10,
  "events": [
    {"event": "user_message", "turn": 3, "text": "Analyze auth module..."},
    {"event": "tool_call", "turn": 3, "tool": "read_file", "args": "src/auth.py"},
    {"event": "response", "turn": 3, "text": "Found 3 issues..."},
    {"event": "error", "turn": 4, "text": "max_turns exceeded"}
  ],
  "summary": "Last 10 of 47 events. Child reached turn 4, failed at max_turns."
}
```

### ctl_cancel (non-blocking)

Terminate a child agent. Kills the process tree.

```
ctl_cancel(
    child_id: str,                # Required.
) -> str
```

**Behavior**:
1. Validate child_id is in parent's children list.
2. Kill child agent process tree (reuses existing `kill_agent_tree` from `agent_kill.py`).
3. Return confirmation.

**Interaction with ctl_await**: `ctl_cancel` and `ctl_await` cannot be called concurrently by the same parent — PydanticAI tool calls are sequential. The typical pattern is: await with timeout → cancel stragglers. After cancel, the child is in terminal state (`killed`), so a subsequent await would return immediately.

**Return value** (JSON string):
```json
{
  "child_id": "inst_abc123",
  "status": "cancelled",
  "summary": "Child inst_abc123 cancelled."
}
```

## Child State Directory

Each child agent has a state directory at `~/.anima/agents/{child_id}/`:

```
~/.anima/agents/{child_id}/
  ├── state.json      # Lifecycle state, turn count, token usage, error
  ├── result.json     # AgentResult: response_text, result_ref, final_state
  └── log.jsonl       # Full event log (user_message, response, tool_call, error, ...)
```

- `state.json` is written/updated by the child process throughout its lifecycle. Terminal state is the signal that the child is done.
- `result.json` is written by the child when it finishes (via `ctl_finish` or auto-complete). Contains `response_text` (the child's final output) and optionally `result_ref` (path to a custom output file).
- `log.jsonl` contains one JSON object per line, one per event. This is the full audit trail used by `ctl_child_logs`.

The parent reads these files via the polling loop (state.json, result.json) and via `ctl_child_logs` (log.jsonl). The parent never writes to the child's directory.

## Runtime Guards

### Guard 1: child_id validation
All tools (await, poll, cancel) verify that `child_id` belongs to the calling parent's `AgentStateData.children` list. Invalid IDs return error envelope:
```json
{
  "status": "error",
  "error": "child_id 'inst_xyz' was not spawned by this agent. Your children: ['inst_abc', 'inst_def'].",
  "summary": "child_id 'inst_xyz' was not spawned by this agent. Your children: ['inst_abc', 'inst_def']."
}
```

### Guard 2: ctl_finish orphan check
When a parent calls `ctl_finish`, check all entries in `children` list. For each child, read `state.json` and check `core.lifecycle.is_terminal(state)`. If any child is non-terminal, block finish:
```json
{
  "status": "error",
  "error": "Cannot finish: 2 un-awaited children still running: ['inst_abc', 'inst_def']. Await or cancel them first.",
  "summary": "Cannot finish: 2 un-awaited children still running: ['inst_abc', 'inst_def']. Await or cancel them first."
}
```

This guard applies unconditionally to all children in the list. Children that reached terminal state via `ctl_delegate` (which blocks until completion) will naturally pass the check.

### Guard 3: auto_rm disabled for spawned children
Children created via `ctl_spawn` or `ctl_delegate` MUST have `auto_rm=False`. If the child's state directory is deleted before the parent reads result.json, the result is lost. Auto-rm cleanup is the parent's responsibility after reading results.

## Polling Strategy

| Check | Interval | Purpose |
|-------|----------|---------|
| state.json read | 0.5s | Primary terminal-state detection |
| PID liveness (`os.kill(pid, 0)`) | Every 5th poll (2.5s) | Crash detection (child died without writing terminal state) |
| result.json read | On terminal state | Result retrieval |

On PID death + non-terminal state: synthesize a failed result with `error="Child process died unexpectedly"`.

## File Structure

| File | Responsibility |
|------|---------------|
| `src/anima/shell/ctl_child.py` | **New**. `ctl_poll`, `ctl_cancel`, `ctl_child_logs` implementations. |
| `src/anima/shell/ctl_delegate.py` | **New**. `ctl_delegate`, `ctl_await` implementations + shared poll loop. |
| `src/anima/shell/ctl_tools.py` | Register 5 new tools (ctl_spawn rename handled separately). |
| `src/anima/shell/ctl_spawn.py` | Rename `ctl_spawn_subagent` → `ctl_spawn`, update return format. |
| `src/anima/shell/ctl.py` | Add children guard to `ctl_finish`. |

## Dependencies

Arrows mean "imports from":

```
ctl_tools.py
  → ctl_delegate.py (delegate, await)
  → ctl_child.py (poll, cancel)
  → ctl_spawn.py (spawn)
  → ctl.py (finish, fail)

ctl_delegate.py
  → ctl_spawn.py (for ctl_delegate internal spawn)
  → shell/state.py (read_state, read_result)
  → shell/agent_kill.py (for PID liveness via _is_pid_alive)

ctl_child.py
  → shell/state.py (read_state, read_log)
  → shell/agent_kill.py (kill_agent_tree)
```

No circular dependencies.

## UX Rendering

### ctl_delegate (single task wait)
```
  ◇ delegate: Analyze auth module ── 0:08
  ◇ delegate: Analyze auth module ── 0:08 ✓     (completed, dim green)
  ◇ delegate: Analyze auth module ── 0:08 ✗     (failed, red)
```

### ctl_await (batch wait)
```
  ◇ awaiting 3 tasks ── 0:34 (2/3 done)
    ├ ✓ Analyze auth module           0:08
    ├ ✓ Review error handling         0:27
    └ ⠿ Check test coverage           0:34
```

Collapses to summary when all done:
```
  ◇ 3 tasks completed ── 0:41
```

### ctl_spawn, ctl_poll, ctl_cancel (instant)
Standard tool call indicator: `◆ ctl_spawn "task description"`

## System Prompt Guidance

```markdown
## Subagent Delegation

### Pattern 1: Single Task (most common)
result = ctl_delegate(task="summarize auth module")
# result contains the child's actual output. Use it directly.

### Pattern 2: Parallel Fan-Out
id1 = ctl_spawn(task="analyze module A", lifecycle="ephemeral")
id2 = ctl_spawn(task="analyze module B", lifecycle="ephemeral")
id3 = ctl_spawn(task="analyze module C", lifecycle="ephemeral")
results = ctl_await(child_ids=[id1, id2, id3])

### Pattern 3: Debug a Child (when result is unexpected or failed)
logs = ctl_child_logs(child_id=result.child_id, last_n=15)
# Now you can see the child's reasoning, tool calls, and errors.
# Use this to decide: retry with different instructions? Report to user?

### Rules
- NEVER fabricate child results. Always use ctl_delegate or ctl_await.
- PREFER ctl_delegate for single tasks. Use ctl_spawn only for 2+ parallel tasks.
- Call ctl_await in the SAME turn or NEXT turn after spawning.
- Include all necessary context in the task string — the child has no memory
  of your conversation. 2-5 sentences is typical.
- Before calling ctl_finish, await or cancel all spawned children.
- Do not delegate trivially simple tasks you can do directly. Delegation has overhead.
- Use ctl_child_logs ONLY when you need to debug — it consumes context budget.
```

## Migration Plan

### Phase 1: Prerequisite — response_text wiring

**Problem**: `ctl_finish` (in `ctl.py` lines 127-131) writes `AgentResult` to `result.json` without `response_text`. The agent_sync layer later overwrites with a complete `AgentResult` including `response_text`. This creates a race window.

**Fix**: Modify `ctl_finish` to accept an internal `response_text: str | None` parameter (not exposed in the LLM tool schema). The agent runner (`agent.py`) extracts `response_text` from the last model response (already captured at `agent.py` line 684) and passes it when calling the internal `ctl_finish` implementation. This ensures `result.json` contains `response_text` on the first write, eliminating the race.

**Also**: For the `ctl_fail` path, `response_text` will be `None` (failed agents have no successful response). This is acceptable — `ctl_await` returns `error` for failed children.

### Phase 2: Core Tools
- Implement `ctl_poll` and `ctl_cancel` in `ctl_child.py`
- Implement `ctl_await` poll loop in `ctl_delegate.py`
- Implement `ctl_delegate` (compose spawn + await)
- Register all tools in `ctl_tools.py`
- Add children guard to `ctl_finish`

### Phase 3: UX
- Task wait indicator (`◇ delegate: ... ── elapsed`)
- Await tree renderer (multi-line with per-child status)
- JSON output events for machine consumers

### Phase 4: Rename (optional, breaking)
- Rename `ctl_spawn_subagent` → `ctl_spawn`
- Update tool schema, tests, documentation

## Acceptance Criteria

1. `ctl_delegate` returns child's `response_text` for a successful ephemeral child.
2. `ctl_delegate` truncates `response_text` to `max_response_chars` (default 4000) with pointer to full result.
3. `ctl_delegate` returns `status: "failed"` with `error`, `current_turn`, `response_text` (partial), and `debug_hint`.
4. `ctl_delegate` returns `status: "timeout"` when timeout expires.
5. `ctl_delegate` detects child PID crash and returns synthetic error.
6. `ctl_spawn` returns `child_id` that can be passed to `ctl_await`.
7. `ctl_await` with 3 children returns all 3 results after all complete.
8. `ctl_await` returns partial results (completed + pending) on timeout.
9. `ctl_await` truncates each child's `response_text` per `max_response_chars`.
10. `ctl_poll` returns current state without blocking.
11. `ctl_cancel` kills child process and returns confirmation.
12. `ctl_child_logs` returns last N filtered conversation events from child's log.jsonl.
13. `ctl_child_logs` with `last_n=10` returns at most 10 events.
14. `ctl_finish` rejects with error when un-awaited children exist.
15. All tools reject invalid `child_id` with descriptive error listing actual children.
16. Fan-out pattern: spawn 3 + await 3 takes ~max(child_durations), not sum.

## Deferred to v2

- `ctl_await_any` — wait for first completed child (needs cancel for orphan cleanup pattern)
- Child-to-parent message passing / streaming partial results
- asyncio.Event fast-path wakeup (replace filesystem polling)
- Parallel await multi-line terminal progress (cursor management)
