# Monitoring

Workflow monitoring gives you live progress, durable history, and external event delivery without turning the workflow engine into a web-only system.

## Monitoring Surfaces

Workflows currently expose state through four channels:

1. CLI progress output during execution
2. durable SQLite records for runs, steps, and events
3. read-only REST endpoints
4. SSE and outbound notification hooks

These surfaces all come from the same engine events.

## CLI Progress

`aroom workflow run` and `aroom workflow resume` emit live progress while the run is executing. The run ID is printed immediately after creation, before execution begins, so you can start monitoring in a second terminal while the run is in progress.

For background execution, use `aroom workflow run --detach` to create the run and return immediately. Then use `aroom workflow watch <run_id>` to observe progress from any terminal.

### Timeline Display

`aroom workflow watch`, `status`, and `history` use a shared timeline visual vocabulary:

- **Status icons**: `✅` completed, `●` running, `○` pending, `❌` failed, `⏭` skipped, `⏸` paused, `⏳` waiting, `🚫` blocked
- **Friendly durations**: `5m 12s`, `< 1s`, `1h 3m` instead of raw milliseconds
- **Live elapsed**: running steps show updating elapsed time
- **Loop nesting**: steps inside loops are indented under their parent
- **Annotations**: `(cached)`, `(fallback: model)`, `(skipped)`, `(compensation)`, `(no progress)`

When `NO_COLOR` is set, icons degrade to ASCII: `[OK]`, `[..]`, `[  ]`, `[XX]`, `[>>]`, etc.

### Run/Resume Progress

Typical output from `aroom workflow run`:

```text
Starting workflow: issue_delivery v0.1.0
Target: issue:123

  ● gate_issue_current
  ✅ gate_issue_current  < 1s
  ● start_work
    Cloning repository...
    Checking out branch issue-123-add-widget
  ✅ start_work  18s
  ● fast_checks
    Running ruff check src/...
    [stderr] src/foo.py:12: E501 line too long
  ❌ fast_checks  Exit code 1
```

This comes from engine event callbacks using the shared timeline formatters (`status_icon`, `format_duration_ms`). Transcript events (stdout, stderr, assistant messages, tool calls) are emitted during execution and rendered inline under the active step. Stdout lines are shown without a label prefix; stderr lines retain a `[stderr]` label. Content-aware colors are applied based on content (errors in red, paths in blue, etc.). Tool calls render in cyan; tool results render in blue. Tool calls and tool results are shown as compact summaries so workflows that read many files or run verbose shell commands stay readable. Use `--no-transcript` to suppress inline transcript output.

After the live stream, the CLI prints a timeline overview of all steps, a diagnosis hint for non-success runs, and the run id. Non-success runs (including `waiting_for_approval` and `waiting_for_input`) also display **recovery actions** — concrete CLI commands for the next step (approve, deny, respond, resume, cancel). The same recovery actions appear in the web UI run detail panel and are available via the `recovery_actions` field in the `GET /api/workflow-runs/{run_id}` API response.

## Durable Records

The monitoring model starts with SQLite tables:

| Table | Purpose |
|---|---|
| `workflow_runs` | run-level state |
| `workflow_steps` | step execution history |
| `workflow_events` | durable event log |
| `workflow_locks` | target concurrency guard |

That durable state powers:

- `workflow status`
- `workflow history`
- recovery and resume
- the read-only API

`workflow status` output includes:

- **Definition hash:** — the content hash of the workflow definition at the time the run was created; used for drift detection on resume
- **Budget:** — current usage vs. configured limits, e.g. `7/50 steps, 23K/100K tokens, 4:05/60:00`

## Event Bus

Workflow events are also published to Anteroom's existing event bus on:

```text
workflow:{run_id}
```

This matters because the workflow command and the web app may be separate processes. The event bus persists events into `change_log`, and the web app's poller reads them back out for SSE delivery.

That is the bridge that makes CLI-started workflows visible to the web side.

## Event Types

The engine emits workflow events such as:

- `run_started`
- `run_resumed`
- `run_paused`
- `run_completed`
- `run_failed`
- `run_cancelled` — emitted when a run is cancelled via API or CLI (#890)
- `step_started` — includes `attempt` field (1-based) when retry is configured; for steps inside a `parallel` branch, the `step_id` is a qualified ID in the form `branch_id/step_id` (e.g., `fetch_api/call_api`)
- `step_finished`
- `step_failed`
- `step_continued` — emitted when a step with `continue_on: [failed]` fails but the workflow proceeds (#1126); payload includes `result_status`, `summary`, and `continue_on`. Distinct from `step_failed`, which means the run aborted
- `step_skipped` — emitted when a `when` clause evaluates to false; payload includes `reason`
- `step_retry` — emitted before a retry delay; payload includes `attempt`, `max_attempts`, `backoff_seconds`, `error`, and `retry_reason` (`exception`, `failed_status`, or `result_condition`) (#1129)
- `failure_branch_started` — emitted when an `on_failure` branch begins execution (#1127); payload includes `branch_steps` (list of step IDs)
- `failure_branch_completed` — emitted when all `on_failure` branch steps succeed (#1127)
- `failure_branch_failed` — emitted when an `on_failure` branch step fails (#1127); payload includes `failing_branch_step`
- `definition_drift_overridden` — emitted when `--force` overrides a definition hash mismatch on resume; the event payload includes the stored hash and the current hash
- `compensation_started` — emitted when the engine enters compensation phase after a step failure (#978)
- `compensation_step_completed` — emitted when a compensation step succeeds
- `compensation_step_failed` — emitted when a compensation step fails (engine continues best-effort)
- `run_compensated` — emitted when all compensation steps succeed (terminal)
- `run_compensation_failed` — emitted when one or more compensation steps fail (terminal)
- `llm_fallback` — emitted when an LLM step falls back to an alternative model in the fallback chain (#986); payload includes `step_id`, `failed_model`, `error`, `next_index`, and `remaining`
- `step_emitted` — emitted by `emit` steps (#1150); payload includes `level` (info/warning/success/error) and `message`

`run_failed` events may carry a `reason` field of `budget_exceeded:duration`, `budget_exceeded:steps`, or `budget_exceeded:tokens` when a run is stopped by budget enforcement.

## Transcript Events

Workflow steps record durable transcripts of what happened inside each step. Agent runners emit transcript events per-turn during execution. Opaque runners (shell, python_script) emit `transcript_stdout` and `transcript_stderr` per-line during execution, not after process exit. Transcript events are stored in the same `workflow_events` table as all other workflow events and are queryable via the same API.

### Transcript Event Types

| Event type | When emitted | Key payload fields |
|---|---|---|
| `transcript_assistant` | The agent produces a text response (per turn) | `content` (truncated to `max_assistant_chars`), `tokens`, `model` |
| `transcript_tool_call` | The agent invokes a tool | `tool_name`, `arguments` |
| `transcript_tool_result` | A tool returns output | `tool_name`, `status`, `output` (truncated to `max_tool_output_chars`) |
| `transcript_stdout` | A runner step emits stdout | `content` (truncated to `max_stdout_chars`) |
| `transcript_stderr` | A runner step emits stderr | `content` (truncated to `max_stderr_chars`) |
| `transcript_usage` | Cumulative token counts after an API call | `prompt_tokens`, `completion_tokens`, `total_tokens` |

### Storage and Filtering

Transcript events share the `workflow_events` table with all other workflow events. To retrieve only transcript events for a run, filter by event type prefix:

```bash
GET /api/workflow-runs/{run_id}/events?event_type=transcript
```

To retrieve the transcript for a specific step:

```bash
GET /api/workflow-runs/{run_id}/steps/{step_id}/transcript
```

### Configuration

Transcript recording is controlled via the `workflow.transcript` config section. It is enabled by default. See the [config reference](../configuration/config-file.md#workflowtranscript) for all knobs.

Content is truncated per-entry at the configured character limits before storage. Truncation is applied to individual entries, not to the aggregate transcript.

### Run-Level Replay

`aroom workflow replay <run_id>` stitches step-level transcript entries into a full run-level conversation view. It merges transcript events with lifecycle events (`step_started`, `step_finished`, `step_failed`), ordered by insertion id, and renders them with step boundary headers, status indicators, and a summary footer showing total steps and tokens. This is the best command for reviewing what an agent actually said and did across all steps.

### Run Statuses

| Status | Active/Terminal | Description |
|---|---|---|
| `pending` | Active | Created but not yet claimed |
| `claimed` | Active | Claimed by executor, not yet running |
| `running` | Active | Executing steps |
| `paused` | Active | Interrupted, can be resumed |
| `waiting_for_approval` | Active | Paused for tool approval |
| `waiting_for_input` | Active | Paused for human decision |
| `compensating` | Active | Executing rollback steps (#978) |
| `blocked` | Resumable | Gate blocked the run — resume after fixing the condition |
| `completed` | Terminal | All steps succeeded |
| `failed` | Terminal | Step failed, no compensation attempted |
| `cancelled` | Terminal | Operator cancelled |
| `compensated` | Terminal | Step failed, all rollback succeeded (#978) |
| `compensation_failed` | Terminal | Step failed, some rollback also failed (#978) |

On the SSE channel, these are published with `workflow_` prefixes, for example:

- `workflow_run_started`
- `workflow_step_finished`

## Notification Hooks

Workflows can optionally notify external consumers.

Supported transports:

- `webhook`
- `unix_socket`

### Webhooks

Webhook delivery is:

- HTTP `POST`
- JSON payload
- best-effort
- bounded by timeout

Webhook URLs are validated against Anteroom's egress policy before execution:

- `ai.allowed_domains`
- `ai.block_localhost_api`

### Unix Socket Hooks

Unix socket hooks send JSON datagrams to a local Unix socket path.

This is useful when you want local observability or integration without opening network listeners.

### Delivery Guarantees

Hooks are intentionally lightweight:

- hook failures are logged
- hook failures do not fail the workflow
- pending hook deliveries are drained with a bounded timeout before process exit

!!! warning
    Hooks are best-effort notifications, not a transactional outbox. If you need a guaranteed audit source, use the durable workflow tables.

## Token and Cost Data in `budget_usage`

Every run tracks token consumption in `budget_usage`, regardless of whether a budget policy is configured. The `budget_usage` object includes:

- `prompt_tokens` — total prompt tokens across all agent steps
- `completion_tokens` — total completion tokens across all agent steps
- `total_tokens` — sum of prompt + completion
- `estimated_cost_usd` — cost estimate based on configured `cli.usage.model_costs` rates
- `steps_completed` — number of steps that finished
- `elapsed_seconds` — wall-clock duration of the run

`workflow status` displays token and cost data when available. `workflow history` includes a Tokens column per step showing per-step token consumption from `result_artifacts`.

When response caching is enabled on an LLM step, cached responses report `prompt_tokens: 0`, `completion_tokens: 0`, and `total_tokens: 0` — they do not count toward budget enforcement. The original token count is preserved in `result_artifacts.original_tokens` for cost reporting. Steps served from cache show a `(cached)` badge in `workflow status` and `workflow history` output.

## Checkpoints

Checkpoints are durable snapshots of workflow state captured at key boundaries during execution. They record step results, budget usage, and run status at the moment they are created.

### When Checkpoints Are Created

| Event | Label format | Description |
|---|---|---|
| Step completes successfully | `after:<step_id>` | Captured after each successful step |
| All steps complete | `completed` | Final checkpoint for the run |
| Step fails (final attempt) | `failure:<step_id>` | Includes the failing step's error data |
| Gate blocks execution | `blocked:<step_id>` | Includes the blocked step's result |
| Approval pause | `pause:approval:<step_id>` | Run transitions to `waiting_for_approval` |
| Human gate pause | `pause:human_gate:<step_id>` | Run transitions to `waiting_for_input` |
| Compensation starts | `compensation_queue` | Stores the ordered queue of compensate steps, plus completed/failed lists. Resume from `compensating` reloads this checkpoint to skip already-processed entries (#978) |

Failure, block, and pause checkpoints include the current step's result data alongside all prior step results. This means a `failure:step_two` checkpoint contains both `step_one`'s success data and `step_two`'s failure data.

### Checkpoint Contents

Each checkpoint stores:

- **`step_results`** — sanitized map of step ID to result data (status, summary, safe artifact keys). Summaries are truncated to 500 characters. Artifact keys are allowlisted to prevent credential leakage
- **`budget_usage`** — token counts, step count, elapsed time, estimated cost at the time of the checkpoint
- **`run_status`** — the run's status when the checkpoint was created

### Checkpoint API

```
GET /api/workflow-runs/{run_id}/checkpoints
```

Returns a JSON object with a `checkpoints` array. Each entry includes `id`, `label`, `step_id`, `step_results`, `budget_usage`, `run_status`, and `created_at`.

### Checkpoint Visibility in CLI

- **`aroom workflow status <run_id>`** — displays the total checkpoint count for the run
- **`aroom workflow history <run_id>`** — lists all checkpoints with their labels, step IDs, run status, and creation timestamps

### Idempotency Keys in Step History

When a publish step has an `idempotency_key`, the resolved key is persisted on the step record before execution. The `aroom workflow history` command displays the idempotency key in the step details table when present.

## Publish Step Results

Publish steps record their outcome in `result_artifacts`:

- `destination` — adapter-prefixed destination (e.g., `file:/tmp/report.txt`, `webhook:https://example.com/hook`)
- `bytes_written` — for file adapter, the number of bytes written
- `status_code` — for webhook adapter, the HTTP response status code

These artifacts appear in `workflow history` step details.

## Failure Triage Artifacts (#1151)

When a step fails and its output contains parseable failure signals (pytest FAILED lines, Python tracebacks, ruff errors), the engine automatically extracts a `failure_triage` object into `result_artifacts`. This provides normalized, machine-readable fields for repair targeting:

| Field | Type | Description |
|---|---|---|
| `exit_code` | integer | Process exit code (from runner artifacts) |
| `first_failing_test` | string | First FAILED test path (pytest) or first error location (ruff) |
| `first_assertion` | string | Assertion message or error description |
| `first_traceback_file` | string | File path from first traceback frame |
| `first_traceback_line` | integer | Line number from first traceback frame |
| `failure_count` | integer | Total number of failures reported |
| `failure_signature` | string | Stable 16-char SHA-256 prefix for deduplication |

Triage extraction is best-effort — if no structured signal is found, the `failure_triage` key is omitted. The `failure_triage` key is included in `_SAFE_ARTIFACT_KEYS` and survives checkpoint sanitization.

## Structured Outputs in Step History (#1131)

Steps that declare `outputs` have their resolved output values stored in `result_outputs_json` on the step record. These appear in:

- **`workflow history`**: an "Outputs" column in the detail table showing `key=value` pairs
- **`workflow status`** timeline: a `[N outputs]` suffix on the step line
- **Web API**: `result_outputs` field in the step response object at `GET /workflow-runs/{run_id}`
- **Checkpoints**: `result_outputs` is preserved in checkpoint data for resume

## Transcript Rendering

Workflow runs that involve agent steps produce transcript events — assistant messages, tool calls, tool results, stdout/stderr, and usage summaries. These are stored as `transcript_*` event types in the workflow events table.

**CLI:** Use `aroom workflow transcript <run_id> [step_id]` to view transcript events, or pass `--transcript` to `aroom workflow history` to append them after the step and event tables.

**Web UI:** The run detail view in the workflows panel automatically fetches and renders transcript events below the step timeline. Transcript entries use chat-style card formatting consistent with the main chat interface.

## Current Web Scope

The web layer supports run monitoring via REST/SSE and run control via API.

You have:

- REST endpoints for run state, steps, events, and checkpoints
- `POST /api/workflow-runs` to enqueue runs for background execution
- Action endpoints for cancel, resume, approve, deny, respond (#890, #986)
- SSE streaming for real-time progress
- a browser workflows panel with run list, detail view, and live SSE updates

## Typical Monitoring Flows

### Local Developer Flow

1. Start a workflow from the CLI
2. Watch live CLI progress
3. Use `workflow status` or `workflow history` for follow-up inspection
4. Optionally subscribe to SSE from the web side

### External Observer Flow

1. Configure hooks in the workflow definition
2. Start the workflow from the CLI
3. Receive step/run events in your webhook consumer or local socket listener

### Web Observer Flow

1. Query `/api/workflow-runs`
2. Open `/api/workflow-runs/{run_id}`
3. Subscribe to `/api/events?workflow_run_id=<run_id>`

## Stale Lock Recovery

When a workflow worker process dies (crash, OOM kill, `kill -9`), the run may be left in `running` or `compensating` status with its concurrency lock held. The engine automatically reclaims these stale locks:

- **Detection**: a run is stale when its `heartbeat_at` is older than `workflow.lock_reclaim_threshold` seconds (default 120) or `NULL`
- **Normal runs**: transitioned to `failed` with `stop_reason: stale_heartbeat_reclaimed`. Use `aroom workflow resume` to restart from the last checkpoint
- **Compensating runs**: transitioned to `paused` with `stop_reason: stale_heartbeat_reclaimed_during_compensation` to preserve resume-for-rollback safety
- **Step repair**: in-flight steps are marked `interrupted`
- **Lock release**: the target lock is deleted so other runs can proceed

Reclamation runs automatically:

1. On every background executor poll cycle
2. On executor startup recovery
3. On lock contention (acquire fails, reclaim stale holder, retry once)

Use `aroom workflow diagnose <run_id>` to see the `stale_recovery` category diagnosis with next actions.

## What Monitoring Is For

Monitoring is not just UI polish. It supports:

- operator awareness
- recovery after interruption
- audit-friendly reconstruction of what happened
- external automation integrations
- CLI/web parity on the same underlying run state

## Troubleshooting

Three commands help diagnose workflow problems without a full transcript review:

- **`aroom workflow tail <run_id>`** -- shows the last N transcript events (default 20). Use `--stderr` to filter to errors only, or `--step <id>` to focus on a single step.
- **`aroom workflow worktree <run_id>`** -- for issue-oriented workflows, reports the worktree path, branch, clean/dirty/conflicted status, ahead/behind main, and venv presence.
- **`aroom workflow delta <run_id> --since-event <id>`** -- shows all events since a given event ID for incremental monitoring.

When a run is stuck, use `why` for a one-line summary or `diagnose` for the full picture:

```bash
# One-liner: what's blocking this run?
aroom workflow why <run_id>

# Full diagnosis with evidence and next actions
aroom workflow diagnose <run_id>
```

For mission-level troubleshooting across multiple items:

```bash
aroom mission why <session_id>
```

See [Workflow Commands](commands.md) for full argument reference.
## See Also

- [Workflow Commands](commands.md)
- [Workflow API Reference](api-reference.md)
- [Workflow Definitions](definitions.md)
