# Skill: Debug a Broken Pipeline

Use when a pipeline fails, produces wrong output, loops, or behaves unexpectedly.
Work through the steps in order — most issues are caught by Step 1 or Step 2.

## Step 1 — `arccrew check`

Run before anything else. `check` validates:
- API keys are set in `.env`
- Provider packages are installed
- The model string is valid and the model is reachable

```bash
arccrew check
```

If `check` fails:
- **API key missing** → add it to `.env`
- **Package not found** → `pip install langchain-anthropic` (or the relevant provider)
- **Model not found** → check the model string in `.env` against the cheat sheet in `skills/switch-provider/SKILL.md`
- **Auth error** → key is set but wrong — regenerate it

## Step 2 — Run with `LOG_LEVEL=DEBUG`

Enable full debug tracing to see every ReAct round:
- Tool name called
- Arguments passed (truncated to 120 chars)
- Tool result (first 200 chars)
- Final LLM response text

Set in `.env` (recommended):
```bash
LOG_LEVEL=DEBUG
```

Or inline for a single run:
```bash
LOG_LEVEL=DEBUG arccrew serve --reload
```

With debug logging, you see:
```
[researcher] Starting ReAct — tools: [web_search] max_rounds: 15
[researcher] ── round 1 → web_search({'query': 'AI trends healthcare'})
[researcher]   ← result: 1. **AI in Healthcare 2025**...
[researcher] ── round 2 → web_search({'query': 'machine learning medical diagnosis'})
[researcher]   ← result: 1. **Deep Learning Diagnostics**...
[researcher] ReAct finished — 2 tool call round(s)
[researcher] Completed in 8.34s
```

## Step 3 — `LOG_LEVEL=DEBUG` in `.env` for Framework Logs

For verbose logs from the framework internals (LangGraph node routing, state updates):

```bash
# .env
LOG_LEVEL=DEBUG
```

This activates all `logger.debug(...)` calls across `arccrew/`. Most useful for:
- Seeing exact state values at each node
- Tracking which node the graph is routing to
- Verifying tool names are being resolved correctly

Apply via `.env` (permanent) or prefix: `LOG_LEVEL=DEBUG arccrew serve`

## Common Failures and Fixes

### `extract_json` Returns `{}`

**Symptom**: agent completes successfully but `data = self.extract_json(result) or {"raw": result}`
always returns the `{"raw": result}` fallback.

**Cause**: The LLM is not outputting valid JSON in its final response.

**Diagnose**: In debug output, find the line after "ReAct finished" — it shows the raw
final response. Look at what the LLM actually returned.

**Fixes**:
1. Add "Output ONLY the JSON." as the last line of the Output Format section in the agent's `.md` prompt
2. Show the exact expected JSON schema in the prompt (not just a description)
3. Add "No prose before or after the JSON" to the instructions
4. Check that the agent is not wrapping in a markdown code block — `extract_json` handles
   ` ```json ` blocks, but not multi-line text with JSON embedded at the end

### Max Rounds Exceeded

**Symptom**: `RecursionError` or the agent stops after many tool calls without returning a result.

**Cause**: The agent is stuck in a loop — calling the same tool repeatedly or unable to
decide it is done.

**Diagnose**: In debug output, look at the round-by-round tool calls. Is the agent:
- Calling `web_search` with slightly different queries and never synthesizing?
- Getting tool errors repeatedly and retrying?
- Calling a tool then ignoring the result?

**Fixes**:
1. Check tool return values — if the tool returns empty results, the agent may loop
2. Add to the prompt: "Stop searching after 3 attempts. Summarize what you found."
3. Add a max search count instruction: "Call `web_search` at most 3 times total."
4. Simplify the task — if it is too complex, split it into two agents
5. Increase `DEFAULT_MAX_ROUNDS` or `MY_AGENT_MAX_ROUNDS` in `.env` as a temporary measure

```bash
# .env
MY_AGENT_MAX_ROUNDS=20   # buy more time to diagnose
```

**Circuit breaker** — if the agent is looping because a tool keeps failing (network error,
rate limit, etc.), the circuit breaker activates automatically. It is active by default:
if a tool fails 3 consecutive times, the circuit breaker raises `CircuitBreakerError` from
within the tool. LangGraph catches this and returns it to the agent as an error message —
the agent reads it and stops. This is a **soft abort**: the agent concludes gracefully rather
than crashing the pipeline.

```
# In logs, a tripped circuit breaker looks like:
WARNING [agent_name] Tool 'web_search' aborted after 3 consecutive errors. Last: TimeoutError: ...
# The agent then receives this as a tool result and returns its final answer
```

To adjust or disable the threshold:
```bash
# .env
CIRCUIT_BREAKER_ERRORS=5   # raise threshold
CIRCUIT_BREAKER_ERRORS=0   # disable circuit breaker entirely
```

### Tool Timeout: `web_search`

**Symptom**: `ERROR: Web search timed out.` in tool results.

**Cause**: DuckDuckGo has a 10-second timeout in `arccrew/tools/agent_tools.py`.

**Fix options**:
1. The agent will see the error string and may retry — this is usually fine
2. If it happens consistently, check network connectivity
3. For tests, mock the timeout (see `tests/test_tools.py` for pattern)

### `Command(goto="wrong_node")`

**Symptom**: `GraphLookupError: Node 'wrong_name' not found` or the graph routes to
an unexpected node.

**Cause**: The `goto` string in `Command(goto=...)` does not exactly match a node name
in `graph.add_node("name", ...)`.

**Fix**: Check every `Command(goto=...)` against the node names in `pipeline.py`.
Node names are case-sensitive. Common typos: `"__END__"` instead of `"__end__"`,
`"increment"` vs `"task_increment"`.

```python
# Verify all goto values
# In your agent's execute():
return Command(goto="writer", update={...})  # must match exactly:

# In your pipeline.py:
graph.add_node("writer", writer_node)         # this is the name
```

### State Field Returns `None`

**Symptom**: `state.get("my_field")` returns `None` when you expected a value.

**Cause**: Either the field was never set, the agent wrote to a different key, or the
field is not in `PipelineState` (or your custom state class) so the reducer does not merge it.

**Fixes**:
1. Check the `Command(update={...})` in the preceding agent — what key did it use?
2. Check that the key exactly matches: `"context"` vs `"ctx"`, `"result"` vs `"results"`
3. If the field is a new custom field, ensure it is defined in your `MyState` TypedDict
4. Add `.get("field", default)` with a sensible default in the reading agent

### Agent Never Calls Tools

**Symptom**: "ReAct finished — no tool calls (direct answer)" in the log, but the agent
should have used tools.

**Cause**: The LLM decided it could answer from memory without calling tools.

**Fixes**:
1. Add explicit instruction: "You MUST use `web_search` before answering. Do not rely on memory."
2. Make the task more specific: "Search for current 2025 data on X" vs "Tell me about X"
3. Check that the tool is actually being passed: add a log statement before `run_react()`

## Reading `agent_timings` to Find the Bottleneck

After a pipeline run, `state["agent_timings"]` contains per-agent elapsed times:

```python
final = await graph.ainvoke(initial_state)
timings = final.get("agent_timings", {})
for agent, timing in timings.items():
    print(f"{agent}: {timing['elapsed_seconds']:.1f}s")
```

Or check in debug logs:
```
[researcher] Completed in 8.34s
[writer] Completed in 4.12s
```

If one agent is disproportionately slow:
- Too many tool calls → add round limit or simplify the task
- Slow tool (web search) → network issue or DuckDuckGo throttling
- Large LLM response → check `LLM_MAX_TOKENS`, consider a faster model

## LangSmith Traces

If LangSmith is configured (`LANGSMITH_TRACING=true` in `.env`), every pipeline run
produces a trace at https://smith.langchain.com. Traces show:

- Each node execution as a nested span
- Per-node LLM inputs and outputs
- Token counts and costs per call
- Tool call arguments and results

See `skills/enable-langsmith/SKILL.md` for setup.

## Quick Diagnostic Checklist

```bash
# 1. Dependencies and keys
arccrew check

# 2. Start server with full debug logging
LOG_LEVEL=DEBUG arccrew serve --reload

# 3. Call your pipeline and capture server output to a log
arccrew serve --reload 2>&1 | tee debug.log &
curl -X POST http://localhost:8000/api/runs \
  -H "Content-Type: application/json" \
  -d '{"pipeline": "my_pipeline", "input": "test"}'

# 4. Check the log for these patterns:
grep "extract_json" debug.log    # JSON extraction warnings
grep "Completed in" debug.log     # timing per agent
grep "ERROR" debug.log            # tool errors
grep "round" debug.log            # ReAct rounds
```
