# Skill: Write Tests for Agents and Pipelines

Use when you need to add tests for a new agent, tool, or pipeline. Read the existing tests
in `tests/` before writing new ones — follow the same patterns.

## pytest Configuration

From `pytest.ini` (or `pyproject.toml`):

```ini
[pytest]
asyncio_mode = auto
testpaths = tests
```

`asyncio_mode = auto` means async test functions run without `@pytest.mark.asyncio`.

Run tests:
```bash
pytest                             # all tests
pytest tests/test_base_agent.py -v # single file, verbose
pytest -k "test_extract_json"      # single test by name
pytest -k "TestExtractJson"        # entire test class
pytest tests/ -v --tb=short        # all with short tracebacks
```

## The 3 Test Patterns

### Pattern 1: Unit Tests — Mock the LLM, Test Agent Logic

Test `extract_json`, `get_max_rounds`, error handling, and state processing without
running a real LLM. This is the fastest and most reliable pattern.

**How to stub heavy imports** (prevents segfaults from numpy/transformers at collection time):

```python
import sys
from unittest.mock import MagicMock, patch

# Stub heavy imports at collection time
for mod in ("langgraph", "langgraph.prebuilt", "langchain_core", "langchain_core.messages"):
    if mod not in sys.modules:
        sys.modules[mod] = MagicMock()

# Provide symbols used at import time in base.py
lc_msgs = sys.modules.setdefault("langchain_core.messages", MagicMock())
lc_msgs.HumanMessage = MagicMock
lc_msgs.SystemMessage = MagicMock
sys.modules.setdefault("langgraph.prebuilt", MagicMock()).create_react_agent = MagicMock()
sys.modules.setdefault("langgraph.types", MagicMock()).Command = MagicMock

from arccrew.agents.base import BaseAgent  # import after stubbing
```

**Minimal concrete subclass** (no LLM needed):

```python
class DummyAgent(BaseAgent):
    def __init__(self, name: str = "dummy"):
        super().__init__(name=name, use_llm=False)  # use_llm=False skips model init

    @property
    def system_prompt(self) -> str:
        return "You are a dummy agent."
```

**Unit test examples** (from `tests/test_base_agent.py`):

```python
class TestExtractJson:
    def setup_method(self):
        self.agent = DummyAgent()

    def test_plain_json(self):
        result = self.agent.extract_json('{"key": "value"}')
        assert result == {"key": "value"}

    def test_json_code_block(self):
        text = '```json\n{"key": "value"}\n```'
        assert self.agent.extract_json(text) == {"key": "value"}

    def test_invalid_json_returns_empty(self):
        assert self.agent.extract_json("{bad json}") == {}

    def test_no_json_returns_empty(self):
        assert self.agent.extract_json("no json here") == {}

class TestGetMaxRounds:
    def test_global_default_used(self):
        agent = DummyAgent(name="dummy")
        mock_settings = MagicMock()
        mock_settings.default_max_rounds = 15
        mock_settings.dummy_max_rounds = None

        with patch("arccrew.config.get_settings", return_value=mock_settings):
            result = agent.get_max_rounds()
        assert result == 15

    def test_per_agent_env_override(self):
        agent = DummyAgent(name="researcher")
        mock_settings = MagicMock()
        mock_settings.default_max_rounds = 15
        mock_settings.researcher_max_rounds = 5

        with patch("arccrew.config.get_settings", return_value=mock_settings):
            result = agent.get_max_rounds()
        assert result == 5
```

### Pattern 2: Tool Tests — Test `@tool` Functions Directly

Test tool functions by mocking external dependencies (HTTP clients, DDGS, file system).
Do not import tools in the same process as stubbed LangChain — clean up stubs first.

**From `tests/test_tools.py`:**

```python
import asyncio
import sys
from unittest.mock import AsyncMock, MagicMock, patch

# Clean up any LangChain stubs from previous test files
if isinstance(sys.modules.get("langchain_core"), MagicMock):
    for key in [k for k in sys.modules if k.startswith("langchain_core")]:
        del sys.modules[key]

from arccrew.tools.agent_tools import web_search

def run(coro):
    return asyncio.get_event_loop().run_until_complete(coro)

class TestWebSearch:
    def test_missing_ddgs_returns_error(self):
        with patch.dict("sys.modules", {"ddgs": None}):
            result = run(web_search.ainvoke({"query": "test"}))
        assert "ERROR" in result

    def test_timeout_returns_error(self):
        import asyncio as _asyncio
        with patch("arccrew.tools.agent_tools.asyncio.wait_for", side_effect=_asyncio.TimeoutError):
            result = run(web_search.ainvoke({"query": "test"}))
        assert "timed out" in result.lower()

    def test_successful_results_formatted(self):
        fake_results = [
            {"title": "Result One", "href": "https://example.com", "body": "Snippet."},
        ]
        mock_ddgs = MagicMock()
        with patch.dict("sys.modules", {"ddgs": mock_ddgs}):
            with patch("asyncio.wait_for", new=AsyncMock(return_value=fake_results)):
                result = run(web_search.ainvoke({"query": "AI trends"}))
        assert "Result One" in result
        assert "https://example.com" in result
```

**Test your custom tool** the same way:

```python
from tools.my_tools import my_custom_tool

class TestMyCustomTool:
    def test_valid_input(self):
        with patch("tools.my_tools._call_api") as mock_api:
            mock_api.return_value = {"price": 9.99}
            result = run(my_custom_tool.ainvoke({"product_id": "abc123"}))
        assert "9.99" in result

    def test_api_error_returns_error_string(self):
        with patch("tools.my_tools._call_api", side_effect=Exception("timeout")):
            result = run(my_custom_tool.ainvoke({"product_id": "abc123"}))
        assert "ERROR" in result

    def test_empty_input_returns_error(self):
        result = run(my_custom_tool.ainvoke({"product_id": ""}))
        assert "ERROR" in result
```

### Pattern 3: Integration Tests — Invoke the Full Graph

Test the full pipeline with a real cheap model (Haiku or Gemini Flash). These tests
are slower and require API keys.

```python
"""Integration tests — requires ANTHROPIC_API_KEY in environment."""
import os
import pytest
from datetime import datetime

# Skip if no API key
pytestmark = pytest.mark.skipif(
    not os.environ.get("ANTHROPIC_API_KEY"),
    reason="ANTHROPIC_API_KEY not set"
)

@pytest.mark.asyncio
async def test_pipeline_runs_end_to_end():
    from pipeline import create_pipeline
    from arccrew.orchestrator.state import PipelineState

    graph = create_pipeline()

    initial_state = {
        "messages": [],
        "tasks": [{"description": "Write a one-sentence summary of Python."}],
        "current_task_index": 0,
        "current_agent": "init",
        "context": {},
        "result": {},
        "verification_result": {},
        "review_result": {},
        "results": [],
        "retry_count": 0,
        "retry_history": [],
        "batch_id": "test-001",
        "start_time": datetime.now(),
        "dry_run": False,
    }

    final = await graph.ainvoke(initial_state)

    assert final is not None
    assert len(final.get("results", [])) >= 1
    result = final["results"][0]
    assert result.get("summary") or result.get("raw")

@pytest.mark.asyncio
async def test_retry_loop_retries_on_failure():
    """Verify that the verifier triggers at least one retry."""
    from pipeline import create_pipeline_with_retry

    graph = create_pipeline_with_retry()
    initial_state = {
        "messages": [],
        "tasks": [{"description": "Intentionally simple task."}],
        "current_task_index": 0,
        "current_agent": "init",
        "context": {},
        "result": {},
        "verification_result": {},
        "review_result": {},
        "results": [],
        "retry_count": 0,
        "retry_history": [],
        "batch_id": "test-retry-001",
        "start_time": datetime.now(),
        "dry_run": False,
    }

    final = await graph.ainvoke(initial_state)
    # Verify the pipeline completed without hanging
    assert final is not None
```

## What to Test — Priority List

1. **`extract_json` edge cases** — most important unit test. The agent's behavior
   degrades silently when JSON extraction fails. Test: empty string, plain JSON, code block,
   embedded JSON in prose, invalid JSON.

2. **`get_max_rounds` priority** — explicit override beats per-agent env beats global default.

3. **Tool error returns** — every tool must return error strings, never raise. Test:
   missing dependency, timeout, empty results, malformed input, API failure.

4. **State field access in `execute()`** — if your agent reads `state.get("my_field")`,
   test that it handles `None` gracefully.

5. **`Command(goto=...)` value** — assert the returned `goto` matches the actual node names
   in your graph (typos here cause confusing runtime errors).

6. **Retry loop termination** — if you have `max_retries=3`, test that the pipeline
   completes (does not loop forever) even when the verifier always fails.

## Common Test Issues

| Issue | Fix |
|-------|-----|
| `ImportError: langchain_core` at test collection | Add the sys.modules stub block at the top of the test file |
| `MagicMock.Command` is not the real Command | Use `isinstance(result, MagicMock)` in assertions instead of `isinstance(result, Command)` |
| Async test runs synchronously | Check `pytest.ini` has `asyncio_mode = auto` |
| Integration test hangs | Use `pytest --timeout=60` (install `pytest-timeout`) |
| `KeyError` in agent when reading state | Add `.get()` with default; test with minimal state |

## File Naming Convention

```
tests/
  test_base_agent.py        # existing: BaseAgent unit tests
  test_tools.py             # existing: web_search tool tests
  test_prompt_manager.py    # existing: PromptManager tests
  test_helpers.py           # existing: utility helper tests
  test_my_project_agents.py # new: your agent unit tests
  test_my_project_tools.py  # new: your tool tests
  test_my_project_integration.py  # new: full pipeline integration
```
