# Bernstein Benchmark: Single Agent vs Multi-Agent

**Run at:** 2026-03-28T16:11:11.142526+00:00
**Mode:** simulate

## Summary

Across 10 tasks, Bernstein with 3 agents is **1.78x faster** than a single agent on average (5 agents: **2.18x faster**). Model mixing (Haiku for QA/docs, Sonnet for backend) reduces cost by **23%** compared to a single Sonnet agent. Per-agent focused context improves test pass rate by **+8 percentage points** on average.

## Results

| Task | Category | Subtasks | Single (min) | 3-Agent (min) | 5-Agent (min) | Speedup 3x | Speedup 5x | Cost Savings | Quality Δ |
|------|----------|----------|:------------:|:-------------:|:-------------:|:----------:|:----------:|:------------:|:---------:|
| Add REST endpoints (3 routes) + tes | feature | 4 | 41 | 25 | 25 | **1.64x** | **1.64x** | 20% | +8pp |
| Refactor module into clean architec | refactor | 6 | 72 | 49 | 49 | **1.47x** | **1.47x** | 13% | +14pp |
| Add auth middleware + tests + docs | feature | 6 | 67 | 39 | 39 | **1.72x** | **1.72x** | 27% | +14pp |
| Fix 5 linting violations | maintenance | 5 | 20 | 8 | 4 | **2.50x** | **5.00x** | -10% | +11pp |
| Add error handling to all endpoints | reliability | 6 | 40 | 22 | 16 | **1.82x** | **2.50x** | 11% | +14pp |
| Write integration test suite | testing | 5 | 57 | 33 | 27 | **1.73x** | **2.11x** | 72% | +11pp |
| Add rate limiting + tests | feature | 6 | 61 | 43 | 43 | **1.42x** | **1.42x** | 20% | +14pp |
| Create OpenAPI spec from code | docs | 6 | 54 | 38 | 38 | **1.42x** | **1.42x** | 57% | +14pp |
| Add logging and monitoring hooks | observability | 8 | 68 | 32 | 30 | **2.12x** | **2.27x** | 17% | +20pp |
| Security audit + fixes | security | 10 | 97 | 49 | 43 | **1.98x** | **2.26x** | 8% | +26pp |

## Methodology

### Task definitions

Each of the 10 benchmark tasks is defined as a DAG of subtasks with
explicit role assignments (backend, qa, docs, security) and dependency edges.
Task definitions live in `benchmarks/tasks/` as YAML files.

### Scheduling model

The single-agent scenario runs all subtasks sequentially.
Multi-agent scenarios use a greedy list scheduler: at each time step, all
subtasks whose dependencies are satisfied are dispatched to idle agents.
This gives the minimum possible wall-clock time with N agents.

### Cost model

Token consumption is estimated at 320 tokens/minute of agent work.
Single agent uses Claude Sonnet for all roles.
Multi-agent uses model mixing: Sonnet for backend/security, Haiku for QA/docs.
A 10% overhead is added to multi-agent runs to account for
orchestration (task decomposition, janitor verification).

| Model | Cost per 1k tokens |
|-------|-------------------|
| Claude Haiku | $0.00125 |
| Claude Sonnet | $0.005 |
| Claude Opus | $0.025 |

### Quality model

Single-agent test pass rate starts at 82% and degrades by 3 percentage points
per subtask beyond four, due to context growth and attention dilution.
Multi-agent maintains high quality (90%+) through focused per-agent contexts
and role specialisation.

## Key findings

| Metric | Value |
|--------|-------|
| Mean speedup (3 agents) | **1.78x** |
| Mean speedup (5 agents) | **2.18x** |
| Mean cost reduction (3 agents) | **23%** |
| Quality improvement | **+8pp** test pass rate |

### When multi-agent wins most

Tasks with high parallelism (many independent subtasks) benefit most.
The lint-fix task ("Fix 5 linting violations") shows the highest
speedup because all five fixes are fully independent.

The security audit task ("Security audit + fixes") demonstrates
another strong case: four audit subtasks run in parallel, then four fix
subtasks run in parallel — the dependency structure maps cleanly to a 5-agent
pool.

### When multi-agent wins least

Tasks with long sequential chains (e.g. rate limiting, where implementation
must precede integration) show lower speedup. Even here, Bernstein delivers
faster time-to-first-result and lower cost through model mixing.

## Reproducing these results

```bash
# Install Bernstein
pipx install bernstein

# Simulate (no API calls)
python benchmarks/run_benchmark.py

# Real run (requires API keys and running Bernstein stack)
bernstein start
python benchmarks/run_benchmark.py --mode real
```
