# Benchmark Results: v0.4.0 Fixture Suite

- Date: 2026-02-15
- Tool version: agentsec 0.4.0
- Commit SHA: `d9a3227`
- Fixture set: 20 fixtures (F-001 through F-020), defined inline in `run_benchmark.py`

## Executive Summary

Results are consistent across all three platforms:

| Platform | Precision | Recall | F1 | Critical Recall | p50 | p95 |
|---|---:|---:|---:|---:|---:|---:|
| Windows 11, Python 3.14.2 | 0.82 | 1.00 | 0.90 | 1.00 | 3.4 ms | 11.4 ms |
| Ubuntu (GitHub Actions), Python 3.12 | 0.82 | 1.00 | 0.90 | 1.00 | 2.3 ms | 30.0 ms |
| macOS ARM (GitHub Actions), Python 3.12 | 0.82 | 1.00 | 0.90 | 1.00 | 2.7 ms | 27.7 ms |

28 true positives, 6 false positives, 0 false negatives across 20 fixtures on each platform.
38 file permission findings (CFS-001/CFS-002) excluded on all platforms (test environment artifact).

## Module Breakdown

| Module | Fixtures | Precision | Recall | F1 | Critical Recall | TP | FP | FN |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| installation | 5 | 0.65 | 1.00 | 0.79 | 1.00 | 11 | 6 | 0 |
| skill | 5 | 1.00 | 1.00 | 1.00 | 1.00 | 6 | 0 | 0 |
| mcp | 4 | 1.00 | 1.00 | 1.00 | 1.00 | 6 | 0 | 0 |
| credential | 3 | 1.00 | 1.00 | 1.00 | 1.00 | 3 | 0 | 0 |
| gate | 3 | 1.00 | 1.00 | 1.00 | 1.00 | 2 | 0 | 0 |

Four of five modules achieve perfect precision and recall. Installation scanner has 6 FPs, all of which are legitimate security findings outside the narrow expected set for each fixture.

## False Positives (All 6)

All FPs come from the installation scanner and are valid findings not in the fixture's expected set:

1. **F-002** - "No WebSocket origin validation configured" - valid for exposed gateway, not in CGW-001/CGW-002 scope
2. **F-002** - "Authentication disabled on non-loopback agent" - overlaps with CGW-002 but counted separately
3. **F-003** - "Exec approvals file missing" (CEX-001) - correct but not in CID-001/CTO-001/CTO-003 expected set
4. **F-003** - "No SSRF protection configured" (CGW-005) - valid check, outside doom combo scope
5. **F-004** - "Exec approvals askFallback set to 'full'" - additional finding beyond expected CEX-002
6. **F-004** - "No SSRF protection configured" (CGW-005) - same check, fires on this fixture too

These represent bonus detection rather than incorrect detection. Expanding the expected set would bring installation precision to 1.00.

## False Negatives

None. All 28 expected findings were detected on every platform.

## Clean Fixture Validation

Fixtures F-001, F-006, F-011, F-015, F-017, F-020 are clean/false-positive-guard fixtures. All produced 0 findings from their target module (excluding file permission checks), confirming no false positive issues on benign configs.

## Repro Commands

```bash
# Install
pip install -e ".[dev]"

# Run full benchmark suite
python docs/benchmarks/run_benchmark.py

# Run individual fixture scan
agentsec scan <fixture_dir> -o json -f output.json --fail-on none
```

## Raw Data

- Windows: `2026-02-15-v0.4.0.json`
- Linux: `2026-02-15-v0.4.0-linux.json`
- macOS: `2026-02-15-v0.4.0-macos.json`

## Limitations

- **File permission findings excluded**: CFS-001/CFS-002 fire in temp dirs due to default umask on all platforms. Excluded from precision/recall counts. In real installations with proper permissions these checks are accurate.
- **Installation FPs are bonus detections**: The 6 FPs are all valid findings not in the narrow expected set.
- **CI runners may differ from production**: GitHub Actions runners have specific configurations that may not match production environments.

