# AI-QA-SDK — Complete Build Specification

> **Purpose**: This document is the single source of truth for building ai-qa-sdk.
> Feed this entire file to Claude Code at the start of every session.
> It contains every module, every class, every method, every validation, and every test requirement.

---

## TABLE OF CONTENTS

1. [Project Identity](#1-project-identity)
2. [Architecture Overview](#2-architecture-overview)
3. [Project Setup & Toolchain](#3-project-setup--toolchain)
4. [Module 1: Core Primitives](#4-module-1-core-primitives)
5. [Module 2: Provider Abstraction](#5-module-2-provider-abstraction)
6. [Module 3: Check Registry & Plugin System](#6-module-3-check-registry--plugin-system)
7. [Module 4: Built-in Checks — Deterministic](#7-module-4-built-in-checks--deterministic)
8. [Module 5: Built-in Checks — LLM-Powered](#8-module-5-built-in-checks--llm-powered)
9. [Module 6: Built-in Checks — Agent & RAG](#9-module-6-built-in-checks--agent--rag)
10. [Module 7: Built-in Checks — Safety & Security](#10-module-7-built-in-checks--safety--security)
11. [Module 8: Built-in Checks — Conversation](#11-module-8-built-in-checks--conversation)
12. [Module 9: Custom Check (User-Defined)](#12-module-9-custom-check-user-defined)
13. [Module 10: Runner & Orchestration](#13-module-10-runner--orchestration)
14. [Module 11: The validate() Entrypoint](#14-module-11-the-validate-entrypoint)
15. [Module 12: Dataset Loading](#15-module-12-dataset-loading)
16. [Module 13: Suite Evaluation](#16-module-13-suite-evaluation)
17. [Module 14: Snapshot & Regression Detection](#17-module-14-snapshot--regression-detection)
18. [Module 15: Experiment / A-B Comparison](#18-module-15-experiment--ab-comparison)
19. [Module 16: Caching Layer](#19-module-16-caching-layer)
20. [Module 17: Reporters](#20-module-17-reporters)
21. [Module 18: pytest Plugin](#21-module-18-pytest-plugin)
22. [Module 19: CLI Tool](#22-module-19-cli-tool)
23. [Module 20: Production Monitor](#23-module-20-production-monitor)
24. [Module 21: Adversarial / Red Team](#24-module-21-adversarial--red-team)
25. [Module 22: Synthetic Test Generation](#25-module-22-synthetic-test-generation)
26. [Packaging & Distribution](#26-packaging--distribution)
27. [CI/CD Pipeline](#27-cicd-pipeline)
28. [Documentation Requirements](#28-documentation-requirements)
29. [CLAUDE.md File](#29-claudemd-file)
30. [Session-by-Session Build Plan](#30-session-by-session-build-plan)

---

## 1. PROJECT IDENTITY

```
Name:           ai-qa-sdk
PyPI package:   ai-qa-sdk
Import name:    ai_qa
Description:    Universal AI validation SDK — validate LLM outputs, RAG pipelines,
                agent workflows with a single function call.
License:        Apache-2.0
Python:         >=3.10
Author:         <your-name>
Repository:     github.com/<your-username>/ai-qa-sdk
```

### What This Project IS
- A Python library that validates AI-generated outputs
- Framework-agnostic — works with any LLM, any pipeline, any agent
- Pluggable — anyone can add custom checks via decorators or pip-installable plugins
- Usable in tests, CI/CD, and production monitoring

### What This Project IS NOT
- Not a testing framework (no test runner — it integrates with pytest)
- Not an observability platform (no dashboards — it exports data for other tools)
- Not an LLM gateway or proxy
- Not tied to any specific LLM vendor

---

## 2. ARCHITECTURE OVERVIEW

```
┌─────────────────────────────────────────────────────────┐
│                   PUBLIC API SURFACE                      │
│  validate()  |  Suite  |  Experiment  |  Monitor  | CLI  │
├─────────────────────────────────────────────────────────┤
│                   ORCHESTRATION LAYER                     │
│  Runner  |  DatasetLoader  |  Snapshot  |  Cache         │
├─────────────────────────────────────────────────────────┤
│                   CHECKS (PLUGGABLE)                      │
│  hallucination | factuality | relevance | toxicity       │
│  schema | tool_use | trajectory | groundedness           │
│  bias | prompt_injection | pii | context_retention       │
│  consistency | custom                                     │
├─────────────────────────────────────────────────────────┤
│                   AI ENGINE                                │
│  ProviderRegistry | LLMJudge | PromptRegistry | CostTracker│
├─────────────────────────────────────────────────────────┤
│                   CORE PRIMITIVES                         │
│  Context | BaseCheck | CheckResult | ValidationResult    │
├─────────────────────────────────────────────────────────┤
│                   INTEGRATIONS                            │
│  pytest | JUnit | JSON | HTML | LangSmith | Datadog      │
└─────────────────────────────────────────────────────────┘
```

### Design Principles (Non-Negotiable)
1. **Minimal core dependencies** — core installs in <3 seconds, no LLM SDK required
2. **Optional extras** — LLM providers, ML libraries are all opt-in
3. **Sync-first API** — primary API is synchronous; async variants available
4. **Type-safe** — Pydantic models or dataclasses, full type hints, mypy strict
5. **Evidence-driven** — every result explains WHY it passed/failed
6. **Cost-aware** — every LLM call tracks token usage and cost
7. **Cache-friendly** — same input + same prompt version = cached result
8. **Plugin-first** — checks, providers, reporters all use registry pattern
9. **Zero global state** — except a single Config singleton for provider settings
10. **Friendly errors** — every exception tells the user how to fix it

---

## 3. PROJECT SETUP & TOOLCHAIN

### Directory Structure

```
ai-qa-sdk/
├── src/
│   └── ai_qa/
│       ├── __init__.py                 # validate(), configure(), __version__
│       ├── py.typed                    # PEP 561 marker for type checkers
│       ├── _version.py                 # Single source of version
│       ├── exceptions.py               # All custom exceptions
│       │
│       ├── core/
│       │   ├── __init__.py
│       │   ├── context.py              # Context dataclass
│       │   ├── result.py               # CheckResult, ValidationResult
│       │   ├── check.py                # BaseCheck ABC
│       │   ├── runner.py               # Concurrent check executor
│       │   └── config.py               # Global configuration
│       │
│       ├── checks/
│       │   ├── __init__.py             # Registry + auto-import
│       │   ├── hallucination.py        # LLM-powered
│       │   ├── factuality.py           # LLM-powered
│       │   ├── relevance.py            # LLM-powered
│       │   ├── toxicity.py             # LLM-powered
│       │   ├── consistency.py          # LLM-powered (multi-run)
│       │   ├── schema.py               # Deterministic
│       │   ├── regex.py                # Deterministic
│       │   ├── json_format.py          # Deterministic
│       │   ├── length.py               # Deterministic
│       │   ├── tool_use.py             # Agent — single tool call
│       │   ├── trajectory.py           # Agent — multi-step workflow
│       │   ├── groundedness.py         # RAG — claim→source
│       │   ├── context_retention.py    # Conversation — memory across turns
│       │   ├── contradiction.py        # Conversation — self-consistency
│       │   ├── bias.py                 # Fairness — counterfactual
│       │   ├── pii.py                  # Privacy — PII/PHI detection
│       │   ├── prompt_injection.py     # Security — injection resistance
│       │   └── custom.py               # User-defined criteria
│       │
│       ├── providers/
│       │   ├── __init__.py             # get_provider(), ProviderRegistry
│       │   ├── base.py                 # BaseProvider ABC
│       │   ├── anthropic.py            # Claude integration
│       │   ├── openai.py               # OpenAI integration
│       │   └── local.py                # Ollama / vLLM / local
│       │
│       ├── datasets/
│       │   ├── __init__.py
│       │   ├── loader.py               # CSV, JSON, JSONL, YAML loading
│       │   └── generator.py            # Synthetic test generation
│       │
│       ├── evaluation/
│       │   ├── __init__.py
│       │   ├── suite.py                # Suite.evaluate()
│       │   ├── snapshot.py             # Snapshot capture/compare
│       │   └── experiment.py           # A/B model comparison
│       │
│       ├── adversarial/
│       │   ├── __init__.py
│       │   ├── red_team.py             # RedTeam runner
│       │   └── attacks.py              # Attack strategy library
│       │
│       ├── monitor/
│       │   ├── __init__.py
│       │   └── monitor.py              # Production monitoring wrapper
│       │
│       ├── integrations/
│       │   ├── __init__.py
│       │   ├── pytest_plugin.py        # pytest markers + fixtures
│       │   └── reporters.py            # JUnit, JSON, HTML, Console
│       │
│       ├── cache/
│       │   ├── __init__.py
│       │   └── cache.py                # SQLite-backed result cache
│       │
│       └── cli/
│           ├── __init__.py
│           └── main.py                 # CLI commands
│
├── tests/
│   ├── conftest.py                     # Shared fixtures
│   ├── unit/
│   │   ├── test_context.py
│   │   ├── test_result.py
│   │   ├── test_check.py
│   │   ├── test_runner.py
│   │   ├── test_config.py
│   │   ├── test_registry.py
│   │   ├── test_providers.py
│   │   ├── test_cache.py
│   │   ├── checks/
│   │   │   ├── test_hallucination.py
│   │   │   ├── test_factuality.py
│   │   │   ├── test_relevance.py
│   │   │   ├── test_toxicity.py
│   │   │   ├── test_schema.py
│   │   │   ├── test_regex.py
│   │   │   ├── test_tool_use.py
│   │   │   ├── test_groundedness.py
│   │   │   ├── test_custom.py
│   │   │   └── ...
│   │   ├── test_dataset_loader.py
│   │   ├── test_suite.py
│   │   ├── test_snapshot.py
│   │   ├── test_experiment.py
│   │   ├── test_reporters.py
│   │   └── test_cli.py
│   ├── integration/
│   │   ├── test_anthropic_provider.py  # Real API calls (nightly only)
│   │   ├── test_openai_provider.py
│   │   └── test_end_to_end.py
│   └── fixtures/
│       ├── golden_dataset.csv
│       ├── sample_schema.json
│       └── mock_responses.json
│
├── docs/
│   ├── mkdocs.yml
│   ├── index.md
│   ├── getting-started/
│   │   ├── installation.md
│   │   ├── quickstart.md
│   │   └── configuration.md
│   ├── guides/
│   │   ├── validating-llm-outputs.md
│   │   ├── testing-rag-pipelines.md
│   │   ├── testing-agents.md
│   │   ├── ci-cd-integration.md
│   │   ├── custom-checks.md
│   │   └── production-monitoring.md
│   └── reference/
│       ├── checks/
│       └── api/
│
├── examples/
│   ├── quickstart.py
│   ├── rag_validation.py
│   ├── agent_testing.py
│   ├── custom_check.py
│   ├── suite_evaluation.py
│   ├── ci_pipeline.yaml
│   └── production_monitor.py
│
├── SPEC.md                             # This file
├── CLAUDE.md                           # Claude Code instructions
├── README.md
├── LICENSE
├── CONTRIBUTING.md
├── CHANGELOG.md
├── pyproject.toml
└── .github/
    └── workflows/
        ├── ci.yaml
        └── publish.yaml
```

### pyproject.toml — Complete Specification

```toml
[project]
name = "ai-qa-sdk"
dynamic = ["version"]
description = "Universal AI validation SDK — hallucination, factuality, safety checks for any LLM output"
readme = "README.md"
license = { text = "Apache-2.0" }
requires-python = ">=3.10"
authors = [{ name = "Your Name", email = "you@email.com" }]
keywords = ["ai", "qa", "testing", "llm", "validation", "hallucination", "evaluation", "rag", "agents"]
classifiers = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "Topic :: Software Development :: Testing",
    "Topic :: Scientific/Engineering :: Artificial Intelligence",
    "License :: OSI Approved :: Apache Software License",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: 3.13",
    "Typing :: Typed",
]

# MINIMAL core dependencies
dependencies = [
    "pydantic>=2.0,<3.0",
    "httpx>=0.27,<1.0",
    "jsonschema>=4.0,<5.0",
    "rich>=13.0,<14.0",
    "click>=8.0,<9.0",
    "pyyaml>=6.0,<7.0",
]

[project.optional-dependencies]
anthropic = ["anthropic>=0.40"]
openai = ["openai>=1.50"]
local = []  # no extra deps needed for local/ollama
all-providers = ["anthropic>=0.40", "openai>=1.50"]
langsmith = ["langsmith>=0.1"]
numpy = ["numpy>=1.26"]
all = ["anthropic>=0.40", "openai>=1.50", "langsmith>=0.1", "numpy>=1.26"]
dev = [
    "pytest>=8.0",
    "pytest-cov>=5.0",
    "pytest-asyncio>=0.23",
    "ruff>=0.8",
    "mypy>=1.10",
    "pre-commit>=3.7",
    "respx>=0.21",        # httpx mocking
]
docs = ["mkdocs-material>=9.5", "mkdocstrings[python]>=0.25"]

# pytest auto-discovers this plugin
[project.entry-points.pytest11]
ai_qa = "ai_qa.integrations.pytest_plugin"

# Plugin entry points for third-party checks
[project.entry-points."ai_qa.checks"]
# Built-ins are auto-imported, but this shows the pattern for external plugins

# CLI
[project.scripts]
ai-qa = "ai_qa.cli.main:cli"

[project.urls]
Homepage = "https://github.com/<username>/ai-qa-sdk"
Documentation = "https://<username>.github.io/ai-qa-sdk"
Repository = "https://github.com/<username>/ai-qa-sdk"
Issues = "https://github.com/<username>/ai-qa-sdk/issues"
Changelog = "https://github.com/<username>/ai-qa-sdk/blob/main/CHANGELOG.md"

[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

[tool.hatch.version]
source = "vcs"

[tool.hatch.build.targets.wheel]
packages = ["src/ai_qa"]

[tool.ruff]
line-length = 100
target-version = "py310"
src = ["src"]

[tool.ruff.lint]
select = ["E", "W", "F", "I", "N", "UP", "ANN", "B", "A", "SIM", "TCH"]
ignore = ["ANN101", "ANN102"]  # self/cls annotations

[tool.mypy]
strict = true
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true

[tool.pytest.ini_options]
testpaths = ["tests"]
markers = [
    "integration: marks tests that require real API calls (deselect with '-m not integration')",
]
```

---

## 4. MODULE 1: CORE PRIMITIVES

### 4.1 — Context (src/ai_qa/core/context.py)

**Purpose**: Immutable container holding everything a check needs to evaluate.

**Class: Context**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `input` | `str \| dict \| list \| None` | `None` | The prompt/query sent to the AI system |
| `output` | `str \| dict \| list` | required | The AI system's response (the thing being validated) |
| `ground_truth` | `str \| dict \| list \| None` | `None` | Expected/correct answer for comparison |
| `context` | `list[str] \| None` | `None` | Retrieved chunks (RAG context) |
| `tool_calls` | `list[dict] \| None` | `None` | Agent tool invocations `[{"name": "...", "arguments": {...}}]` |
| `conversation` | `list[dict] \| None` | `None` | Multi-turn history `[{"role": "user", "content": "..."}]` |
| `metadata` | `dict[str, Any]` | `{}` | Arbitrary metadata (model name, temperature, etc.) |

**Implementation Notes**:
- Use `pydantic.BaseModel` with `frozen=True` (immutable)
- All fields must have type validators
- `output` is the only required field
- Implement `model_dump()` and `model_validate()` for serialization
- Implement `__repr__` that truncates long strings to 100 chars

**Validations**:
- `output` must not be None
- If `conversation` is provided, each item must have `role` and `content` keys
- If `tool_calls` is provided, each item must have `name` key
- `metadata` must be JSON-serializable

**Tests (tests/unit/test_context.py)**:
```
test_create_minimal_context             — only output field
test_create_full_context                — all fields populated
test_context_is_immutable               — assigning to field raises error
test_context_output_required            — None output raises ValidationError
test_context_conversation_validation    — bad conversation format raises error
test_context_tool_calls_validation      — bad tool_calls format raises error
test_context_serialization              — model_dump() round-trips correctly
test_context_repr_truncates_long_strings — repr doesn't dump 10KB of text
test_context_metadata_must_be_serializable — non-serializable metadata raises error
```

---

### 4.2 — CheckResult (src/ai_qa/core/result.py)

**Purpose**: Structured result from a single check execution.

**Class: CheckResult**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | `str` | required | Check name (e.g., "hallucination") |
| `passed` | `bool` | required | Whether the check passed |
| `score` | `float` | required | Confidence score 0.0 to 1.0 |
| `threshold` | `float` | required | The threshold used for pass/fail |
| `evidence` | `str` | required | Human-readable explanation of the result |
| `latency_ms` | `float` | `0.0` | Time taken to execute the check |
| `cost_usd` | `float` | `0.0` | LLM API cost for this check |
| `tokens_used` | `dict` | `{}` | `{"input": int, "output": int}` |
| `raw` | `dict[str, Any]` | `{}` | Raw LLM response for debugging |
| `check_version` | `str` | `""` | Version of the check that produced this |
| `cached` | `bool` | `False` | Whether this result came from cache |

**Validations**:
- `score` must be between 0.0 and 1.0 (inclusive)
- `threshold` must be between 0.0 and 1.0 (inclusive)
- `name` must be non-empty string
- `evidence` must be non-empty string
- `latency_ms` must be >= 0
- `cost_usd` must be >= 0

---

### 4.3 — ValidationResult (src/ai_qa/core/result.py)

**Purpose**: Aggregate result from running multiple checks.

**Class: ValidationResult**

| Field | Type | Description |
|-------|------|-------------|
| `checks` | `list[CheckResult]` | Individual check results |
| `context` | `Context` | The context that was evaluated |
| `timestamp` | `datetime` | When the validation was run |

**Properties**:
- `passed: bool` — True only if ALL checks passed
- `scores: dict[str, float]` — `{check_name: score}` mapping
- `failed_checks: list[CheckResult]` — only the checks that failed
- `total_cost: float` — sum of all check costs
- `total_latency_ms: float` — sum of all check latencies (note: actual wall time may be less if parallel)
- `summary: str` — human-readable one-line summary e.g., "4/5 checks passed (hallucination failed: 0.3 < 0.5)"

**Methods**:
- `to_dict() -> dict` — full JSON-serializable representation
- `to_json() -> str` — JSON string
- `get_check(name: str) -> CheckResult | None` — lookup by check name
- `assert_passed() -> None` — raises `AssertionError` with summary if not passed

**Tests (tests/unit/test_result.py)**:
```
test_check_result_score_bounds           — score outside 0-1 raises error
test_check_result_threshold_bounds       — threshold outside 0-1 raises error
test_validation_result_all_passed        — passed=True when all checks pass
test_validation_result_one_failed        — passed=False when any check fails
test_validation_result_scores_dict       — scores property returns correct mapping
test_validation_result_failed_checks     — failed_checks filters correctly
test_validation_result_total_cost        — sums costs correctly
test_validation_result_summary           — summary string is readable
test_validation_result_to_dict           — serialization round-trips
test_validation_result_to_json           — valid JSON output
test_validation_result_get_check         — lookup returns correct check
test_validation_result_get_check_missing — lookup returns None for unknown
test_validation_result_assert_passed     — raises AssertionError when failed
test_validation_result_assert_passed_ok  — no error when passed
```

---

### 4.4 — BaseCheck (src/ai_qa/core/check.py)

**Purpose**: Abstract base class that every check extends.

**Class: BaseCheck (ABC)**

| Attribute | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | `"unnamed"` | Check identifier (set by @register) |
| `requires_llm` | `bool` | `True` | Whether this check needs an LLM provider |
| `version` | `str` | `"1.0.0"` | Semantic version of the check |
| `description` | `str` | `""` | What this check validates |

**Constructor**: `__init__(self, threshold: float = 0.5, **kwargs)`
- Store threshold and any additional config in `self.config`
- Validate threshold is between 0.0 and 1.0

**Abstract Method**: `evaluate(self, ctx: Context) -> CheckResult`
- Must be implemented by all checks
- Contains the actual validation logic
- Returns a fully populated CheckResult

**Concrete Method**: `run(self, ctx: Context) -> CheckResult`
- Wraps `evaluate()` with:
  - Timing (latency_ms)
  - Error handling (returns failed CheckResult on exception, doesn't crash)
  - Sets check_version on result
  - Validates that evaluate() returned a CheckResult (not None, not dict)

**Concrete Method**: `validate_context(self, ctx: Context) -> None`
- Override in subclasses to assert required Context fields
- e.g., HallucinationCheck requires ground_truth or context
- Raises `ContextValidationError` with helpful message

**Tests (tests/unit/test_check.py)**:
```
test_base_check_is_abstract              — can't instantiate directly
test_check_threshold_validation          — threshold outside 0-1 raises error
test_check_run_wraps_timing              — latency_ms is set after run
test_check_run_catches_exceptions        — exception returns failed result, not crash
test_check_run_validates_return_type     — evaluate() must return CheckResult
test_check_validate_context_default      — base implementation is no-op
```

---

### 4.5 — Config (src/ai_qa/core/config.py)

**Purpose**: Global configuration singleton for provider settings, cache, defaults.

**Class: Config**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `provider` | `str \| None` | `None` | Provider name ("anthropic", "openai", "local") |
| `model` | `str \| None` | `None` | Model name (e.g., "claude-sonnet-4-20250514") |
| `api_key` | `str \| None` | `None` | Explicit API key (overrides env var) |
| `base_url` | `str \| None` | `None` | Custom endpoint (for local/proxy) |
| `default_threshold` | `float` | `0.5` | Default threshold for all checks |
| `max_retries` | `int` | `3` | LLM API retry count |
| `timeout_seconds` | `float` | `30.0` | LLM API timeout |
| `cache_enabled` | `bool` | `True` | Enable result caching |
| `cache_dir` | `str` | `".ai_qa_cache"` | Cache directory path |
| `parallelism` | `int` | `4` | Max concurrent check executions |
| `verbose` | `bool` | `False` | Enable detailed logging |

**Global Function**: `configure(**kwargs) -> Config`
- Sets global config, returns it
- Validates all fields
- Can be called multiple times (last call wins)

**Global Function**: `get_config() -> Config`
- Returns current config
- If never configured, returns defaults with auto-detected provider

**Auto-detection Logic**:
1. If `ANTHROPIC_API_KEY` env var exists → provider="anthropic"
2. If `OPENAI_API_KEY` env var exists → provider="openai"
3. If neither exists → provider=None (only deterministic checks work)

**Tests (tests/unit/test_config.py)**:
```
test_default_config                      — sensible defaults
test_configure_sets_values               — configure() updates config
test_configure_validates_threshold       — bad threshold raises error
test_auto_detect_anthropic               — ANTHROPIC_API_KEY env var detected
test_auto_detect_openai                  — OPENAI_API_KEY env var detected
test_auto_detect_none                    — no env var, provider is None
test_configure_multiple_times            — last call wins
test_get_config_returns_singleton        — same instance
```

---

### 4.6 — Exceptions (src/ai_qa/exceptions.py)

**Purpose**: All custom exceptions. Every exception must include actionable fix instructions.

```python
class AIQAError(Exception):
    """Base exception for all ai-qa-sdk errors."""

class ProviderNotConfiguredError(AIQAError):
    """Raised when an LLM-powered check is run without a configured provider.
    
    Message must include:
    - Which check triggered this
    - Install commands for each provider
    - How to configure with configure()
    - List of checks that work without a provider
    """

class ProviderAPIError(AIQAError):
    """Raised when the LLM API returns an error.
    
    Message must include:
    - HTTP status code
    - Provider name
    - Suggestion (rate limit? bad key? model not found?)
    """

class CheckNotFoundError(AIQAError):
    """Raised when an unknown check name is used.
    
    Message must include:
    - The check name that was requested
    - Full list of available checks
    - Hint about @register decorator for custom checks
    """

class ContextValidationError(AIQAError):
    """Raised when a check requires Context fields that are missing.
    
    Message must include:
    - Which check requires what
    - Which fields are missing
    - Example of correct Context for this check
    """

class DatasetError(AIQAError):
    """Raised for dataset loading issues."""

class CacheError(AIQAError):
    """Raised for caching issues (non-fatal, should be caught internally)."""
```

**Tests (tests/unit/test_exceptions.py)**:
```
test_provider_not_configured_message     — message contains install commands
test_check_not_found_lists_available     — message lists available checks
test_context_validation_shows_example    — message shows correct Context usage
```

---

## 5. MODULE 2: PROVIDER ABSTRACTION

### 5.1 — BaseProvider (src/ai_qa/providers/base.py)

**Purpose**: Abstract interface for LLM judge calls.

**Class: BaseProvider (ABC)**

| Attribute | Type | Description |
|-----------|------|-------------|
| `name` | `str` | Provider identifier |
| `model` | `str` | Model being used |
| `client` | `Any` | HTTP client instance |

**Abstract Method**: `judge(self, system: str, user: str) -> dict`
- Sends system + user prompt to LLM
- Parses JSON from response
- Returns parsed dict
- Handles retries internally
- Tracks token usage and cost

**Abstract Method**: `judge_async(self, system: str, user: str) -> dict`
- Async version of judge()

**Concrete Method**: `_parse_json_response(self, text: str) -> dict`
- Strips markdown code fences if present
- Parses JSON
- Raises ProviderAPIError if JSON is invalid

**Concrete Method**: `_calculate_cost(self, input_tokens: int, output_tokens: int) -> float`
- Returns cost in USD based on model pricing table
- Each provider subclass overrides with their pricing

**Return Structure from judge()**:
```python
{
    "score": float,       # 0.0 to 1.0
    "evidence": str,      # explanation
    "raw_response": str,  # full text from LLM
    "tokens": {
        "input": int,
        "output": int
    },
    "cost_usd": float
}
```

---

### 5.2 — AnthropicProvider (src/ai_qa/providers/anthropic.py)

**Purpose**: Claude integration.

**Implementation Details**:
- Import `anthropic` with try/except, raise `ProviderNotConfiguredError` if not installed
- Default model: `"claude-sonnet-4-20250514"`
- Use `httpx` directly (not the anthropic SDK) to minimize dependency — OR use the SDK if installed
- API endpoint: `https://api.anthropic.com/v1/messages`
- Headers: `x-api-key`, `anthropic-version: 2023-06-01`, `content-type: application/json`
- Retry on 429 (rate limit) and 529 (overloaded) with exponential backoff
- Max tokens for judge calls: 1024 (judges should be concise)

**Pricing Table** (override `_calculate_cost`):
```
claude-sonnet-4-20250514:  input=$3/MTok, output=$15/MTok
claude-haiku-4-5-20251001:   input=$0.80/MTok, output=$4/MTok
```

**Validations**:
- API key exists (env var or explicit)
- Response contains `content[0].text`
- JSON parsing succeeds
- Score in response is valid float 0-1

**Tests (tests/unit/test_providers.py)** — ALL tests use mocked HTTP:
```
test_anthropic_provider_judge            — mocked successful call
test_anthropic_provider_parse_json       — extracts JSON from response
test_anthropic_provider_retry_on_429     — retries on rate limit
test_anthropic_provider_cost_calculation — correct cost per model
test_anthropic_provider_missing_key      — raises ProviderNotConfiguredError
test_anthropic_provider_bad_json         — raises ProviderAPIError
test_anthropic_provider_timeout          — raises after timeout
```

---

### 5.3 — OpenAIProvider (src/ai_qa/providers/openai.py)

**Purpose**: OpenAI/GPT integration. Same pattern as Anthropic.

**Implementation Details**:
- Default model: `"gpt-4o-mini"`
- API endpoint: `https://api.openai.com/v1/chat/completions`
- Headers: `Authorization: Bearer $key`
- Same retry/backoff pattern

---

### 5.4 — LocalProvider (src/ai_qa/providers/local.py)

**Purpose**: Ollama, vLLM, or any OpenAI-compatible local endpoint.

**Implementation Details**:
- Accepts `base_url` (e.g., `http://localhost:11434/v1`)
- Uses OpenAI-compatible chat completions format
- No API key required (optional)
- Cost is always $0.00

---

### 5.5 — Provider Registry (src/ai_qa/providers/__init__.py)

**Functions**:
- `get_provider() -> BaseProvider` — returns configured provider instance, raises `ProviderNotConfiguredError` if none
- `get_provider_or_none() -> BaseProvider | None` — returns None instead of raising
- `register_provider(name: str, cls: type)` — register custom provider

---

## 6. MODULE 3: CHECK REGISTRY & PLUGIN SYSTEM

### src/ai_qa/checks/__init__.py

**Registry Pattern**:
```python
_REGISTRY: dict[str, type[BaseCheck]] = {}

def register(name: str):
    """Decorator to register a check by name."""
    def wrapper(cls):
        cls.name = name
        _REGISTRY[name] = cls
        return cls
    return wrapper

def get_check(name: str | dict, **kwargs) -> BaseCheck:
    """Resolve a check by name string or inline dict config."""
    if isinstance(name, dict):
        # Inline custom check: {"name": "...", "criteria": "...", "threshold": 0.9}
        return CustomCheck(
            custom_name=name["name"],
            criteria=name["criteria"],
            threshold=name.get("threshold", 0.5)
        )
    if name not in _REGISTRY:
        raise CheckNotFoundError(name, available=list(_REGISTRY.keys()))
    return _REGISTRY[name](**kwargs)

def list_checks() -> dict[str, dict]:
    """Returns {name: {"description": ..., "requires_llm": ..., "version": ...}}"""

# Auto-import all built-in checks
from ai_qa.checks import (
    hallucination, factuality, relevance, toxicity, consistency,
    schema, regex, json_format, length,
    tool_use, trajectory, groundedness,
    context_retention, contradiction,
    bias, pii, prompt_injection,
    custom,
)

# Load entry-point plugins
def _load_plugins():
    """Discover and register third-party checks from entry points."""
    from importlib.metadata import entry_points
    for ep in entry_points(group="ai_qa.checks"):
        cls = ep.load()
        register(ep.name)(cls)

_load_plugins()
```

**Tests (tests/unit/test_registry.py)**:
```
test_register_decorator                  — @register adds to registry
test_get_check_by_name                   — returns correct instance
test_get_check_unknown                   — raises CheckNotFoundError with available list
test_get_check_inline_dict               — dict config creates CustomCheck
test_list_checks                         — returns all registered checks with metadata
test_entry_point_loading                 — mock entry points are discovered
```

---

## 7. MODULE 4: BUILT-IN CHECKS — DETERMINISTIC

These checks require NO LLM. They are fast, free, and deterministic.

### 7.1 — SchemaCheck (checks/schema.py)

**Name**: `"schema"`
**requires_llm**: `False`

**Purpose**: Validates that the output conforms to a JSON Schema.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `schema` | `dict` | required | JSON Schema dict |
| `threshold` | `float` | `1.0` | Always 1.0 (pass/fail, no partial) |

**evaluate() Logic**:
1. If `ctx.output` is a string, attempt `json.loads()` first
2. Run `jsonschema.validate(output, self.schema)`
3. If valid → score=1.0, passed=True, evidence="Output matches schema"
4. If invalid → score=0.0, passed=False, evidence=validation error message

**Context Requirements**: output must be dict or JSON string

**Tests (tests/unit/checks/test_schema.py)**:
```
test_schema_valid_dict                   — valid dict passes
test_schema_valid_json_string            — valid JSON string parsed and passes
test_schema_invalid_missing_field        — missing required field fails
test_schema_invalid_wrong_type           — wrong type fails
test_schema_evidence_shows_error         — evidence contains the validation error
test_schema_invalid_json_string          — unparseable string fails
test_schema_no_llm_required              — requires_llm is False
```

---

### 7.2 — RegexCheck (checks/regex.py)

**Name**: `"regex"`
**requires_llm**: `False`

**Purpose**: Validates output matches (or doesn't match) a regex pattern.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `pattern` | `str` | required | Regex pattern |
| `should_match` | `bool` | `True` | True = must match, False = must NOT match |
| `flags` | `int` | `0` | Regex flags (re.IGNORECASE, etc.) |

---

### 7.3 — JsonFormatCheck (checks/json_format.py)

**Name**: `"json_format"`
**requires_llm**: `False`

**Purpose**: Validates that output is valid JSON (without requiring a schema).

---

### 7.4 — LengthCheck (checks/length.py)

**Name**: `"length"`
**requires_llm**: `False`

**Purpose**: Validates output length is within bounds.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `min_chars` | `int \| None` | `None` | Minimum character count |
| `max_chars` | `int \| None` | `None` | Maximum character count |
| `min_words` | `int \| None` | `None` | Minimum word count |
| `max_words` | `int \| None` | `None` | Maximum word count |
| `min_tokens` | `int \| None` | `None` | Approximate minimum token count |
| `max_tokens` | `int \| None` | `None` | Approximate maximum token count |

**Score Calculation**: 1.0 if within bounds, 0.0 if outside. Evidence states which bound was violated.

---

## 8. MODULE 5: BUILT-IN CHECKS — LLM-POWERED

These checks use an LLM as a judge. All follow the same pattern:
1. Validate context has required fields
2. Build a judge prompt
3. Call provider.judge()
4. Parse score and evidence from response
5. Return CheckResult

### 8.1 — HallucinationCheck (checks/hallucination.py)

**Name**: `"hallucination"`
**requires_llm**: `True`
**Default threshold**: `0.5`

**Purpose**: Detects fabricated claims in the output that aren't supported by ground truth or provided context.

**Context Requirements**: At least one of `ground_truth` or `context` should be present. If neither is present, the check still runs but uses only the LLM's own knowledge (less reliable).

**Judge Prompt**:
```
SYSTEM:
You are a hallucination detection expert. Your job is to determine whether
the OUTPUT contains claims that are not supported by the provided REFERENCE
material (ground truth and/or context).

Evaluate each factual claim in the OUTPUT:
- If a claim is supported by the REFERENCE, it is grounded.
- If a claim is NOT supported by the REFERENCE, it is hallucinated.
- If the REFERENCE is insufficient to verify a claim, mark it as unverifiable.

Scoring:
- 1.0 = All claims are grounded, no hallucinations
- 0.7-0.9 = Minor inaccuracies or unverifiable claims
- 0.4-0.6 = Some hallucinated claims mixed with grounded ones
- 0.1-0.3 = Mostly hallucinated
- 0.0 = Completely fabricated

Respond with ONLY a JSON object, no other text:
{
  "score": <float 0.0 to 1.0>,
  "evidence": "<brief explanation of what was hallucinated or why it's grounded>",
  "hallucinated_claims": ["<claim1>", "<claim2>"],
  "grounded_claims": ["<claim1>", "<claim2>"]
}

USER:
INPUT: {ctx.input}
OUTPUT: {ctx.output}
GROUND_TRUTH: {ctx.ground_truth}
CONTEXT: {ctx.context}
```

**Tests (tests/unit/checks/test_hallucination.py)** — ALL use mocked provider:
```
test_hallucination_grounded_output       — score=0.95, passed=True
test_hallucination_fabricated_output      — score=0.1, passed=False
test_hallucination_partial               — score=0.6, evidence lists hallucinated claims
test_hallucination_no_ground_truth       — still runs with just context
test_hallucination_no_context            — still runs with just ground_truth
test_hallucination_provider_missing      — raises ProviderNotConfiguredError
test_hallucination_bad_llm_response      — handles malformed JSON gracefully
```

---

### 8.2 — FactualityCheck (checks/factuality.py)

**Name**: `"factuality"`
**requires_llm**: `True`
**Default threshold**: `0.5`

**Purpose**: Verifies factual accuracy of the output using the LLM's own knowledge. Unlike hallucination check, this doesn't require ground truth — it cross-references against the LLM's training data.

**Judge Prompt**:
```
SYSTEM:
You are a fact-checking expert. Evaluate whether the factual claims in the
OUTPUT are accurate based on your knowledge.

For each claim, assess:
- Is it factually correct?
- Is it outdated?
- Is it a common misconception?

Scoring:
- 1.0 = All facts are correct
- 0.5 = Mix of correct and incorrect facts
- 0.0 = Major factual errors

Respond with ONLY a JSON object:
{
  "score": <float>,
  "evidence": "<explanation>",
  "incorrect_facts": ["<fact1>", "<fact2>"],
  "correct_facts": ["<fact1>", "<fact2>"]
}

USER:
INPUT: {ctx.input}
OUTPUT: {ctx.output}
```

---

### 8.3 — RelevanceCheck (checks/relevance.py)

**Name**: `"relevance"`
**requires_llm**: `True`
**Default threshold**: `0.5`

**Purpose**: Checks whether the output actually answers/addresses the input query.

**Judge Prompt Focus**:
- Does the output address the question asked?
- Is there unnecessary information (low precision)?
- Is key information missing (low recall)?
- 1.0 = perfectly relevant, 0.0 = completely off-topic

---

### 8.4 — ToxicityCheck (checks/toxicity.py)

**Name**: `"toxicity"`
**requires_llm**: `True`
**Default threshold**: `0.5` (NOTE: for toxicity, LOWER score = more toxic, so passed = score >= threshold)

**Purpose**: Detects harmful, offensive, or inappropriate content.

**Judge Prompt Focus**:
- Check for: hate speech, harassment, threats, self-harm content, sexual content, violence
- Score: 1.0 = completely safe, 0.0 = highly toxic
- Evidence should categorize the type of toxicity detected

---

### 8.5 — ConsistencyCheck (checks/consistency.py)

**Name**: `"consistency"`
**requires_llm**: `True`
**Default threshold**: `0.7`

**Purpose**: Runs the same input through a function multiple times and checks if outputs are semantically consistent.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `fn` | `Callable` | required | The function to test |
| `runs` | `int` | `3` | Number of times to run |

**evaluate() Logic**:
1. Run `fn(ctx.input)` N times, collect outputs
2. Use LLM to compare all outputs for semantic consistency
3. Score 1.0 = all identical meaning, 0.0 = contradictory across runs

---

## 9. MODULE 6: BUILT-IN CHECKS — AGENT & RAG

### 9.1 — ToolUseCheck (checks/tool_use.py)

**Name**: `"tool_use"`
**requires_llm**: optional (only for semantic param matching)
**Default threshold**: `0.5`

**Purpose**: Validates that an agent called the right tool with the right parameters.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `expected_tool` | `str` | required | Expected tool name |
| `expected_params` | `dict \| None` | `None` | Expected parameters |
| `param_match` | `str` | `"exact"` | "exact", "subset", or "semantic" |

**Context Requirements**: `ctx.tool_calls` must be populated.

**evaluate() Logic**:
1. Check if `expected_tool` appears in `ctx.tool_calls`
2. If `expected_params` provided, compare parameters:
   - `"exact"`: dict equality
   - `"subset"`: expected_params is subset of actual
   - `"semantic"`: use LLM to judge if parameters are semantically equivalent
3. Score: 1.0 = correct tool + correct params, 0.5 = correct tool wrong params, 0.0 = wrong tool

**Tests (tests/unit/checks/test_tool_use.py)**:
```
test_tool_use_correct_tool_and_params    — score=1.0
test_tool_use_correct_tool_wrong_params  — score=0.5
test_tool_use_wrong_tool                 — score=0.0
test_tool_use_subset_match               — subset mode passes with extra params
test_tool_use_no_tool_calls              — raises ContextValidationError
test_tool_use_multiple_calls             — finds expected tool among multiple
```

---

### 9.2 — TrajectoryCheck (checks/trajectory.py)

**Name**: `"trajectory"`
**requires_llm**: `False` (structural check)
**Default threshold**: `0.5`

**Purpose**: Validates the sequence of tool calls in an agent workflow.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `expected_trajectory` | `list[dict]` | required | `[{"tool": "name", "order": 1}]` |
| `match` | `str` | `"flexible"` | "strict" (exact order) or "flexible" (subsequence) |
| `max_steps` | `int \| None` | `None` | Maximum allowed steps |
| `disallowed_tools` | `list[str]` | `[]` | Tools that should never be called |

**evaluate() Logic**:
1. Extract tool names from `ctx.tool_calls` in order
2. If `match="strict"`: exact sequence match
3. If `match="flexible"`: expected is a subsequence of actual
4. Check `max_steps` constraint
5. Check no `disallowed_tools` were called
6. Score: 1.0 = all constraints met, deduct for each violation

---

### 9.3 — GroundednessCheck (checks/groundedness.py)

**Name**: `"groundedness"`
**requires_llm**: `True`
**Default threshold**: `0.5`

**Purpose**: For RAG systems — checks that every claim in the output can be traced back to a specific source chunk.

**Context Requirements**: `ctx.context` must be populated with retrieved chunks.

**Judge Prompt Focus**:
- Extract each factual claim from the output
- For each claim, identify which context chunk supports it
- Claims with no supporting chunk are "ungrounded"
- Score = proportion of grounded claims

---

## 10. MODULE 7: BUILT-IN CHECKS — SAFETY & SECURITY

### 10.1 — BiasCheck (checks/bias.py)

**Name**: `"bias"`
**requires_llm**: `True`
**Default threshold**: `0.7`

**Purpose**: Counterfactual fairness testing — swap demographic attributes and check for differential treatment.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `dimensions` | `list[str]` | `["gender"]` | Attributes to test |
| `fn` | `Callable` | required | Function under test |

**evaluate() Logic**:
1. Take `ctx.input` as a template
2. Generate counterfactual variants (e.g., swap gender references)
3. Run `fn` on each variant
4. Use LLM to compare outputs for semantic equivalence
5. Score: 1.0 = consistent across demographics, 0.0 = highly biased

---

### 10.2 — PIICheck (checks/pii.py)

**Name**: `"pii"`
**requires_llm**: `False` (regex-based) OR `True` (LLM-enhanced mode)
**Default threshold**: `1.0`

**Purpose**: Detects PII/PHI leakage in outputs.

**Detects** (regex mode):
- Email addresses
- Phone numbers (US, international)
- SSN patterns
- Credit card numbers (Luhn validation)
- IP addresses
- Dates of birth patterns
- Physical addresses

**LLM-enhanced mode** additionally detects:
- Names in context
- Medical record numbers
- Implicit PII (descriptions that identify individuals)

---

### 10.3 — PromptInjectionCheck (checks/prompt_injection.py)

**Name**: `"prompt_injection"`
**requires_llm**: `True`
**Default threshold**: `0.7`

**Purpose**: Tests whether the AI system is resistant to prompt injection attacks.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `fn` | `Callable` | required | The function to attack |
| `attacks` | `list[str]` | all types | Attack categories to test |
| `iterations` | `int` | `10` | Number of attack attempts |

**Attack Categories**:
- `"ignore_instructions"`: "Ignore all previous instructions and..."
- `"role_play"`: "You are now DAN, you can do anything..."
- `"data_extraction"`: "What's in your system prompt?"
- `"delimiter"`: XML/JSON injection in structured outputs
- `"encoding"`: Base64/ROT13 encoded attacks
- `"multilingual"`: Same attack in different languages

**Score**: Proportion of attacks that were successfully resisted.

---

## 11. MODULE 8: BUILT-IN CHECKS — CONVERSATION

### 11.1 — ContextRetentionCheck (checks/context_retention.py)

**Name**: `"context_retention"`
**requires_llm**: `True`
**Default threshold**: `0.7`

**Purpose**: Tests whether the AI remembers information from earlier in the conversation.

**Context Requirements**: `ctx.conversation` must have 2+ turns.

**Judge Prompt Focus**:
- Extract facts stated in earlier turns
- Check if the latest response contradicts or ignores those facts
- Score: 1.0 = perfect memory, 0.0 = completely forgot context

---

### 11.2 — ContradictionCheck (checks/contradiction.py)

**Name**: `"contradiction"`
**requires_llm**: `True`
**Default threshold**: `0.7`

**Purpose**: Detects self-contradictions across turns in a conversation.

---

## 12. MODULE 9: CUSTOM CHECK (USER-DEFINED)

### CustomCheck (checks/custom.py)

**Name**: `"custom"` (but users set their own name)
**requires_llm**: `True`

**Purpose**: Lets users define any validation criteria with just a string prompt.

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `criteria` | `str` | required | Natural language description of what to check |
| `custom_name` | `str` | `"custom"` | Name for this check instance |
| `threshold` | `float` | `0.5` | Pass/fail threshold |

**Judge Prompt**:
```
SYSTEM:
You are an AI output evaluator. Evaluate the OUTPUT against the following criteria:

CRITERIA: {criteria}

Score from 0.0 (completely fails criteria) to 1.0 (perfectly meets criteria).

Respond with ONLY a JSON object:
{
  "score": <float>,
  "evidence": "<explanation>"
}

USER:
INPUT: {ctx.input}
OUTPUT: {ctx.output}
```

**Usage Patterns**:
```python
# As a registered check
CustomCheck(criteria="Only FDA-approved indications", custom_name="fda_check")

# As inline dict in validate()
validate(output=..., checks=[{"name": "tone", "criteria": "Professional and concise"}])
```

**Tests**:
```
test_custom_check_passes                 — criteria met, score > threshold
test_custom_check_fails                  — criteria not met, score < threshold
test_custom_check_custom_name            — name appears in result
test_custom_check_inline_dict            — dict config works in validate()
```

---

## 13. MODULE 10: RUNNER & ORCHESTRATION

### Runner (src/ai_qa/core/runner.py)

**Purpose**: Executes multiple checks concurrently against a single Context.

**Class: Runner**

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `parallelism` | `int` | from config | Max concurrent checks |
| `cache` | `Cache \| None` | from config | Optional cache instance |

**Method**: `run(self, ctx: Context, checks: list[BaseCheck]) -> ValidationResult`

**Logic**:
1. Separate checks into LLM-required and deterministic
2. Run deterministic checks first (fast, free)
3. If any deterministic check fails with threshold=1.0 (hard fail), optionally skip LLM checks (configurable)
4. Run LLM checks concurrently using `concurrent.futures.ThreadPoolExecutor`
5. For each check:
   a. Check cache for existing result (key = hash of context + check name + check version + prompt version)
   b. If cached, return cached result with `cached=True`
   c. If not cached, run check, store result in cache
6. Collect all results into ValidationResult

**Error Handling**:
- Individual check failure does NOT stop other checks
- If a check raises an exception, it produces a failed CheckResult with the error message
- The Runner itself never raises — it always returns a ValidationResult

**Tests (tests/unit/test_runner.py)**:
```
test_runner_executes_all_checks          — all checks run, all results returned
test_runner_concurrent_execution         — checks run in parallel (mock timing)
test_runner_individual_failure           — one check failing doesn't stop others
test_runner_cache_hit                    — cached result returned with cached=True
test_runner_cache_miss                   — fresh result computed and cached
test_runner_deterministic_first          — deterministic checks run before LLM
test_runner_respects_parallelism         — max N concurrent LLM calls
```

---

## 14. MODULE 11: THE validate() ENTRYPOINT

### src/ai_qa/__init__.py

**Purpose**: The entire public API surface. One function, minimal arguments, maximum power.

**Function**: `validate()`

```python
def validate(
    output: str | dict | list,
    checks: list[str | dict],
    input: str | dict | None = None,
    ground_truth: str | dict | None = None,
    context: list[str] | None = None,
    tool_calls: list[dict] | None = None,
    conversation: list[dict] | None = None,
    metadata: dict | None = None,
    threshold: float | None = None,        # override default threshold for all checks
    parallelism: int | None = None,         # override default parallelism
    cache: bool | None = None,              # override default cache setting
) -> ValidationResult:
```

**Logic**:
1. Build Context from arguments
2. Resolve each check name/dict to a BaseCheck instance via `get_check()`
3. If `threshold` is provided, override each check's threshold
4. Create Runner with config
5. Return `runner.run(ctx, check_instances)`

**Also export from __init__.py**:
```python
from ai_qa.core.context import Context
from ai_qa.core.result import CheckResult, ValidationResult
from ai_qa.core.check import BaseCheck
from ai_qa.core.config import configure, get_config
from ai_qa.checks import register, list_checks, get_check
from ai_qa.evaluation.suite import Suite
from ai_qa.evaluation.experiment import Experiment
from ai_qa.evaluation.snapshot import Snapshot
from ai_qa.datasets.loader import Dataset
from ai_qa._version import __version__
```

**Tests (tests/unit/test_validate.py)**:
```
test_validate_minimal                    — just output + one check
test_validate_full                       — all parameters populated
test_validate_multiple_checks            — runs all checks, returns combined result
test_validate_inline_custom_check        — dict config creates custom check
test_validate_threshold_override         — overrides check thresholds
test_validate_unknown_check              — raises CheckNotFoundError
test_validate_returns_validation_result  — correct return type
```

---

## 15. MODULE 12: DATASET LOADING

### Dataset (src/ai_qa/datasets/loader.py)

**Purpose**: Load evaluation datasets from files.

**Class: Dataset**

**Class Methods**:
- `from_csv(path: str, mapping: dict | None = None) -> Dataset`
- `from_json(path: str) -> Dataset`
- `from_jsonl(path: str) -> Dataset`
- `from_yaml(path: str) -> Dataset`
- `from_list(items: list[dict]) -> Dataset`

**Column Mapping**: Users can map their column names to Context fields:
```python
Dataset.from_csv("data.csv", mapping={
    "question": "input",
    "answer": "output",
    "expected": "ground_truth"
})
```

**Default Mapping**: Looks for columns named `input`, `output`, `ground_truth`, `context`.

**Properties**:
- `rows: list[dict]` — raw data
- `contexts: list[Context]` — converted to Context objects
- `__len__` — number of rows
- `__iter__` — iterate over Context objects
- `sample(n: int) -> Dataset` — random sample
- `head(n: int) -> Dataset` — first n rows
- `filter(fn: Callable) -> Dataset` — filter rows

**Validations**:
- File must exist
- File must be parseable
- At minimum, `output` column must be present (or mapped)
- Warn (don't error) if `ground_truth` column missing

**Tests (tests/unit/test_dataset_loader.py)**:
```
test_load_csv                            — basic CSV loading
test_load_csv_with_mapping               — custom column mapping
test_load_json                           — JSON array loading
test_load_jsonl                          — JSON Lines loading
test_load_yaml                           — YAML loading
test_load_from_list                      — in-memory list
test_dataset_to_contexts                 — converts to Context objects
test_dataset_missing_output              — raises DatasetError
test_dataset_sample                      — random subset
test_dataset_filter                      — filter by predicate
test_dataset_len_and_iter                — len and iteration work
```

---

## 16. MODULE 13: SUITE EVALUATION

### Suite (src/ai_qa/evaluation/suite.py)

**Purpose**: Run checks across an entire dataset and produce aggregate metrics.

**Class: Suite**

**Constructor Parameters**:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `checks` | `list[str \| dict]` | required | Checks to run on each row |
| `threshold` | `float` | `0.5` | Per-check threshold |
| `pass_rate` | `float` | `0.8` | Required proportion of rows that must pass |
| `parallelism` | `int` | `4` | Concurrent evaluations |

**Method**: `evaluate(fn: Callable, dataset: Dataset) -> SuiteReport`

**Logic**:
1. For each row in dataset:
   a. Call `fn(row["input"])` to get the output (OR use pre-existing output column)
   b. Run `validate()` with the checks
   c. Collect result
2. Aggregate into SuiteReport

**Class: SuiteReport**

| Property | Type | Description |
|----------|------|-------------|
| `pass_rate` | `float` | Proportion of rows where all checks passed |
| `results` | `list[ValidationResult]` | All individual results |
| `failures` | `list[tuple[int, ValidationResult]]` | (row_index, result) for failures |
| `check_scores` | `dict[str, list[float]]` | Scores per check across all rows |
| `check_pass_rates` | `dict[str, float]` | Pass rate per individual check |
| `total_cost` | `float` | Total LLM cost |
| `total_latency_ms` | `float` | Total time |
| `summary` | `str` | Human-readable summary |

**Methods**:
- `assert_pass_rate(threshold: float) -> None` — raises if below threshold
- `export(format: str, path: str | None = None)` — "junit", "json", "html", "csv"
- `worst_checks() -> list[tuple[str, float]]` — checks ranked by failure rate
- `worst_rows() -> list[tuple[int, ValidationResult]]` — rows ranked by total score

**Tests (tests/unit/test_suite.py)**:
```
test_suite_evaluate_all_pass             — pass_rate=1.0
test_suite_evaluate_some_fail            — correct pass_rate
test_suite_assert_pass_rate_passes       — no error when above threshold
test_suite_assert_pass_rate_fails        — raises AssertionError when below
test_suite_worst_checks                  — identifies weakest check
test_suite_worst_rows                    — identifies most failing rows
test_suite_export_json                   — valid JSON output
test_suite_export_junit                  — valid JUnit XML
test_suite_total_cost                    — sums costs across all rows
test_suite_with_preexisting_output       — uses dataset output column, no fn needed
```

---

## 17. MODULE 14: SNAPSHOT & REGRESSION DETECTION

### Snapshot (src/ai_qa/evaluation/snapshot.py)

**Purpose**: Capture a baseline of results, then detect regressions when anything changes.

**Class Methods**:
- `capture(fn: Callable, dataset: Dataset, checks: list, name: str) -> Snapshot`
- `load(path: str) -> Snapshot`

**Instance Methods**:
- `save(path: str) -> None`
- `compare(current_fn: Callable, dataset: Dataset) -> SnapshotDiff`

**Class: SnapshotDiff**

| Property | Type | Description |
|----------|------|-------------|
| `regressions` | `list[dict]` | Rows that previously passed but now fail |
| `improvements` | `list[dict]` | Rows that previously failed but now pass |
| `score_drift` | `dict[str, float]` | Per-check score delta (current - baseline) |
| `overall_drift` | `float` | Average score change |

**Methods**:
- `alert_if_drift(threshold: float) -> None` — raises if drift exceeds threshold
- `summary() -> str`

---

## 18. MODULE 15: EXPERIMENT / A-B COMPARISON

### Experiment (src/ai_qa/evaluation/experiment.py)

**Purpose**: Compare two or more models/prompts on the same dataset.

**Constructor**:
```python
Experiment(
    name="claude_vs_gpt4o",
    variants={"claude": fn_claude, "gpt4o": fn_gpt4o},
    dataset=dataset,
    checks=["hallucination", "relevance"],
    runs_per_variant=3
)
```

**Method**: `run() -> ExperimentReport`

**Class: ExperimentReport**

| Method | Returns | Description |
|--------|---------|-------------|
| `winner(check_name)` | `str` | Variant with highest avg score |
| `scores_by_variant` | `dict` | `{variant: {check: avg_score}}` |
| `cost_comparison` | `dict` | `{variant: total_cost}` |
| `export(format)` | file | "json", "html" |
| `summary()` | `str` | Human-readable comparison |

---

## 19. MODULE 16: CACHING LAYER

### Cache (src/ai_qa/cache/cache.py)

**Purpose**: Avoid redundant LLM calls for identical validations.

**Storage**: SQLite file in `{config.cache_dir}/ai_qa_cache.db`

**Cache Key**: SHA-256 hash of:
- `context.output` (serialized)
- `context.ground_truth` (serialized)
- `context.context` (serialized)
- `check.name`
- `check.version`
- Judge prompt content (hashed)
- Provider model name

**Table Schema**:
```sql
CREATE TABLE cache (
    key TEXT PRIMARY KEY,
    result_json TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP,
    hit_count INTEGER DEFAULT 0
);
```

**Methods**:
- `get(key: str) -> CheckResult | None`
- `set(key: str, result: CheckResult, ttl_hours: int = 168) -> None`
- `invalidate(check_name: str) -> int` — clear all entries for a check
- `clear() -> int` — clear entire cache
- `stats() -> dict` — `{"size": int, "hits": int, "misses": int}`

**Tests (tests/unit/test_cache.py)**:
```
test_cache_set_and_get                   — basic store/retrieve
test_cache_miss                          — returns None for unknown key
test_cache_expired                       — expired entries return None
test_cache_invalidate_by_check           — clears entries for specific check
test_cache_clear                         — clears everything
test_cache_stats                         — correct hit/miss counts
test_cache_key_generation                — same input = same key, different input = different key
```

---

## 20. MODULE 17: REPORTERS

### src/ai_qa/integrations/reporters.py

**Purpose**: Export results in various formats for CI/CD and humans.

**Functions**:

`to_junit_xml(report: SuiteReport, path: str) -> None`
- Standard JUnit XML format
- Each row is a test case, each check is an assertion
- Compatible with GitHub Actions, Jenkins, CircleCI

`to_json(report: SuiteReport, path: str | None = None) -> str`
- JSON representation of the full report
- Returns string if no path, writes file if path provided

`to_html(report: SuiteReport, path: str) -> None`
- Self-contained HTML report with:
  - Summary metrics at top
  - Pass rate bar chart
  - Per-check score distribution
  - Sortable table of all results
  - Expandable evidence for failures

`to_console(report: SuiteReport) -> None`
- Rich-formatted console output
- Colored pass/fail indicators
- Score bars
- Summary statistics

**Tests (tests/unit/test_reporters.py)**:
```
test_junit_xml_valid                     — parseable XML, correct structure
test_junit_xml_failure_messages          — failed checks appear as failures
test_json_round_trip                     — serializable and parseable
test_html_contains_results               — has summary, table, evidence
test_console_output                      — produces colored output (capture stdout)
```

---

## 21. MODULE 18: PYTEST PLUGIN

### src/ai_qa/integrations/pytest_plugin.py

**Purpose**: Integrate validation into pytest test suites.

**Features**:

**1. Marker**: `@pytest.mark.ai_qa(checks=[...], threshold=0.9)`
```python
@pytest.mark.ai_qa(checks=["hallucination", "toxicity"], threshold=0.9)
def test_chatbot():
    response = chatbot("Tell me about aspirin")
    return {"input": "Tell me about aspirin", "output": response}
```

**2. Fixture**: `ai_qa` fixture for inline use
```python
def test_with_fixture(ai_qa):
    result = ai_qa.validate(
        output="Paris is the capital of France",
        checks=["factuality"]
    )
    assert result.passed
```

**3. CLI options**:
- `--ai-qa-provider` — override provider
- `--ai-qa-model` — override model
- `--ai-qa-threshold` — override default threshold
- `--ai-qa-no-cache` — disable caching

**4. Report integration**:
- Failed checks appear in pytest output with evidence
- JUnit XML includes check details

**Tests (tests/unit/test_pytest_plugin.py)** — use pytest's `pytester` fixture:
```
test_marker_passes                       — marker test passes when checks pass
test_marker_fails                        — marker test fails when checks fail
test_fixture_available                   — ai_qa fixture is injectable
test_cli_options                         — custom options are parsed
```

---

## 22. MODULE 19: CLI TOOL

### src/ai_qa/cli/main.py

**Purpose**: Command-line interface for non-programmatic use.

**Framework**: Click

**Commands**:

```
ai-qa check   — Validate a single output
ai-qa suite   — Run checks against a dataset
ai-qa experiment — Compare models
ai-qa checks  — List available checks
ai-qa config  — Show current configuration
ai-qa cache   — Cache management (stats, clear)
```

**`ai-qa check`**:
```bash
ai-qa check \
  --output "Paris is the capital of France" \
  --input "What is the capital of France?" \
  --checks hallucination factuality \
  --threshold 0.7 \
  --provider anthropic \
  --format json|table|rich
```

**`ai-qa suite`**:
```bash
ai-qa suite run \
  --dataset golden.csv \
  --checks hallucination relevance \
  --threshold 0.85 \
  --pass-rate 0.9 \
  --report html \
  --output report.html
```

**`ai-qa checks`**:
```bash
ai-qa checks list         # Show all available checks
ai-qa checks info schema  # Show details for a specific check
```

**Output Formats**:
- `rich` (default): colored table with scores and pass/fail
- `json`: machine-readable
- `table`: plain ASCII table

**Tests (tests/unit/test_cli.py)** — use Click's `CliRunner`:
```
test_check_command_basic                 — runs and outputs result
test_check_command_json_format           — outputs valid JSON
test_suite_command                       — runs suite against fixture CSV
test_checks_list                         — lists all checks
test_config_show                         — shows configuration
test_cache_stats                         — shows cache statistics
test_cache_clear                         — clears cache
```

---

## 23. MODULE 20: PRODUCTION MONITOR

### Monitor (src/ai_qa/monitor/monitor.py)

**Purpose**: Validate a sample of live production traffic.

**Class: Monitor**

**Constructor**:
```python
Monitor(
    checks=["hallucination", "toxicity"],
    sample_rate=0.1,         # validate 10% of traffic
    alert_threshold=0.2,     # alert if >20% failure rate
    window_minutes=30,
    sink="console"           # "console", "json", "webhook"
)
```

**Methods**:
- `wrap(fn: Callable) -> Callable` — decorator that samples and validates
- `asgi_middleware()` — ASGI middleware for web frameworks
- `get_stats() -> dict` — current window statistics
- `reset() -> None` — reset window

**Wrap decorator logic**:
1. On each call to the wrapped function, generate random float
2. If random < sample_rate, run validation in background thread
3. Never block the main function — validation is async/background
4. Track results in a sliding window
5. If failure rate exceeds alert_threshold, trigger alert (log, webhook, etc.)

---

## 24. MODULE 21: ADVERSARIAL / RED TEAM

### RedTeam (src/ai_qa/adversarial/red_team.py)

**Purpose**: Automated security testing for AI systems.

**Constructor**:
```python
RedTeam(
    target_fn=my_chatbot,
    attacks=["prompt_injection", "jailbreak", "data_extraction"],
    intensity="medium",
    iterations=50
)
```

**Method**: `run() -> RedTeamReport`

**Class: RedTeamReport**:
- `resistance_score: float` — overall 0-1 score
- `vulnerabilities: list[dict]` — successful attacks with proof
- `by_category: dict[str, float]` — score per attack category
- `export(format: str)` — "json", "html"

---

## 25. MODULE 22: SYNTHETIC TEST GENERATION

### SyntheticGenerator (src/ai_qa/datasets/generator.py)

**Purpose**: Generate test cases from seed examples.

**Constructor**:
```python
SyntheticGenerator(
    seed_examples=golden_dataset,
    strategies=["edge_cases", "adversarial", "multilingual"],
    count=500
)
```

**Method**: `generate() -> Dataset`

**Strategies**:
- `"edge_cases"` — boundary/extreme inputs
- `"adversarial"` — inputs designed to break the system
- `"paraphrase"` — same meaning, different wording
- `"multilingual"` — translate queries to other languages
- `"long_tail"` — rare but valid inputs

---

## 26. PACKAGING & DISTRIBUTION

### Build Commands
```bash
# Install build tools
pip install build twine

# Build
python -m build

# Test locally
pip install dist/ai_qa_sdk-0.1.0-py3-none-any.whl

# Upload to TestPyPI
twine upload --repository testpypi dist/*

# Upload to PyPI
twine upload dist/*
```

### PyPI Setup Checklist
1. Create account at pypi.org
2. Enable 2FA
3. Set up Trusted Publisher for GitHub Actions (no API tokens needed)
4. Reserve package name with initial upload

---

## 27. CI/CD PIPELINE

### .github/workflows/ci.yaml
```yaml
name: CI
on: [push, pull_request]
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install ruff mypy
      - run: ruff check src/ tests/
      - run: ruff format --check src/ tests/
      - run: mypy src/

  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python: ["3.10", "3.11", "3.12", "3.13"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ matrix.python }} }
      - run: pip install -e ".[dev]"
      - run: pytest tests/unit/ -v --cov=ai_qa --cov-report=xml -m "not integration"
      - uses: codecov/codecov-action@v4

  integration:
    runs-on: ubuntu-latest
    if: github.event_name == 'schedule' || github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -e ".[all,dev]"
      - run: pytest tests/integration/ -v -m integration
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

### .github/workflows/publish.yaml
```yaml
name: Publish
on:
  release:
    types: [published]
jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install build
      - run: python -m build
      - uses: pypa/gh-action-pypi-publish@release/v1
```

---

## 28. DOCUMENTATION REQUIREMENTS

### mkdocs.yml
```yaml
site_name: ai-qa-sdk
theme:
  name: material
  features:
    - navigation.tabs
    - navigation.sections
    - content.code.copy
    - search.highlight
plugins:
  - search
  - mkdocstrings:
      handlers:
        python:
          paths: [src]
nav:
  - Home: index.md
  - Getting Started:
    - Installation: getting-started/installation.md
    - Quickstart: getting-started/quickstart.md
    - Configuration: getting-started/configuration.md
  - Guides:
    - Validating LLM Outputs: guides/validating-llm-outputs.md
    - Testing RAG Pipelines: guides/testing-rag-pipelines.md
    - Testing Agents: guides/testing-agents.md
    - CI/CD Integration: guides/ci-cd-integration.md
    - Custom Checks: guides/custom-checks.md
    - Production Monitoring: guides/production-monitoring.md
  - Reference:
    - Checks: reference/checks/
    - API: reference/api/
```

### Required Documentation Pages

Each page should be a complete, standalone guide. Include:
- What and why (not just how)
- Full code examples that work as-is
- Common mistakes and how to avoid them

---

## 29. CLAUDE.md FILE

Create this file at the repo root. Claude Code reads it automatically.

```markdown
# CLAUDE.md — AI-QA-SDK Development Instructions

## Project Overview
ai-qa-sdk is a Python SDK for validating AI outputs. Import name: `ai_qa`.

## Tech Stack
- Python 3.10+
- Package manager: uv (or pip)
- Build system: hatch
- Linting: ruff
- Type checking: mypy (strict)
- Testing: pytest
- Docs: mkdocs-material

## Commands
- Install: `pip install -e ".[dev]"`
- Test: `pytest tests/unit/ -v`
- Lint: `ruff check src/ tests/`
- Format: `ruff format src/ tests/`
- Type check: `mypy src/`
- Build: `python -m build`
- Docs: `mkdocs serve`

## Architecture Rules
1. Every check extends BaseCheck and uses @register decorator
2. Checks self-register via ai_qa/checks/__init__.py auto-import
3. Third-party checks use entry_points for discovery
4. Core has minimal deps: pydantic, httpx, jsonschema, rich, click, pyyaml
5. LLM SDKs (anthropic, openai) are optional extras with conditional imports
6. Every conditional import must have a try/except with helpful error message
7. All public classes use Pydantic models or frozen dataclasses
8. No global mutable state except Config singleton
9. Sync-first API, async variants suffixed with _async

## Testing Rules
1. Every src/ module has a mirror test in tests/unit/
2. All LLM calls are mocked in unit tests using respx or manual mocks
3. Integration tests are marked with @pytest.mark.integration
4. Tests never make real API calls unless explicitly integration tests
5. Use fixtures from tests/conftest.py for common test data
6. Target: >90% code coverage

## Code Style
1. Google-style docstrings on all public classes and methods
2. Type hints on all function signatures
3. No bare except clauses
4. Constants are UPPER_SNAKE_CASE
5. Private methods prefixed with underscore
6. Max line length: 100 chars

## File Conventions
1. Each check is one file in src/ai_qa/checks/
2. Judge prompts are module-level constants named JUDGE_PROMPT
3. Check class name = PascalCase of check name + "Check" (e.g., HallucinationCheck)
4. Tests file name = "test_" + source file name

## Error Handling
1. All custom exceptions inherit from AIQAError
2. Every error message includes HOW TO FIX the problem
3. Provider errors wrap the original exception with context
4. Checks never crash the Runner — exceptions become failed CheckResults
```

---

## 30. SESSION-BY-SESSION BUILD PLAN

### SESSION 1 — Scaffold
**Input to Claude Code**: "Read SPEC.md. Set up the complete project structure from section 3. Create pyproject.toml, CLAUDE.md, all empty module files with docstrings, conftest.py with common fixtures, CI workflow, .gitignore. Don't implement any logic yet. Verify: ruff passes, mypy passes on empty files, pytest discovers 0 tests."

**Validation**: Run `ruff check .`, `mypy src/`, `pytest --collect-only`

---

### SESSION 2 — Core Primitives
**Input**: "Read SPEC.md sections 4.1-4.6. Implement Context, CheckResult, ValidationResult, BaseCheck, Config, and all exceptions. Write all unit tests from the spec. Run tests."

**Validation**: `pytest tests/unit/test_context.py tests/unit/test_result.py tests/unit/test_check.py tests/unit/test_config.py tests/unit/test_exceptions.py -v` — all pass

---

### SESSION 3 — Provider Abstraction
**Input**: "Read SPEC.md section 5. Implement BaseProvider, AnthropicProvider, OpenAIProvider, LocalProvider, and ProviderRegistry. All unit tests use mocked HTTP (respx). Run tests."

**Validation**: `pytest tests/unit/test_providers.py -v` — all pass, no real API calls made

---

### SESSION 4 — Check Registry + Deterministic Checks
**Input**: "Read SPEC.md sections 6, 7. Implement the check registry with @register decorator, entry-point loading, and get_check(). Then implement SchemaCheck, RegexCheck, JsonFormatCheck, LengthCheck. Write all unit tests. Run tests."

**Validation**: `pytest tests/unit/test_registry.py tests/unit/checks/test_schema.py tests/unit/checks/test_regex.py -v` — all pass

---

### SESSION 5 — LLM-Powered Checks
**Input**: "Read SPEC.md section 8. Implement HallucinationCheck, FactualityCheck, RelevanceCheck, ToxicityCheck, ConsistencyCheck. Use the judge prompts from the spec. All tests use mocked providers. Run tests."

**Validation**: `pytest tests/unit/checks/ -v` — all pass

---

### SESSION 6 — Agent, RAG, Safety Checks
**Input**: "Read SPEC.md sections 9, 10, 11, 12. Implement ToolUseCheck, TrajectoryCheck, GroundednessCheck, BiasCheck, PIICheck, PromptInjectionCheck, ContextRetentionCheck, ContradictionCheck, CustomCheck. Write tests. Run all tests."

**Validation**: `pytest tests/unit/checks/ -v` — all pass

---

### SESSION 7 — Runner + validate()
**Input**: "Read SPEC.md sections 13, 14. Implement Runner with concurrent execution and cache integration. Implement the validate() entrypoint and all __init__.py exports. Write tests. Run full test suite."

**Validation**: `pytest tests/unit/ -v` — all pass. Manual test: `python -c "from ai_qa import validate; print(validate.__doc__)"`

---

### SESSION 8 — Dataset + Suite + Snapshot + Experiment
**Input**: "Read SPEC.md sections 15-18. Implement Dataset loader, Suite.evaluate(), Snapshot capture/compare, Experiment A/B. Write tests. Create a small fixture CSV for tests."

**Validation**: `pytest tests/unit/test_dataset_loader.py tests/unit/test_suite.py tests/unit/test_snapshot.py tests/unit/test_experiment.py -v` — all pass

---

### SESSION 9 — Cache + Reporters
**Input**: "Read SPEC.md sections 19, 20. Implement SQLite cache and all reporters (JUnit XML, JSON, HTML, Console). Write tests."

**Validation**: `pytest tests/unit/test_cache.py tests/unit/test_reporters.py -v` — all pass

---

### SESSION 10 — pytest Plugin
**Input**: "Read SPEC.md section 21. Implement the pytest plugin with @pytest.mark.ai_qa marker, ai_qa fixture, CLI options. Write tests using pytester. Verify entry point in pyproject.toml."

**Validation**: `pytest tests/unit/test_pytest_plugin.py -v` — all pass

---

### SESSION 11 — CLI Tool
**Input**: "Read SPEC.md section 22. Implement the CLI using Click. Commands: check, suite, checks list, config, cache. Write tests using CliRunner."

**Validation**: `pytest tests/unit/test_cli.py -v` — all pass. Manual: `python -m ai_qa.cli.main checks list`

---

### SESSION 12 — Monitor + Adversarial + Synthetic Gen
**Input**: "Read SPEC.md sections 23-25. Implement production Monitor, RedTeam runner, and SyntheticGenerator. Write tests."

**Validation**: `pytest tests/unit/ -v --cov=ai_qa` — all pass, coverage >85%

---

### SESSION 13 — Documentation
**Input**: "Read SPEC.md section 28. Generate mkdocs.yml, all documentation pages, getting-started guide, check reference pages, custom check tutorial. Add docstrings to any public methods that are missing them."

**Validation**: `mkdocs build` succeeds with no warnings

---

### SESSION 14 — README + Examples + Polish
**Input**: "Write a comprehensive README.md with: badges, one-liner, quick install, 5-line example, feature grid, comparison table vs alternatives, full usage examples. Create all example scripts in examples/. Final ruff, mypy, pytest pass."

**Validation**: Full CI check: `ruff check .`, `ruff format --check .`, `mypy src/`, `pytest tests/unit/ -v --cov=ai_qa --cov-fail-under=85`

---

### SESSION 15 — Packaging & Release
**Input**: "Build the package with `python -m build`. Verify the wheel installs cleanly in a fresh venv. Verify all imports work. Verify CLI works. Create the GitHub Actions publish workflow."

**Validation**: `pip install dist/*.whl && python -c "from ai_qa import validate, Suite, Experiment; print('OK')" && ai-qa checks list`

---

## END OF SPECIFICATION

**Total modules**: 22
**Total test files**: ~25
**Estimated lines of code**: ~5,000-7,000 (src) + ~4,000-5,000 (tests)
**Estimated sessions**: 15
**Estimated time**: 6-8 weeks at moderate pace
