# CLAUDE.md — AI-QA-SDK Development Instructions

## Project Overview
ai-qa-sdk is a Python SDK for validating AI outputs. Import name: `ai_qa`.
Read SPEC.md for the complete build specification.

## Tech Stack
- Python 3.10+
- Package manager: uv (or pip)
- Build system: hatch
- Linting: ruff
- Type checking: mypy (strict)
- Testing: pytest
- Docs: mkdocs-material

## Commands
```bash
# Install
pip install -e ".[dev]"

# Test
pytest tests/unit/ -v

# Test with coverage
pytest tests/unit/ -v --cov=ai_qa --cov-report=term-missing

# Lint
ruff check src/ tests/

# Format
ruff format src/ tests/

# Type check
mypy src/

# Build package
python -m build

# Serve docs locally
mkdocs serve
```

## Architecture Rules
1. Every check extends BaseCheck and uses @register decorator
2. Checks self-register via ai_qa/checks/__init__.py auto-import
3. Third-party checks use setuptools entry_points for discovery
4. Core has minimal deps: pydantic, httpx, jsonschema, rich, click, pyyaml
5. LLM SDKs (anthropic, openai) are optional extras with conditional imports
6. Every conditional import must have a try/except with helpful ProviderNotConfiguredError
7. All public classes use Pydantic BaseModel with frozen=True or dataclasses
8. No global mutable state except Config singleton via configure()/get_config()
9. Sync-first API; async variants suffixed with _async
10. The validate() function in __init__.py is the primary public API

## Dependency Rules
- Core package: pydantic, httpx, jsonschema, rich, click, pyyaml — NOTHING ELSE
- anthropic SDK → optional extra [anthropic]
- openai SDK → optional extra [openai]
- numpy → optional extra [numpy]
- langsmith → optional extra [langsmith]
- Pattern for conditional imports:
  ```python
  try:
      import anthropic
  except ImportError:
      raise ProviderNotConfiguredError(
          "Anthropic provider requires the anthropic package.\n"
          "Install it with: pip install ai-qa-sdk[anthropic]"
      )
  ```

## Testing Rules
1. Every src/ai_qa/ module has a mirror test in tests/unit/
2. ALL LLM calls are mocked in unit tests — never make real API calls
3. Use respx for httpx mocking or manual mock providers
4. Integration tests are in tests/integration/ and marked @pytest.mark.integration
5. Use fixtures from tests/conftest.py for common test data (mock providers, sample contexts)
6. Target: >90% code coverage on unit tests
7. Every check has tests for: passing case, failing case, edge cases, missing context fields

## Code Style
1. Google-style docstrings on ALL public classes and methods
2. Type hints on ALL function signatures and return types
3. No bare except clauses — always catch specific exceptions
4. Constants are UPPER_SNAKE_CASE (e.g., JUDGE_PROMPT)
5. Private methods prefixed with underscore
6. Max line length: 100 chars (ruff enforced)
7. Imports sorted by ruff (isort rules)

## File Naming Conventions
- Each check: src/ai_qa/checks/{check_name}.py
- Each test: tests/unit/checks/test_{check_name}.py
- Judge prompts: module-level constant named JUDGE_PROMPT in each check file
- Check class: PascalCase of check name + "Check" (HallucinationCheck, ToolUseCheck)
- Registration: @register("check_name") decorator above the class

## Error Handling Philosophy
1. All custom exceptions inherit from AIQAError (in src/ai_qa/exceptions.py)
2. Every error message MUST include how to fix the problem
3. ProviderNotConfiguredError → lists install commands and alternative checks
4. CheckNotFoundError → lists all available checks
5. ContextValidationError → shows example of correct Context
6. Provider API errors wrap original exception with status code and suggestion
7. Checks NEVER crash the Runner — exceptions become failed CheckResults with evidence

## Check Implementation Pattern
Every LLM-powered check follows this exact pattern:
```python
from ai_qa.checks import register
from ai_qa.core.check import BaseCheck
from ai_qa.core.context import Context
from ai_qa.core.result import CheckResult
from ai_qa.providers import get_provider

JUDGE_PROMPT = """..."""  # Module-level constant

@register("check_name")
class CheckNameCheck(BaseCheck):
    requires_llm = True
    version = "1.0.0"
    description = "What this check does"

    def validate_context(self, ctx: Context) -> None:
        # Assert required fields, raise ContextValidationError if missing
        ...

    def evaluate(self, ctx: Context) -> CheckResult:
        self.validate_context(ctx)
        provider = get_provider()
        resp = provider.judge(
            system=JUDGE_PROMPT,
            user=f"INPUT: {ctx.input}\nOUTPUT: {ctx.output}\n..."
        )
        return CheckResult(
            name=self.name,
            score=resp["score"],
            passed=resp["score"] >= self.threshold,
            threshold=self.threshold,
            evidence=resp["evidence"],
            cost_usd=resp.get("cost_usd", 0.0),
            tokens_used=resp.get("tokens", {}),
            check_version=self.version,
        )
```

## Key Design Decisions
- validate() returns ValidationResult, never raises (unless misconfigured)
- Runner executes deterministic checks first (fast/free), then LLM checks (slow/paid)
- Cache keys include check version + prompt hash — version bump invalidates cache
- SuiteReport.assert_pass_rate() is the CI gate mechanism
- The CLI is a thin wrapper over the Python API — never duplicates logic
