# Session 4 Summary — Check Registry & Deterministic Checks

## Overview
Session 4 implemented the check registration system and four deterministic (non-LLM) checks:
- **CheckRegistry**: Pattern for registering and discovering checks
- **@register decorator**: Automatic check registration
- **SchemaCheck**: JSONSchema validation
- **RegexCheck**: Regex pattern matching
- **JsonFormatCheck**: JSON format validation
- **LengthCheck**: Output length bounds validation

**Status**: ✅ COMPLETE
- 76 new tests added (all passing)
- Total test count: 160 (all passing)
- Code quality: mypy strict ✅, ruff ✅

## Implementation Details

### 1. Check Registry (`src/ai_qa/checks/registry.py`)
- **CheckRegistry class**: Manages check registration
  - `register(name, check_class)`: Register a check class
  - `get(name)`: Retrieve check class by name (raises `CheckNotFoundError` if not found)
  - `list()`: List all registered check names (sorted)
  - `instantiate(name, **kwargs)`: Create check instance with kwargs
- **@register decorator**: Decorator for class-level registration
  - Sets `check.name` attribute
  - Calls `CheckRegistry.register()` automatically
- **Entry-point loading**: Auto-imported in `__init__.py` ensures all decorated classes register on module import

### 2. SchemaCheck (`src/ai_qa/checks/schema.py`)
- **Purpose**: Validate output against a JSONSchema
- **Configuration**:
  - `schema`: JSONSchema dict (required)
  - `threshold`: Score threshold (default 0.5)
- **Behavior**:
  - Accepts dict/list output directly
  - Parses JSON strings to dict/list
  - Validates using `jsonschema.validate()`
  - Returns 1.0 if valid, 0.0 if invalid
- **Example**:
  ```python
  schema = {
      "type": "object",
      "properties": {"name": {"type": "string"}},
      "required": ["name"]
  }
  check = SchemaCheck(schema=schema)
  result = check.evaluate(Context(output={"name": "John"}))
  assert result.passed
  ```
- **Tests**: 13 tests covering valid/invalid dicts, JSON strings, lists, complex schemas

### 3. RegexCheck (`src/ai_qa/checks/regex.py`)
- **Purpose**: Match output against a regex pattern
- **Configuration**:
  - `pattern`: Regex pattern string (required)
  - `flags`: Regex flags (e.g., `re.IGNORECASE`, default 0)
  - `threshold`: Score threshold (default 0.5)
- **Behavior**:
  - Validates context ensures output is a string (raises `ContextValidationError` otherwise)
  - Uses `pattern.search()` (finds pattern anywhere)
  - Returns 1.0 if match found, 0.0 if no match
- **Example**:
  ```python
  check = RegexCheck(pattern=r"^[a-z]+@[a-z]+\.[a-z]+$")
  result = check.evaluate(Context(output="user@example.com"))
  assert result.passed
  ```
- **Tests**: 17 tests covering email patterns, case sensitivity, multiline flags, complex URL patterns

### 4. JsonFormatCheck (`src/ai_qa/checks/json_format.py`)
- **Purpose**: Validate output is valid JSON
- **Configuration**:
  - `threshold`: Score threshold (default 0.5)
- **Behavior**:
  - Dict/list outputs are valid JSON (no parsing needed)
  - Parses JSON strings using `json.loads()`
  - Returns 1.0 if valid JSON, 0.0 if invalid
  - Handles edge cases: empty objects, arrays, null, booleans, numbers
- **Example**:
  ```python
  check = JsonFormatCheck()
  result = check.evaluate(Context(output='{"key": "value"}'))
  assert result.passed
  ```
- **Tests**: 16 tests covering valid JSON strings, dicts, lists, invalid JSON, edge cases

### 5. LengthCheck (`src/ai_qa/checks/length.py`)
- **Purpose**: Validate output length is within bounds
- **Configuration**:
  - `min_length`: Minimum length (-1 for no minimum)
  - `max_length`: Maximum length (-1 for no maximum)
  - `threshold`: Score threshold (default 0.5)
- **Behavior**:
  - String length: character count
  - Dict/list length: number of items
  - Returns 1.0 if within bounds, 0.0 if outside bounds
  - Requires at least one of min_length or max_length
- **Example**:
  ```python
  check = LengthCheck(min_length=5, max_length=20)
  result = check.evaluate(Context(output="hello world"))
  assert result.passed
  ```
- **Tests**: 22 tests covering strings, dicts, lists, exact boundaries, unicode

### 6. Registry Tests (`tests/unit/checks/test_registry.py`)
- 9 tests covering:
  - `@register` decorator registration
  - `CheckRegistry.get()` success and error cases
  - `CheckRegistry.list()` sorted output
  - `CheckRegistry.instantiate()` with kwargs
  - Name attribute setting
  - Registration overwriting

## Architecture Decisions

1. **Class-level registration**: Check name is stored both as decorator argument and class attribute
2. **No entry-point config required**: Deterministic checks are auto-imported and registered on module load
3. **Context validation**: Each check validates required context fields (e.g., RegexCheck requires string output)
4. **Type narrowing**: Used `assert isinstance()` to help mypy understand type narrowing after validation
5. **Default thresholds**: All checks default to threshold=0.5 for pass/fail
6. **Immutable checks**: No state modification after initialization

## Test Coverage

**New Session 4 Tests**: 76 tests
- test_registry.py: 9 tests
- test_schema.py: 13 tests
- test_regex.py: 17 tests
- test_json_format.py: 16 tests
- test_length.py: 21 tests

**Total Project Tests**: 160 tests (all passing)
- Sessions 1-3 core: 84 tests
- Session 4 checks: 76 tests

## Code Quality

✅ **mypy strict**: Success (53 source files)
✅ **ruff**: All checks passed (E, F, W)
✅ **pytest**: 160/160 passed (0.83s)

## Files Modified/Created

### New Files
- `src/ai_qa/checks/registry.py` — Check registry and @register decorator
- `tests/unit/checks/test_registry.py` — Registry tests

### Modified Files
- `src/ai_qa/checks/schema.py` — SchemaCheck implementation
- `src/ai_qa/checks/regex.py` — RegexCheck implementation
- `src/ai_qa/checks/json_format.py` — JsonFormatCheck implementation
- `src/ai_qa/checks/length.py` — LengthCheck implementation
- `src/ai_qa/checks/__init__.py` — Auto-import registry and all checks
- `tests/unit/checks/test_schema.py` — SchemaCheck tests
- `tests/unit/checks/test_regex.py` — RegexCheck tests
- `tests/unit/checks/test_json_format.py` — JsonFormatCheck tests
- `tests/unit/checks/test_length.py` — LengthCheck tests
- `pyproject.toml` — Updated Python requirement to >=3.9 (from >=3.10)

### Removed Files
- `tests/unit/test_registry.py` — Duplicate stub file (consolidated into tests/unit/checks/test_registry.py)

## Key Patterns Established

1. **Check Implementation Pattern**:
   ```python
   @register("check_name")
   class CheckNameCheck(BaseCheck):
       requires_llm = False
       version = "1.0.0"
       description = "..."

       def __init__(self, threshold=0.5, required_param=None, **kwargs):
           super().__init__(threshold=threshold, **kwargs)
           # Validate required params

       def validate_context(self, ctx):
           # Validate required context fields

       def evaluate(self, ctx):
           self.validate_context(ctx)
           # Logic
           return CheckResult(...)
   ```

2. **Registry Usage**:
   ```python
   check = CheckRegistry.instantiate("schema", schema={...})
   result = check.run(context)
   ```

3. **Test Pattern**:
   - Passing case
   - Failing case
   - Edge cases
   - Configuration validation
   - Attribute checks (name, version, requires_llm)

## Next Steps (Session 5)

Session 5 will implement LLM-powered checks:
1. HallucinationCheck (hallucination detection)
2. FactualityCheck (factuality validation)
3. RelevanceCheck (relevance scoring)
4. ToxicityCheck (toxicity detection)
5. ConsistencyCheck (output consistency)

All LLM checks will:
- Use provider.judge() for LLM evaluation
- Define JUDGE_PROMPT module constant
- Support caching and cost tracking
- Include comprehensive test coverage

## Notes

- Python 3.9 compatibility maintained via `from __future__ import annotations`
- All deterministic checks are synchronous (async variants planned for later)
- Entry-point loading enables future third-party check plugins
- CheckRegistry validates check names and provides helpful error messages
