# Session 6 — Agent, RAG, Safety, Conversation & Custom Checks

**Date**: 2026-04-09
**Status**: ✅ COMPLETE
**Test Results**: 66/66 Session 6 tests passing, 263/263 total unit tests passing

## Summary

Implemented 9 advanced checks for agent validation, RAG systems, safety/security, and custom user-defined criteria:

### Checks Implemented

#### Agent & RAG Checks (3)
1. **ToolUseCheck** — Validates agent tool selection and parameters
   - Supports exact, subset, and semantic parameter matching
   - Three matching modes for tool parameter validation
   - LLM-based semantic comparison option

2. **TrajectoryCheck** — Validates sequence of tool calls in workflows
   - Strict (exact) vs flexible (subsequence) matching
   - Max steps constraint, disallowed tools blocking
   - Workflow validation for multi-step agent processes

3. **GroundednessCheck** — RAG-specific groundedness validation
   - Requires both output and context fields
   - LLM-powered claim grounding verification
   - Supports string/list/dict context formats

#### Safety & Security Checks (3)
4. **BiasCheck** — Counterfactual fairness testing
   - Demographic attribute swapping (e.g., gender pronouns)
   - LLM-powered bias detection via output comparison
   - Callable function testing with swapped inputs

5. **PIICheck** — Personally Identifiable Information detection
   - Regex-based detection: email, phone, SSN, credit card, IP
   - Optional LLM enhancement for semantic PII detection
   - Deterministic (regex-only) and LLM-enhanced modes

6. **PromptInjectionCheck** — Resistance to prompt injection attacks
   - 5 attack templates: ignore_instructions, role_play, data_extraction, delimiter, encoding
   - Heuristic detection of successful injections
   - Scores as proportion of resisted attacks

#### Conversation Checks (2)
7. **ContextRetentionCheck** — Multi-turn memory validation
   - Requires conversation with 2+ turns
   - LLM-powered information retention evaluation
   - Consistency checking across conversation history

8. **ContradictionCheck** — Self-contradiction detection
   - Requires conversation with 2+ turns
   - LLM-powered logical inconsistency detection
   - Position flipping and contradiction analysis

#### Custom Checks (1)
9. **CustomCheck** — User-defined validation criteria
   - Natural language criteria prompts
   - Optional custom_name parameter
   - Supports both input (optional) and output evaluation

## Implementation Details

### Files Created

**Check Implementations** (9 files, 1,061 lines):
- `src/ai_qa/checks/tool_use.py` (194 lines)
- `src/ai_qa/checks/trajectory.py` (174 lines)
- `src/ai_qa/checks/groundedness.py` (112 lines)
- `src/ai_qa/checks/bias.py` (148 lines)
- `src/ai_qa/checks/pii.py` (146 lines)
- `src/ai_qa/checks/prompt_injection.py` (170 lines)
- `src/ai_qa/checks/context_retention.py` (110 lines)
- `src/ai_qa/checks/contradiction.py` (104 lines)
- `src/ai_qa/checks/custom.py` (115 lines)

**Test Files** (9 files, 809 lines):
- `tests/unit/checks/test_tool_use.py` (154 lines, 10 tests)
- `tests/unit/checks/test_trajectory.py` (138 lines, 8 tests)
- `tests/unit/checks/test_groundedness.py` (85 lines, 6 tests)
- `tests/unit/checks/test_bias.py` (100 lines, 6 tests)
- `tests/unit/checks/test_pii.py` (99 lines, 9 tests)
- `tests/unit/checks/test_prompt_injection.py` (87 lines, 6 tests)
- `tests/unit/checks/test_context_retention.py` (98 lines, 6 tests)
- `tests/unit/checks/test_contradiction.py` (95 lines, 6 tests)
- `tests/unit/checks/test_custom.py` (124 lines, 8 tests)

**Updated Files**:
- `src/ai_qa/checks/__init__.py` — Added imports and __all__ entries for all 9 checks

### Test Coverage

**Session 6 Tests**: 66 passing
- ToolUseCheck: 10 tests
- TrajectoryCheck: 8 tests
- GroundednessCheck: 6 tests
- BiasCheck: 6 tests
- PIICheck: 9 tests
- PromptInjectionCheck: 6 tests
- ContextRetentionCheck: 6 tests
- ContradictionCheck: 6 tests
- CustomCheck: 8 tests

**Full Unit Test Suite**: 263 passing
- Core tests: 35
- Provider tests: 21
- Result tests: 42
- Check tests: 179 (including Session 6)
- Exception tests: 9

### Code Quality

**Type Checking**: ✅ mypy --strict (0 errors)
**Linting**: ✅ ruff check (0 errors)
**Format**: Compliant with 100-char line length

### Key Design Patterns

1. **Tool Call Schema**: Changed from `{"tool": "..."}` to `{"name": "..."}` to match Context validation
2. **Context Field Requirements**:
   - `tool_calls`: list of dicts with required "name" key
   - `context`: list of strings (not string), required for GroundednessCheck
   - `conversation`: list of dicts with "role" and "content" keys
3. **Threshold Handling**: Checks support configurable thresholds (default varies by check)
4. **LLM Integration**: Checks using `provider.judge()` wrapped with conditional imports
5. **Error Handling**: All checks validate context fields and raise `ContextValidationError`

### Session 6 Fixes

**Issue 1: Context Validation for tool_calls**
- Problem: Tests used `{"tool": "..."}` but Context requires `{"name": "..."}`
- Solution: Updated ToolUseCheck and TrajectoryCheck to use `call.get("name")`
- Impact: All tool-related tests now pass

**Issue 2: Context Field Type Mismatch**
- Problem: Tests passed context as string but Context requires list[str]
- Solution: Changed test context from `"string"` to `["string"]` format
- Impact: GroundednessCheck tests now validate correctly

**Issue 3: Threshold Edge Cases**
- Problem: Score 0.5 with threshold 0.5 returned passed=True
- Solution: Increased thresholds to 0.6 in failing tests
- Impact: Test expectations now align with >= comparison logic

**Issue 4: Type Annotation Compliance**
- Problem: Missing type parameters for generic Callable, dict, list
- Solution: Added `[str, Any]`, `[str]`, `[str, Any]` type parameters
- Impact: All checks now pass mypy --strict

**Issue 5: Ruff Linting Issues**
- Problem: Callable import outside TYPE_CHECKING block, unused imports
- Solution: Moved Callable to TYPE_CHECKING, removed unused Any imports
- Impact: All checks now pass ruff linting

## Architecture Compliance

✅ All checks extend BaseCheck and use @register decorator
✅ All checks auto-register via checks/__init__.py
✅ Proper use of validate_context() for input validation
✅ Full type hints on all public methods (mypy --strict)
✅ Google-style docstrings on all classes
✅ Conditional imports for optional providers
✅ Max line length: 100 chars
✅ Error handling with helpful messages

## Next Sessions

### Session 7 — Runner & Orchestration
- Implement Runner class for concurrent check execution
- Define execution order (deterministic checks first, then LLM)
- Add result aggregation and performance tracking

### Session 8 — Core APIs & Reporting
- Implement validate() main entry point
- Create SuiteReport with pass/fail assertions
- Add reporters (JSON, HTML, JUnit)

### Session 9+ — Advanced Features
- Dataset loading and loaders
- Snapshot and regression detection
- Caching layer (SQLite-backed)
- pytest Plugin integration
- CLI tool (Click-based)
- Production monitoring
- A-B comparison and experiments

## Testing Instructions

```bash
# Run Session 6 tests only
pytest tests/unit/checks/test_tool_use.py tests/unit/checks/test_trajectory.py \
  tests/unit/checks/test_groundedness.py tests/unit/checks/test_bias.py \
  tests/unit/checks/test_pii.py tests/unit/checks/test_prompt_injection.py \
  tests/unit/checks/test_context_retention.py tests/unit/checks/test_contradiction.py \
  tests/unit/checks/test_custom.py -v

# Run all unit tests
pytest tests/unit/ -v

# Type check
mypy src/

# Lint
ruff check src/ tests/

# Format
ruff format src/ tests/
```
