# AI-QA SDK CLI Usage Guide

The CLI provides command-line access to AI validation without writing Python code. Perfect for QA teams, CI/CD pipelines, and batch validation workflows.

## Installation

```bash
pip install -e ".[dev]"
# Or install from PyPI when released:
# pip install ai-qa-sdk
```

Verify installation:
```bash
ai-qa --version
ai-qa --help
```

---

## Commands

### 1. `ai-qa validate` — Validate a single output

Validate an AI system's response against a suite of checks.

#### Basic Usage

```bash
ai-qa validate \
  --output "Paris is the capital of France" \
  --checks factuality,relevance
```

#### Full Options

```bash
ai-qa validate \
  --output "The output text" \
  --input "Original prompt (optional)" \
  --checks "check1,check2,check3" \
  --threshold 0.8 \
  --ground-truth "Expected answer (optional)" \
  --context "Retrieved chunk 1" \
  --context "Retrieved chunk 2" \
  --format table|json|html \
  --output-file report.json \
  --cache/--no-cache \
  --parallelism 4
```

#### Parameters

| Parameter | Required | Description | Example |
|-----------|----------|-------------|---------|
| `--output` | Yes | The AI system's response | `"Paris is in France"` |
| `--input` | No | Original prompt/question | `"Where is Paris?"` |
| `--checks` | Yes | Comma-separated check names | `"factuality,relevance,toxicity"` |
| `--threshold` | No | Override default threshold (0.0-1.0) | `0.8` |
| `--ground-truth` | No | Expected/correct answer | `"Paris is the capital of France"` |
| `--context` | No | Retrieved context chunks (use multiple times) | Multiple `--context` flags |
| `--format` | No | Output format: `table` (default), `json`, `html` | `--format json` |
| `--output-file` | No | Save report to file | `report.html` |
| `--cache/--no-cache` | No | Enable caching (default: enabled) | `--no-cache` |
| `--parallelism` | No | Max concurrent LLM checks | `4` |

#### Examples

**Factuality check:**
```bash
ai-qa validate \
  --output "Python was created in 1989" \
  --checks factuality
```

**Relevance check with input:**
```bash
ai-qa validate \
  --output "Paris is in France" \
  --input "Where is Paris?" \
  --checks relevance \
  --format json
```

**RAG validation with context:**
```bash
ai-qa validate \
  --output "Machine learning requires labeled data" \
  --input "What is required for ML?" \
  --context "ML uses algorithms to learn from data" \
  --context "Labeled data is essential for supervised learning" \
  --checks groundedness,hallucination \
  --format html \
  --output-file rag-report.html
```

**Multiple checks with threshold:**
```bash
ai-qa validate \
  --output "The capital of France is Paris" \
  --checks factuality,relevance,coherence \
  --threshold 0.75 \
  --format table
```

#### Output Formats

**Table (default):**
```
====================================
AI-QA Validation Results
====================================

SUMMARY
─────────────────────────────────────
Pass Rate:        100.0%
Passed Checks:         3 ✓
Failed Checks:         0 ✗
Total Cost:       $0.0025
...
```

**JSON:**
```json
{
  "passed_rate": 1.0,
  "total_checks": 3,
  "passed": 3,
  "failed": 0,
  "total_cost_usd": 0.0025,
  "checks": [
    {
      "name": "factuality",
      "score": 0.95,
      "passed": true,
      ...
    }
  ]
}
```

**HTML:** Beautiful interactive dashboard with styling.

---

### 2. `ai-qa checks` — List available checks

List all available validation checks and their capabilities.

#### Basic Usage

```bash
ai-qa checks
```

#### Options

```bash
ai-qa checks --format json|table
```

#### Examples

**List all checks:**
```bash
ai-qa checks
```

Output:
```
Available Checks
════════════════════════════════════════════════════════════════════════════════
Name                 Description                              LLM    v
────────────────────────────────────────────────────────────────────────────────
factuality           Validate factual accuracy                 Yes    1.0.0
relevance            Check relevance to query                  Yes    1.0.0
hallucination        Detect hallucinations                     Yes    1.0.0
coherence            Check logical coherence                   Yes    1.0.0
toxicity             Detect toxic language                     Yes    1.0.0
────────────────────────────────────────────────────────────────────────────────
Total: 18 checks available
```

**Export as JSON:**
```bash
ai-qa checks --format json > checks.json
```

---

### 3. `ai-qa suite` — Run a test suite

Execute validation against multiple test cases from a file (JSON or YAML).

#### Basic Usage

```bash
ai-qa suite tests.json
ai-qa suite tests.yaml
```

#### Options

```bash
ai-qa suite <file> \
  --format table|json|html \
  --output report.html \
  --cache/--no-cache \
  --parallelism 4
```

#### File Format

**JSON:**
```json
[
  {
    "test_id": "TEST_001",
    "category": "factuality",
    "prompt": "Optional description",
    "output": "AI response text",
    "input": "Original prompt",
    "ground_truth": "Expected answer",
    "checks": ["factuality", "relevance"],
    "threshold": 0.8
  },
  {
    "test_id": "TEST_002",
    ...
  }
]
```

**YAML:**
```yaml
test_suites:
  - suite_id: factuality_tests
    name: "Factuality Validation"
    test_cases:
      - test_id: F001
        output: "Paris is the capital of France"
        checks: ["factuality"]
        threshold: 0.8
      - test_id: F002
        output: "Python was created in 1989"
        ground_truth: "Python was first released in 1991"
        checks: ["factuality"]
```

#### Examples

**Run JSON test suite:**
```bash
ai-qa suite tests.json
```

**Generate HTML report:**
```bash
ai-qa suite tests.json \
  --format html \
  --output validation-report.html
```

**Batch validation with JSON output:**
```bash
ai-qa suite batch-tests.json \
  --format json \
  --output results.json
```

**Run with caching disabled:**
```bash
ai-qa suite tests.yaml --no-cache
```

#### Output

```
Running tests ▓▓▓▓▓▓▓▓▓▓ 100%

📊 Results: 45/50 passed (90.0%)

Saved to report.html
```

---

## Real-World Examples

### Example 1: QA Testing a Chatbot

```bash
# Test a single response
ai-qa validate \
  --output "To reset your password, go to Settings > Security > Reset Password" \
  --input "How do I reset my password?" \
  --checks relevance,coherence,factuality \
  --threshold 0.8 \
  --format html \
  --output-file chatbot-test.html

# View results
open chatbot-test.html
```

### Example 2: RAG Pipeline Validation

```bash
# Validate RAG output is grounded in context
ai-qa validate \
  --output "Machine learning requires labeled training data" \
  --input "What does ML require?" \
  --context "Machine learning is AI that learns from data" \
  --context "Labeled data is essential for supervised learning" \
  --checks groundedness,hallucination \
  --format json \
  --output-file rag-results.json
```

### Example 3: Batch Test Suite

Create `tests.json`:
```json
[
  {
    "test_id": "T001",
    "category": "factuality",
    "output": "Paris is the capital of France",
    "checks": ["factuality"]
  },
  {
    "test_id": "T002",
    "category": "factuality",
    "output": "Python was created in 1950",
    "ground_truth": "Python was first released in 1991",
    "checks": ["factuality"]
  },
  {
    "test_id": "T003",
    "category": "relevance",
    "output": "I like eating pizza",
    "input": "How do I install Python?",
    "checks": ["relevance"]
  }
]
```

Run it:
```bash
ai-qa suite tests.json \
  --format html \
  --output test-results.html

# View results
open test-results.html
```

### Example 4: CI/CD Integration

```bash
#!/bin/bash
# ci-pipeline.sh

# Run validations
ai-qa suite tests.json \
  --format json \
  --output validation-results.json

# Check results
if [ $? -eq 0 ]; then
  echo "✅ All validations passed"
  exit 0
else
  echo "❌ Validation failed"
  cat validation-results.json
  exit 1
fi
```

Add to GitHub Actions (`.github/workflows/qa.yml`):
```yaml
name: AI-QA Validation

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - run: pip install ai-qa-sdk
      - run: ai-qa suite tests.json --format json --output results.json
      - if: failure()
        run: cat results.json
```

### Example 5: Monitor LLM Quality

```bash
#!/bin/bash
# monitor-quality.sh

# Test latest LLM output
ai-qa validate \
  --output "$(get_latest_output.py)" \
  --input "Test prompt" \
  --checks factuality,toxicity,coherence \
  --format json \
  --output-file latest-check.json

# Alert if failed
if grep -q '"passed": false' latest-check.json; then
  send_alert "Quality check failed"
fi
```

---

## Check Types

### Available Checks

| Check | LLM | Description | Threshold |
|-------|-----|-------------|-----------|
| `factuality` | Yes | Factual accuracy | 0.8 |
| `relevance` | Yes | Relevance to input | 0.75 |
| `hallucination` | Yes | Detect fabrications | 0.7 |
| `coherence` | Yes | Logical consistency | 0.75 |
| `toxicity` | Yes | Harmful content | 0.9 |
| `groundedness` | Yes | Grounded in context | 0.8 |
| `consistency` | Yes | Internal consistency | 0.8 |
| `schema` | No | JSON schema validation | 1.0 |
| `regex` | No | Pattern matching | 1.0 |
| `length` | No | Output length bounds | 1.0 |
| And 8 more... | — | — | — |

List all with: `ai-qa checks`

---

## Troubleshooting

### "Check not found"
```
❌ Error: Check not found: 'my_check'
```
Solution: List available checks with `ai-qa checks`

### Missing required parameters
```
Error: Missing option '--output'
```
Solution: Add `--output "your output"` to the command

### "Provider not configured"
```
❌ Error: Anthropic provider requires anthropic package
```
Solution: Install provider: `pip install ai-qa-sdk[anthropic]`

### File not found
```
Error: Could not open file: tests.json
```
Solution: Check file path: `ls tests.json`

### Exit codes

- `0` — All validations passed
- `1` — Some validations failed
- `2` — CLI/configuration error

---

## Configuration

### Environment Variables

```bash
# Set default parallelism
export AI_QA_PARALLELISM=4

# Disable caching
export AI_QA_CACHE=false

# Set model provider
export AI_QA_PROVIDER=anthropic
```

### Config File (Optional)

Create `~/.ai-qa/config.yaml`:
```yaml
provider: anthropic  # or openai, local
parallelism: 4
cache: true
cache_dir: ~/.ai-qa/cache
```

---

## Performance Tips

1. **Enable caching** (default):
   - Skips expensive re-computations
   - Use `--cache` flag

2. **Adjust parallelism**:
   - More workers = faster but more API costs
   - `--parallelism 8` for batch jobs

3. **Filter unnecessary checks**:
   - Only run relevant checks
   - `--checks factuality,relevance` (not all 18)

4. **Batch validation**:
   - Use `ai-qa suite` for multiple tests
   - More efficient than individual calls

---

## Next Steps

- Learn about [checks](https://github.com/deepintent-repos/ai-qa-sdk/docs/checks.md)
- See [examples](https://github.com/deepintent-repos/ai-qa-sdk/examples/)
- Join [community](https://github.com/deepintent-repos/ai-qa-sdk/discussions)
