---
name: ao-calibrator
description: "Meta-skill that challenges and calibrates confidence levels. Disagrees by default to catch miscalibration before and after iterations."
category: analysis
invokes: [ao-interview]
invoked_by: [ao-planning, ao-retrospective]
state_files:
  read: [constitution.md, focus.json, issues/active.jsonl, memory.md]
  write: [memory.md, issues/events.jsonl]
references: [.ao/references/confidence.md]
---

# Confidence Calibration

> **"Disagree by default. Challenge every confidence assignment."**

Humans and agents both miscalibrate confidence. This meta-skill exists specifically to catch:
- **Overconfidence** → leads to insufficient testing, bugs in production
- **Underconfidence** → wastes effort on trivial issues

## Purpose

- Challenge stated confidence levels with evidence
- Compare against historical outcomes for similar work
- Identify confidence anchoring bias
- Recommend adjustments with justification
- Track calibration accuracy over time

## Invocation Points

1. **Pre-Implementation**: After planning, before implementation begins
2. **Post-Retrospective**: After work completes, before archiving

```
Planning → [CALIBRATOR] → Implementation → Validation → Review → Retrospective → [CALIBRATOR] → Archive
```

## Calibration Signals

### Complexity Indicators (raise confidence concern)

| Signal | Weight | Interpretation |
|--------|--------|----------------|
| 5+ files modified | High | Cross-cutting change |
| New dependencies added | Medium | Integration risk |
| Security-tagged | High | Requires extra scrutiny |
| API changes | High | Contract risk |
| No tests in plan | High | Validation gap |
| First-time pattern | Medium | Learning curve |

### Simplicity Indicators (lower confidence concern)

| Signal | Weight | Interpretation |
|--------|--------|----------------|
| Single file change | Low | Isolated change |
| Well-tested area | Low | Safety net exists |
| Routine refactor | Low | Mechanical change |
| Documentation only | Low | No runtime risk |
| Team has done similar | Low | Known territory |

### Historical Patterns

| Pattern | Confidence Impact |
|---------|-------------------|
| Similar issues had rework | LOWER |
| Similar issues passed first time | RAISE |
| Similar issues regressed later | LOWER |
| Similar issues under-scoped | LOWER |

## Calibration Process

### Phase 1: Gather Evidence

```markdown
### Calibration Evidence

**Issue**: {issue_id}
**Stated Confidence**: {low | normal | high}

**Complexity Signals**:
- [ ] Files to modify: {count}
- [ ] New dependencies: {yes/no}
- [ ] Security-related: {yes/no}
- [ ] API changes: {yes/no}
- [ ] Test coverage: {adequate/gap/unknown}

**Historical Similar Issues**:
- {issue_1}: stated {confidence} → actual {outcome}
- {issue_2}: stated {confidence} → actual {outcome}
```

### Phase 2: Analyze Calibration

**Compare stated vs signals:**

```
stated = issue.confidence  # low, normal, high

complexity_score = sum(complexity_signals)
# 0-2: simple, 3-5: moderate, 6+: complex

historical_accuracy = analyze_similar_issues()
# overconfident, accurate, underconfident

# Calculate recommended confidence
if complexity_score >= 6 and stated == "high":
    recommend = "LOWER"
elif complexity_score <= 2 and stated == "low":
    recommend = "RAISE"
elif historical_accuracy == "overconfident":
    recommend = "LOWER"
else:
    recommend = "KEEP"
```

### Phase 3: Produce Recommendation

```markdown
## Calibration Recommendation

**Issue**: {issue_id} — {title}
**Stated Confidence**: {low | normal | high}

### Analysis

| Factor | Value | Impact |
|--------|-------|--------|
| Files to modify | {N} | {raises/lowers/neutral} |
| Dependencies | {yes/no} | {raises/lowers/neutral} |
| Security-related | {yes/no} | {raises/lowers/neutral} |
| Historical accuracy | {pattern} | {raises/lowers/neutral} |
| Test coverage | {status} | {raises/lowers/neutral} |

### Recommendation

**{RAISE | KEEP | LOWER}** confidence to **{new_level}**

**Justification**: {explanation}

### If Confidence Changes >1 Level

{Warning about significant miscalibration and suggestion to re-plan}
```

## Bias Detection

### Anchoring Bias

**Pattern**: Always marking issues as "normal" regardless of actual complexity.

**Detection**:
```
if count_by_confidence["normal"] > 80%:
    warn("Potential anchoring bias: most issues marked NORMAL")
```

### Overconfidence Bias

**Pattern**: Consistently marking HIGH, but outcomes show rework.

**Detection**:
```
if high_confidence_rework_rate > 30%:
    warn("Potential overconfidence: 30%+ of HIGH issues required rework")
```

### Risk Aversion Bias

**Pattern**: Consistently marking LOW, wasting effort on trivial work.

**Detection**:
```
if low_confidence_first_pass_rate > 90%:
    warn("Potential underconfidence: 90%+ of LOW issues passed first time")
```

## Configuration

In `.agent/ops/constitution.md`:

```yaml
confidence_policy:
  calibrator_enabled: true
  calibrator_strictness: moderate  # lenient | moderate | strict
  auto_accept_keep: true           # Skip confirmation if KEEP
  require_justification_override: true
  track_accuracy: true
```

### Strictness Levels

| Level | Behavior |
|-------|----------|
| **lenient** | Only flag egregious miscalibration (>5 signals off) |
| **moderate** | Flag moderate miscalibration (>3 signals off) |
| **strict** | Challenge any discrepancy between signals and stated confidence |

## Post-Retrospective Calibration

After work completes, compare prediction vs outcome:

```markdown
## Post-Completion Calibration

**Issue**: {issue_id}
**Stated Confidence**: {level}
**Outcome**: {first_pass | minor_rework | major_rework | failed}

### Calibration Accuracy

| Metric | Predicted | Actual |
|--------|-----------|--------|
| Pass first time | {yes/no} | {yes/no} |
| Validation failures | {0-1} | {N} |
| Review findings | {minor} | {severity} |
| Rework required | {no} | {yes/no} |

### Calibration Result

**Was confidence accurate?** {yes | overconfident | underconfident}

### Durable Learning

{If miscalibrated, what signal was missed? Record for future.}
```
