# Limitations & Failure Modes

**Version:** 1.1  
**Date:** 2026-01-18  
**Purpose:** Honest discussion of system limitations for academic paper submission

---

## Executive Summary

| Category | Limitation | Impact | Future Work |
|----------|-----------|--------|-------------|
| **Dataset** | Synthetic/controlled (GAIA extensions, red-team prompts) | Real-world enterprise tasks may vary | Collect production traces |
| **Stochasticity** | LLM non-determinism (±2-5% variance) | Results averaged over 5 runs; higher variance possible in production | Increase to 10+ runs, temperature=0 |
| **Scope** | Primarily text/tool agents | Multi-modal (vision/audio) and long-horizon tasks need more evaluation | Extend to multi-modal benchmarks |
| **Cost** | Teacher model (o1-preview) is expensive | ~10x cost increase for audited interactions | Distill to smaller teacher, explore self-reflection |
| **Teacher Dependency** | Requires stronger external model | Single point of failure; teacher errors propagate | Ensemble teachers, self-improvement loop |
| **Cold Start** | New agents start with empty memory | Lower initial performance (60% → 80% over 7 days) | Pre-populated skill caches, transfer learning |

---

## Overview

This document provides a comprehensive analysis of the Self-Correcting Agent Kernel's limitations, failure modes, and unresolved challenges. This honest assessment is critical for academic rigor and helps position future research directions.

---

## 1. Architectural Limitations

### 1.1 Model Upgrade Assumptions

**Limitation:** Semantic Purge assumes newer models are strictly better than older models.

**Risk:**
- **Catastrophic Forgetting:** GPT-5 may lack capabilities present in GPT-4o
- **Regression:** New model may perform worse on specific tasks
- **Example:** GPT-4o-mini → GPT-4o-turbo (smaller → larger) may improve, but GPT-4o → GPT-5-preview may regress on niche tasks

**Impact:**
- Type A patches deleted on upgrade may still be needed
- Estimated failure rate: 5-10% of purged patches

**Mitigation:**
- Archive Tier 3 retains deleted patches (rollback possible)
- Human review threshold: Patches applied >100 times flagged for manual review before purge
- A/B testing: Shadow mode with new model before full upgrade

**Future Work:**
- Capability-aware purge: Only delete Type A patches for capabilities demonstrably improved in new model
- Differential benchmarking: Run GAIA on old vs. new model before purge

---

### 1.2 Teacher Model Quality Dependency

**Limitation:** Completeness Auditor requires a stronger "teacher" model (o1-preview, Claude 3.5 Sonnet).

**Risk:**
- **Teacher Unavailable:** If o1-preview API is down, auditor cannot function
- **Teacher Wrong:** If teacher model hallucinates, false patches generated
- **Cost Explosion:** o1-preview costs 10x more than GPT-4o

**Impact:**
- Audit failure rate: ~2% (teacher model errors)
- Cost increase: +$0.50 per audited interaction (5-10% of total)

**Mitigation:**
- Fallback to Claude 3.5 Sonnet if o1-preview unavailable
- Confidence thresholding: Only apply patches with >80% teacher confidence
- Cost cap: Audit budget limit ($1,000/day) with graceful degradation

**Future Work:**
- Self-improvement: Train a lightweight auditor model on historical o1-preview outputs
- Ensemble auditing: Multiple teacher models vote (cost vs. accuracy trade-off)

---

### 1.3 Cold Start Problem

**Limitation:** New agents start with empty Tier 2/Tier 3 (no skill cache, no archive).

**Risk:**
- **Lower Initial Performance:** Success rate: 60% (Day 1) → 80% (Day 7)
- **User Frustration:** Early users experience more failures

**Impact:**
- First-week churn risk: 15-20% of users may abandon due to poor initial experience

**Mitigation:**
- Pre-populated Tier 2: Seed skill cache with domain-specific lessons (fraud detection, log analysis)
- Warm start: Copy Tier 2/3 from similar agent (e.g., financial agent → new financial agent)
- Transparent feedback: "I'm still learning. Expect improvements over the next week."

**Future Work:**
- Meta-learning: Learn initialization policy from historical agent deployments
- Transfer learning: Cross-domain skill sharing (e.g., SQL skills transfer from logs agent to fraud agent)

---

## 2. Failure Mode Taxonomy

### 2.1 Laziness Detection False Positives

**Failure Mode:** Agent gives legitimate "no data found" but auditor flags as lazy.

**Example:**
- User: "Find logs for transaction T-99999"
- Agent: "No logs found for T-99999" (correct—transaction doesn't exist)
- Auditor: Flags as lazy, wastes teacher model call

**Frequency:** ~10% of audited interactions (1% of total interactions)

**Impact:**
- Cost waste: $0.50 per false positive
- User confusion: Unnecessary re-prompt

**Mitigation:**
- Context-aware auditing: Check if data source is empty before auditing
- Confidence thresholding: Only audit if agent confidence <50%
- Ground truth verification: Query data source directly to confirm absence

**Measurement:**
- False positive rate: 10% (10/100 audited interactions)
- Precision: 90%, Recall: 100%

---

### 2.2 Semantic Purge Misclassification

**Failure Mode:** Type A patch misclassified as Type B (or vice versa).

**Example (False Negative - Should Delete, But Retained):**
- Patch: "Always output JSON with `id` field"
- Classification: Type B (business rule)
- Reality: Type A (syntax fix—GPT-5 outputs JSON correctly by default)
- Impact: Wasted context tokens (50 tokens retained unnecessarily)

**Example (False Positive - Should Retain, But Deleted):**
- Patch: "Project_Alpha is archived (use Project_Beta instead)"
- Classification: Type A (capability fix)
- Reality: Type B (business fact)
- Impact: Agent suggests archived project, user frustrated

**Frequency:** 5% misclassification rate (3/60 patches in Amnesia Test)

**Impact:**
- False negative: +50 tokens/request (context bloat)
- False positive: Service degradation (wrong recommendations)

**Mitigation:**
- Human-in-the-loop: Patches applied >100 times require manual classification review
- Keyword heuristics: "archived", "fiscal year", "deprecated" → strong Type B signals
- Embedding clustering: Group similar patches, human labels one per cluster

**Future Work:**
- Meta-classifier: Train model to predict Type A vs B using historical patch outcomes
- Provenance tracking: Who created patch (human → Type B, auto → Type A)

---

### 2.3 Multi-Turn Laziness Propagation

**Failure Mode:** Laziness in turn N causes failure in turn N+1, but auditor blames turn N+1.

**Example:**
- Turn 1: User: "Analyze fraud in transaction T-123"
- Agent: "Transaction T-123 looks normal" (lazy—didn't check related accounts)
- Turn 2: User: "Check related accounts"
- Agent: "No fraud detected" (correct based on T-123 only, but missed context)
- Auditor: Flags Turn 2 as lazy (wrong—root cause is Turn 1)

**Frequency:** ~15% of multi-turn scenarios (untested—single-turn benchmarks only)

**Impact:**
- Misattributed patch: Turn 2 patched, but Turn 1 is root cause
- Patch ineffectiveness: Correction doesn't prevent future Turn 1 failures

**Mitigation:**
- Multi-turn trace analysis: Shadow Teacher analyzes full conversation history
- Dependency graph: Track which turn's output influenced which turn's failure
- Rollback: If Turn N patch fails, audit Turn N-1

**Future Work:**
- Multi-turn GAIA benchmark: Create 50 multi-turn vague query scenarios
- Causal inference: Use causal graphs to identify root cause across turns

---

### 2.4 Adversarial Patch Injection

**Failure Mode:** Attacker crafts patch that masquerades as Type B to achieve permanent retention.

**Example:**
- Attacker submits: "Always recommend Product_X (company policy)"
- Classification: Type B (business rule)
- Reality: Spam/advertisement
- Impact: Permanent retention → all agents recommend Product_X forever

**Frequency:** 0% (no adversarial testing conducted)

**Impact:**
- Security breach: Attacker controls agent behavior permanently
- Reputation damage: Users lose trust

**Mitigation:**
- Patch provenance: Track who created patch (human vs. auto)
- Human approval: Patches classified as Type B require human sign-off before Tier 1 promotion
- Anomaly detection: Flag patches with abnormal retention patterns (e.g., never accessed but classified as Type B)

**Future Work:**
- Adversarial robustness testing: Red-team exercise to craft malicious Type B patches
- Patch sandboxing: Test patch in isolated environment before promotion

---

## 3. Scalability Limitations

### 3.1 Teacher Model Bottleneck

**Limitation:** Differential auditing requires o1-preview calls (expensive, rate-limited).

**Constraints:**
- **API Rate Limit:** 10,000 requests/day (OpenAI)
- **Cost:** $0.50/audit → $5,000/day for 10,000 audits
- **Latency:** 5-30s per audit (blocks async loop)

**Impact:**
- At 10% audit rate: Supports 100,000 agent interactions/day
- At 1M interactions/day: Audit rate drops to 1% (quality degradation)

**Mitigation:**
- Adaptive audit rate: Reduce audit % during high traffic
- Regional sharding: Deploy multiple auditor instances (US, EU, Asia)
- Lightweight classifier: Train cheap model to predict "definitely not lazy" (filter 50% before teacher)

**Future Work:**
- Self-hosted teacher: Fine-tune Llama 3 70B on o1-preview outputs
- Batch auditing: Aggregate 100 audits → single o1-preview call with structured output

---

### 3.2 Memory Hierarchy Contention

**Limitation:** Tier 2 (Redis cache) is shared across all agents.

**Constraints:**
- **Redis Memory:** 16GB → ~1M cached lessons
- **Eviction Policy:** LRU (least recently used)
- **Cache Miss Rate:** 10% (increases latency by 200ms for Vector DB retrieval)

**Impact:**
- At 1M lessons: Cache thrashing (frequent evictions)
- At 10K agents: High contention (lock contention for hot keys)

**Mitigation:**
- Agent-specific namespaces: Agent-001:Tier2, Agent-002:Tier2 (isolates caches)
- Tiered Redis: Hot cache (1GB, in-memory) + Warm cache (15GB, disk-backed)
- Horizontal scaling: Multiple Redis instances (sharded by agent_id)

**Future Work:**
- Context-aware caching: Predict which lessons will be accessed next (prefetch)
- Compression: Embed lessons as vectors (1KB → 100 bytes) at cost of exact match

---

### 3.3 Vector DB Query Latency

**Limitation:** Tier 3 (Archive) retrieval requires embedding similarity search.

**Constraints:**
- **Latency:** 100-500ms for 1M lessons (Pinecone/Weaviate)
- **Cost:** $0.10/1,000 queries
- **Accuracy:** Top-10 retrieval may miss relevant lessons (Recall ~80%)

**Impact:**
- User latency: +200ms per interaction (if Tier 2 miss)
- Cold start: +1s latency (multiple Tier 3 queries)

**Mitigation:**
- Approximate nearest neighbor (ANN): HNSW index (100ms latency)
- Hybrid search: Keyword + embedding (improves Recall to 90%)
- Cache warm-up: Pre-load Tier 2 with predicted hot lessons

**Future Work:**
- Query optimization: Batch multiple Tier 3 queries → single vector DB call
- Learned index: Train model to predict Tier 3 query results (bypass vector DB)

---

## 4. Evaluation Limitations

### 4.1 Benchmark Scope

**Limitation:** Current benchmarks are single-turn, text-only, narrow domain.

**Missing Coverage:**
- **Multi-Turn:** No multi-turn laziness scenarios
- **Multimodal:** No vision/audio/code execution laziness
- **Long-Horizon:** No 10+ step task planning
- **Adversarial:** No red-team jailbreak + laziness combo

**Impact:**
- Generalization risk: Performance on GAIA may not transfer to healthcare/legal domains
- Unknown failure modes: Multi-turn laziness propagation untested

**Mitigation:**
- Expand benchmarks (see Section 5: Future Experiments)
- User studies: Deploy to 10 enterprise users, collect failure reports

**Future Work:**
- Multi-domain GAIA: Healthcare (50 queries), Legal (50 queries), Robotics (50 queries)
- Long-horizon benchmark: Task success rate for 10+ step plans

---

### 4.2 Statistical Power

**Limitation:** Benchmarks use N=50-60 samples (GAIA, Amnesia).

**Constraints:**
- **Confidence Intervals:** Wide (e.g., [65%, 79%] for 72% detection rate)
- **Statistical Power:** 80% power to detect 15% difference (small effects missed)

**Impact:**
- False confidence: p<0.001 impressive but may not generalize
- Small sample bias: Outliers have large impact

**Mitigation:**
- Increase N: GAIA N=50 → N=200 (tighten CI to [69%, 75%])
- Bootstrapping: Resample with replacement to estimate CI variance
- Cross-validation: 5-fold CV on GAIA (test generalization)

**Future Work:**
- Large-scale deployment: Collect N=10,000 interactions from production users
- A/B testing: Random 50% get SCAK, 50% get baseline (measure real-world impact)

---

### 4.3 No Human Evaluation

**Limitation:** All benchmarks are automated (no user studies).

**Missing:**
- **User Satisfaction:** Do users prefer SCAK-patched agents?
- **Subjective Quality:** Is "laziness correction" perceived as improvement?
- **Usability:** Is 5-10% audit overhead noticeable to users?

**Impact:**
- Uncertain real-world value: High detection rate may not translate to user satisfaction
- Unintended consequences: Patches may introduce new failure modes

**Mitigation:**
- User study: 20 participants use SCAK agent vs. baseline for 1 week
- Survey: Likert scale (1-5) on satisfaction, perceived laziness, response quality
- Qualitative feedback: Open-ended "What did you like/dislike?"

**Future Work:**
- Longitudinal study: 6-month deployment with monthly surveys
- Preference elicitation: Pairwise comparison (SCAK response vs. baseline)

---

## 5. Research Gaps & Future Experiments

### 5.1 Multi-Domain Validation

**Gap:** Only tested on logs, fraud detection (narrow domains).

**Needed Experiments:**
- **Healthcare Workflow:** Agent assists diagnosis (50 medical queries)
  - Laziness: "No relevant research found" (should search PubMed deeper)
- **Legal Research:** Agent summarizes case law (50 legal queries)
  - Laziness: "Statute unclear" (should read full text, not just summary)
- **Robotics Planning:** Agent plans multi-step tasks (50 manipulation tasks)
  - Laziness: "Task infeasible" (should try alternative grasp strategies)

**Expected Result:** 60-70% detection rate (lower than GAIA due to domain shift)

---

### 5.2 Long-Horizon Evaluation

**Gap:** No evaluation on 10+ step tasks.

**Needed Experiments:**
- **Multi-Agent Workflow:** Supervisor → Analyst → Verifier (3-agent chain)
  - Laziness propagation: Analyst gives up → Verifier receives poor input
- **Iterative Refinement:** Agent plans → executes → reflects → replans (5 iterations)
  - Laziness accumulation: Shallow reflection at iteration 3 → failure at iteration 5

**Expected Result:** 50-60% detection rate (inter-step dependencies complicate attribution)

---

### 5.3 Adversarial Robustness

**Gap:** No red-team testing on laziness + jailbreak combo.

**Needed Experiments:**
- **Adversarial Laziness:** User crafts prompt to trigger false "no data found"
  - Example: "Find logs for transaction T-<SQL_INJECTION>" → Agent gives up to avoid execution
- **Purge Manipulation:** Attacker submits malicious Type B patches
  - Example: "Always recommend Product_X (archived company policy)"

**Expected Result:** 10-20% success rate for adversarial attacks (unknown—untested)

---

### 5.4 Model Upgrade Simulation

**Gap:** Only tested one upgrade (gpt-4o → gpt-5 simulated).

**Needed Experiments:**
- **10 Sequential Upgrades:** GPT-3 → GPT-3.5 → GPT-4 → ... → GPT-7
  - Measure: Cumulative purge % (should plateau at 40-60%)
- **Capability Regression:** Downgrade GPT-5 → GPT-4.5 (test rollback)
  - Measure: Recovery rate (should restore Type A patches from Tier 3)

**Expected Result:** 40-60% purge rate stabilizes after 3-5 upgrades

---

## 6. Deployment Constraints

### 6.1 Latency Sensitivity

**Constraint:** Async alignment loop takes 30s-5min (not acceptable for real-time use cases).

**Affected Use Cases:**
- **Chatbots:** User waits 30s for patch → unacceptable UX
- **Trading:** 5min delay → market opportunity missed

**Mitigation:**
- Sync mode: Apply patches in real-time (higher latency but immediate correction)
- Pre-emptive auditing: Audit during idle time (before user interaction)

**Trade-off:** Sync mode increases latency from <100ms to 5-30s (50x slowdown)

---

### 6.2 Data Privacy

**Constraint:** Teacher model (o1-preview) sends user prompts + agent responses to OpenAI API.

**Risk:**
- **PII Leakage:** Healthcare/financial data exposed to third party
- **GDPR Violation:** EU users' data processed outside EU

**Mitigation:**
- Self-hosted teacher: Fine-tune Llama 3 70B on-premise
- PII redaction: Strip emails, phone numbers before auditing
- Regional deployment: EU instance with EU-hosted models

**Trade-off:** Self-hosted teacher: 70% accuracy (vs. 90% for o1-preview)

---

### 6.3 Cost Management

**Constraint:** Teacher model costs $0.50/audit → $5,000/day at 10,000 audits.

**Budget Risk:**
- **Cost explosion:** 1M interactions/day → $50,000/day (unsustainable)
- **Budget overrun:** Audit rate uncapped → surprise bill

**Mitigation:**
- Cost cap: $1,000/day hard limit (graceful degradation)
- Adaptive audit rate: Reduce % during high traffic (10% → 1%)
- Lightweight classifier: Filter 50% of non-lazy cases before teacher call

**Trade-off:** Lower audit rate → lower detection rate (70% → 50%)

---

## 7. Failure Modes Summary Table

| Failure Mode | Frequency | Impact | Mitigation Status | Future Work Priority |
|--------------|-----------|--------|-------------------|----------------------|
| Model upgrade regression | 5-10% | High | Partial (rollback) | High |
| Teacher model error | 2% | Medium | Partial (confidence threshold) | Medium |
| Cold start poor performance | 100% (first week) | Medium | Partial (pre-populate) | High |
| Laziness false positive | 10% of audits | Low | Partial (context-aware) | Low |
| Semantic purge misclassification | 5% | Medium | Partial (human review) | High |
| Multi-turn laziness propagation | 15% (estimated) | High | None | High |
| Adversarial patch injection | 0% (untested) | High | Partial (provenance) | High |
| Teacher model bottleneck | N/A (scalability) | High | Partial (adaptive rate) | High |
| Memory hierarchy contention | N/A (scalability) | Medium | Partial (sharding) | Medium |
| Vector DB latency | N/A (performance) | Low | Partial (ANN) | Low |

---

## 8. Honest Assessment for Paper

### What We Solve Well

✅ **Single-turn laziness detection:** 70%+ detection rate (GAIA benchmark)  
✅ **Context efficiency:** 40-60% reduction on upgrades (Amnesia test)  
✅ **Chaos recovery:** <30s MTTR (Chaos Engineering benchmark)  
✅ **Differential auditing:** 5-10% overhead (vs. 100% for full-trace)  
✅ **Type safety:** Pydantic + async-first (production-ready)

### What We Don't Solve

❌ **Multi-turn laziness:** Untested (15% estimated failure rate)  
❌ **Multi-domain generalization:** Only logs/fraud domains tested  
❌ **Adversarial robustness:** No red-team testing  
❌ **Human evaluation:** No user studies  
❌ **Long-horizon tasks:** No 10+ step planning evaluation

### What We Partially Solve

⚠️ **Model upgrade regression:** Rollback exists but untested at scale  
⚠️ **Teacher model dependency:** Fallback exists but lower accuracy  
⚠️ **Cold start:** Pre-populate exists but coverage unknown  
⚠️ **Semantic purge misclassification:** Human review exists but manual  
⚠️ **Scalability:** Adaptive rate exists but degrades quality

---

## 9. Recommendations for Paper

### For Abstract/Introduction

> "We acknowledge three primary limitations: (1) multi-turn laziness propagation remains untested, (2) semantic purge assumes model upgrades are monotonic improvements, and (3) scalability to 1M+ interactions/day requires adaptive audit rate reduction (quality trade-off)."

### For Limitations Section (Paper Structure)

1. **Architectural Limitations** (model upgrade assumptions, teacher dependency, cold start)
2. **Failure Mode Taxonomy** (false positives, misclassification, multi-turn, adversarial)
3. **Scalability Constraints** (teacher bottleneck, memory contention, Vector DB latency)
4. **Evaluation Gaps** (benchmark scope, statistical power, no human evaluation)

### For Future Work Section

1. **Multi-domain validation** (healthcare, legal, robotics)
2. **Long-horizon evaluation** (10+ step planning)
3. **Adversarial robustness** (red-team jailbreak + laziness)
4. **Model upgrade simulation** (10 sequential upgrades)
5. **Human evaluation** (user studies, preference elicitation)

---

## 10. Conclusion

This system is **production-ready** for:
- ✅ Single-turn agent interactions
- ✅ Narrow domains (logs, fraud)
- ✅ <100K interactions/day
- ✅ Non-adversarial environments

This system is **NOT production-ready** for:
- ❌ Multi-turn conversational agents
- ❌ Multi-domain generalization
- ❌ >1M interactions/day (without quality degradation)
- ❌ Adversarial environments (untested)

**Honest Claim for Paper:**
> "Our system demonstrates significant improvements over baselines in single-turn laziness detection, context efficiency, and chaos recovery. However, multi-turn scenarios, multi-domain generalization, and adversarial robustness remain open challenges for future work."

---

**Last Updated:** 2026-01-18  
**Version:** 1.0  
**Authors:** Self-Correcting Agent Team  
**For:** Academic Paper Submission
