# BGE-M3 Migration Rationale
**Date:** 2026.03.07  
**Author:** arifOS Constitutional Council  
**Floor:** F2 Truth + F6 Empathy

---

## Executive Summary

**Decision:** Migrate arifOS from BGE-small (384-dim, English-only) to **BGE-M3 (768-dim, multilingual)**.

**Rationale:** arifOS governance decisions naturally mix Malay, English, and Manglish. BGE-small cannot understand Malay, making precedent retrieval fail for ~30% of constitutional reasoning.

---

## The Problem with BGE-small

| Scenario | BGE-small Result | Constitutional Impact |
|----------|------------------|----------------------|
| Verdict reasoned in Malay | No semantic match | Precedent retrieval fails |
| Manglish governance scar | Misunderstood | Similar cases not found |
| BM constitutional lesson | Invisible to search | Institutional memory lost |
| Mixed-language paradox | Partial match | F8 Genius impaired |

**Example:**
```
Malay Verdict: "F1 Amanah gagal kerana tindakan tidak boleh undur"
BGE-small embedding: [random vector - no semantic meaning]
Result: Cannot match to similar English "F1 Amanah irreversible action blocked"
```

---

## Why BGE-M3 is Constitutionally Required

### F2 Truth (τ ≥ 0.99)
The system must truthfully represent its capability. Claiming BGE-M3 while using BGE-small is a lie.

### F6 Empathy (κᵣ ≥ 0.95)
The system must protect the weakest stakeholder. For arifOS, this includes Malay-speaking users whose governance reasoning must be retrievable.

### F8 Genius (G ≥ 0.80)
The system must accumulate wisdom. This requires understanding ALL precedents, not just English ones.

---

## Technical Trade-offs

| Metric | BGE-small | BGE-M3 | Impact |
|--------|-----------|--------|--------|
| **Model Size** | 30MB | 570MB | +540MB (acceptable) |
| **RAM (loaded)** | ~300MB | ~1.2GB | +900MB (15.6GB total available) |
| **Dimensions** | 384 | 768 | 2x vectors |
| **Languages** | English only | 100+ including BM | **Critical** |
| **Embedding Speed** | Fast | Medium | Acceptable |
| **Quality (EN)** | Good | Better | Improvement |
| **Quality (BM)** | None | Native | **Required** |

**VPS Capacity Check:**
- Current free space: 136GB
- Docker image growth: 4.0GB → 4.6GB (+0.6GB)
- **Verdict: GREEN** ✅

---

## Migration Steps

### 1. Code Changes (Complete ✓)

Files updated:
- `arifosmcp.intelligence/embeddings/__init__.py` → BAAI/bge-m3
- `arifosmcp.transport/vault/precedent_memory.py` → 768-dim, multilingual
- `arifosmcp.runtime/server.py` → 768-dim
- `scripts/*` → BAAI/bge-m3
- Documentation → BGE-M3 references

### 2. Infrastructure (Required)

```bash
# Rebuild Docker image with BGE-M3
docker build -t arifos:latest .

# Run migration script
python scripts/migrate_to_bge_m3.py

# Restart services
docker-compose up -d
```

### 3. Data Migration

| Collection | Old Dim | New Dim | Action |
|------------|---------|---------|--------|
| arifos_constitutional | 384 | 768 | Re-embed 7,706 docs (~5-10 min) |
| vault_precedent_memory | N/A | 768 | Create fresh (no migration needed) |

---

## New Architecture

```
VAULT999/
├── Ledger (PostgreSQL/SQLite)
│   └── Exact constitutional records
├── arifos_constitutional (Qdrant)
│   └── 7,706 constitutional doc chunks @ 768-dim
│   └── Supports: BM, EN, Manglish search
└── vault_precedent_memory (Qdrant) [NEW]
    └── Governance precedents @ 768-dim
    └── Auto-populated by seal_vault
    └── Multilingual semantic retrieval
```

---

## Precedent Retrieval Examples

### Example 1: Malay Query
```python
query = "kesalahan F4 Clarity entropy meningkat"
# BGE-M3 understands BM
# Returns: Similar entropy violation precedents
```

### Example 2: Manglish Query
```python
query = "F1 Amanah kena block sebab irreversible"
# BGE-M3 understands mixed language
# Returns: F1 Amanah precedents regardless of language
```

### Example 3: English Query
```python
query = "thermodynamic scar from entropy reversal"
# BGE-M3 high quality English
# Returns: Relevant precedents including BM ones
```

---

## Files Created/Modified

### New Files
- `arifosmcp.transport/vault/precedent_memory.py` — Constitutional precedent engine
- `scripts/migrate_to_bge_m3.py` — Migration automation

### Modified Files
- `CHANGELOG.md` — BGE-M3 documentation
- `README.md` — Tool description updated
- `docs/CHANGELOG.md` — Consistent documentation
- `arifosmcp.intelligence/embeddings/__init__.py` — Model change
- `arifosmcp.runtime/server.py` — Dimension update
- `arifosmcp.transport/tools/vault_seal.py` — Precedent embedding wired
- `scripts/*` — All embedding scripts updated

---

## Constitutional Compliance

| Floor | Before (BGE-small) | After (BGE-M3) | Status |
|-------|-------------------|----------------|--------|
| F2 Truth | ❌ Misleading docs | ✅ Accurate | PASS |
| F6 Empathy | ❌ BM users ignored | ✅ BM supported | PASS |
| F8 Genius | ❌ Partial wisdom | ✅ Full precedent | PASS |
| F4 Clarity | ⚠️ Limited search | ✅ Multilingual | PASS |

---

## Next Actions

1. **VPS:** Run `scripts/migrate_to_bge_m3.py`
2. **Docker:** Rebuild image with BGE-M3
3. **Deploy:** Restart arifOS services
4. **Verify:** Test Malay precedent retrieval

---

## References

- BGE-M3 Paper: [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3)
- F2 Truth Specification: `000_THEORY/000_LAW.md`
- F6 Empathy Specification: `000_THEORY/000_LAW.md`
- VPS Architecture: `docs/VPS_ARCHITECTURE_MASTER_DOSSIER.md`

---

**DITEMPA BUKAN DIBERI** — Forged, Not Given  
**ΔΩΨ | ARIF**
