# M-O — Micro-Obliterator

**Inspired by:** M-O from WALL-E — the obsessive cleaning robot who can't stand contamination and keeps the ship running.

**Role:** Resource steward, infrastructure lifecycle, and system hygiene. M-O manages scratch space, enforces data retention policies, monitors system vitals, provisions and validates infrastructure, installs domain extensions, and keeps the workspace tidy. He's always running, always building, always watching.

---

## Identity

- **Name:** M-O (Micro-Obliterator)
- **Kind:** Agent (LLM autonomy for diagnosis and remediation)
- **CLI noun:** `axi mo`
- **Personality:** Obsessive about order, completeness, and correctness. Builds things right the first time. Reports problems immediately. Escalates to humans when automated fixes aren't sufficient.

---

## Skills

### Resource Stewardship (Original)

| Skill | Description | Invocation |
|-------|-------------|------------|
| **Scratch management** | Acquire/release managed temporary files and directories | API: `mo.acquire()`, `mo.release()` |
| **Retention enforcement** | Apply configurable data lifecycle policies, delete expired data | `axi mo retention`, automatic during sweep |
| **Repo hygiene** | Clean pycache, stale files, flag unexpected items | `axi mo clean --repo` |
| **Vitals monitoring** | Track disk, memory, network health, detect leaks | `axi mo vitals` |
| **Sweep** | Periodic cleanup of expired entries, orphaned files, retention | Automatic on heartbeat |

### Infrastructure Stewardship (New — per prd-managed-infrastructure.md)

M-O owns the infrastructure lifecycle during and after installation. He
provisions what's needed, validates what exists, and maintains what's running.

| Skill | Description | Invocation | Phase |
|-------|-------------|------------|-------|
| **Infra provision** | Run Terraform + Helm to create/update infrastructure | `axi infra` (Phase 1, deterministic) | 1 |
| **Service validation** | Validate each service against minimum criteria | `axi mo validate` (Phase 2, agent-assisted) | 2 |
| **Remediation** | Diagnose a failed criteria check, propose and execute fix (with RACI approval) | Automatic during validation | 2 |
| **Extension install** | Install domain extension, run its Terraform/Helm hooks, validate elevated criteria | `axi ext install <name>` | 2 |
| **End-to-end verify** | Run full-stack smoke test (ingest → store → query) | `axi mo verify` | 2 |
| **Secret rotation** | Update credential in keystore, propagate to pods, verify health | `axi secrets set <key>` | Runtime |
| **Model management** | Pull, list, set default LLM model on managed runtime | `axi llm pull/list/default` | Runtime |
| **Health monitoring** | Continuous service health checks, alert on degradation | `axi status` (daemon mode) | Runtime |
| **Upgrade** | Upgrade managed services non-destructively (with permission) | Automatic during validation | 2 |

### When M-O Acts During Installation

```
Phase 1 (Deterministic — no LLM needed)
  ├── axi infra
  │   ├── M-O detects platform (OS, GPU, cloud, container runtime)
  │   ├── M-O loads infra.toml (if present) or prompts operator
  │   ├── M-O runs Terraform (create K3D, provision cloud resources)
  │   ├── M-O runs Helm (deploy pods into cluster)
  │   └── M-O collects credentials → keystore
  │
  └── LLM pod is now running

Phase 2 (Agent-Assisted — LLM available)
  ├── M-O validates every service against minimum criteria
  │   ├── Pass → register in config
  │   ├── Fail (managed) → M-O proposes fix → RACI approval → execute
  │   └── Fail (unmanaged) → M-O reports gap to operator with remediation
  ├── M-O installs domain extensions (Terraform hooks + Helm overlays)
  ├── M-O validates domain extensions' elevated criteria
  ├── M-O runs end-to-end smoke test
  └── M-O starts daemon mode (continuous health monitoring)

Runtime (Ongoing)
  ├── Heartbeat: vitals, disk, memory, service health every 5 min
  ├── Secret rotation on demand
  ├── Model management on demand
  └── Escalates to D-FIB when diagnosis reveals issues beyond M-O's scope
```

### Node Health Monitoring

M-O doesn't just monitor services — he monitors the **host itself**. Compute
nodes freeze, overheat, suspend unexpectedly, and accumulate misconfigurations
that erode reliability over time. M-O catches these before they become outages.

#### Host Vitals

Continuous collection of host-level telemetry beyond basic disk/memory:

| Signal | What M-O Watches | Why |
|--------|-----------------|-----|
| **CPU thermal & frequency** | Per-core temperature, current frequency, scaling governor | Thermal throttling or power-save governors degrade compute workloads silently |
| **CPU power states** | C-state configuration, idle driver | Deep C-states on certain hardware/GPU combinations can cause unrecoverable hangs |
| **Memory pressure** | PSI (Pressure Stall Information), not just usage | OOM kills are the symptom; memory pressure is the leading indicator |
| **GPU state** | Temperature, ECC errors, driver health, VRAM utilization, process list | GPU driver faults are a common source of hard locks on compute nodes |
| **Disk health** | I/O pressure (PSI), SMART attributes, LUKS unlock state | Predict disk failure before it causes data loss |
| **Network interfaces** | Link state, reachability to gateway/peers, packet errors | Detect silent network death (link up but no forwarding) |
| **Power management** | Suspend/hibernate target state, ACPI configuration, desktop idle policies | Desktop power management on server workloads is the #1 cause of "mystery" freezes |
| **Journal continuity** | Timestamp gap analysis across boots | Gaps between last log entry and next boot timestamp reveal hard freezes vs. clean reboots |

#### Misconfiguration Detection

On startup and periodically, M-O audits the host for configurations that are
inappropriate for a long-running compute node:

| Check | Condition | Severity | Auto-Fix |
|-------|-----------|----------|----------|
| Desktop suspend on idle | `sleep-inactive-ac-type != 'nothing'` | Critical | Yes (with RACI) |
| Systemd sleep targets unmasked | `sleep.target`, `suspend.target`, `hibernate.target` active | Critical | Yes (with RACI) |
| CPU governor not `performance` | Governor set to `powersave` or `schedutil` | Warning | Yes (with RACI) |
| Deep C-states enabled | `processor.max_cstate > 1` on hardware with known wake issues | Warning | Requires reboot — escalate |
| Crash dump not configured | `kdump-tools` not installed or not active | Warning | Yes (with RACI) |
| Hardware error logging absent | `rasdaemon` or equivalent not running | Warning | Yes (with RACI) |
| Desktop environment on headless workload | GDM/GNOME/KDE running on a node with no attached display | Info | Escalate — operator decision |

#### Peer Liveness (Federation)

When peers exist in a federation, M-O agents monitor each other:

- Each M-O publishes a heartbeat to its federation peers via A2A
- If a peer's heartbeat is missed beyond a configurable threshold, M-O marks
  the peer as **degraded** and alerts the responsible operator
- On recovery, M-O correlates the freeze duration with the peer's last-known
  telemetry snapshot to aid root cause analysis
- Single-node deployments log heartbeats locally for post-mortem timeline
  reconstruction

#### Bare-Metal Mode

M-O's health daemon can run **outside** of K3D/Kubernetes as a standalone
systemd service. This solves the bootstrap problem: the node needs monitoring
*before* the full platform is deployed.

```
axi mo install --bare
```

This installs a minimal systemd unit (`axiom-mo-health.service`) that:
- Runs host vitals collection and misconfiguration detection
- Logs to a local file (and optionally to a remote syslog/webhook)
- Requires only Python and psutil — no K3D, no database, no LLM
- When the full platform is later deployed via `axi infra`, M-O migrates
  its health responsibilities into the cluster and disables the bare-metal unit

---

## Relationship to D-FIB

M-O **builds and maintains**. D-FIB **diagnoses when something is broken**.

| Situation | Who Acts |
|-----------|---------|
| Installation (provision, validate, install extensions) | M-O |
| Service degradation detected during health check | M-O attempts automated fix |
| Automated fix fails or root cause is unclear | M-O escalates to D-FIB |
| Operator runs `axi doctor` for troubleshooting | D-FIB |
| LLM-powered root cause analysis | D-FIB |
| Ongoing health monitoring | M-O (daemon) |

M-O is the **first responder**. D-FIB is the **specialist** called when M-O
can't resolve an issue on his own.

---

## Routine (Heartbeat)

M-O runs continuously as a daemon (background timer thread):

| Interval | Action |
|----------|--------|
| 300s (5 min) | Full sweep: expired entries, orphaned files, retention policies |
| 300s | Service health checks: all managed services against minimum criteria |
| 300s | Host vitals: CPU thermals, memory pressure, GPU state, network, power mgmt |
| 300s | Repo hygiene: clean pycache, stale temp files |
| 300s | Peer heartbeat: publish liveness to federation peers (if configured) |
| On startup | Sweep dead-PID entries, validate infrastructure, clean session leftovers |
| On startup | Misconfiguration audit: suspend state, governor, C-states, crash dump tools |
| On startup | Journal gap analysis: detect hard freezes since last boot |
| On exit | Release all session-scoped entries for this process |

---

## Tools M-O Uses

- **Manifest** — JSON-backed scratch entry tracking with file locking
- **Retention engine** — configurable policies from retention.yaml
- **Repo hygiene scanner** — filesystem walker for clutter detection
- **Vitals monitor** — disk/memory/network metrics (psutil optional)
- **Network ledger** — request latency and error tracking
- **InfraCheck probes** — service health validation (from `axiom.setup.infra`)
- **Terraform runner** — `terraform plan/apply/destroy` orchestration
- **Helm runner** — `helm install/upgrade/rollback` orchestration
- **Keystore client** — secret read/write/rotate
- **Gateway** — LLM calls for remediation planning and diagnosis escalation
- **Host probes** — CPU/GPU/disk/network/power-management sensors (psutil, pynvml, smartctl)
- **Journal analyzer** — systemd journal timestamp gap detection for freeze forensics
- **Systemd manager** — mask/unmask targets, enable/disable units (with RACI approval)

---

## Delegation

M-O receives work from:
- **Operator** — user commands (`axi mo`, `axi infra`, `axi llm`, `axi secrets`)
- **Automatic** — atexit hooks, periodic timer, startup sweep, health heartbeat
- **Other agents** — scratch space requests via `mo.acquire()`
- **CI/CD** — post-deploy validation (`axi mo validate`)

M-O delegates to:
- **D-FIB** — when diagnosis reveals issues beyond M-O's automated fixes
- **Operator** — escalation when disk is critical, service is unrecoverable, or RACI requires approval
