# Observability

## Overview

Implementation-level rules for building observable systems across all AI-DLC
phases. Each rule is a blocking constraint: a phase or stage cannot complete
until all applicable checks pass or are marked N/A.

### Relationship to EGS

EGS defines observability policy (§5.1: metrics/logs/traces required; §5.2:
health checks; §4.3: structured logging format). The security extension covers
security-specific logging (OPS-01: structured logging, OPS-02: access logging,
OPS-03: alerting and log integrity). This extension covers distributed tracing
propagation, SLI/SLO methodology, metric taxonomy, correlation ID design, and
dashboard patterns. It does not redefine log format or alerting thresholds.

### Blocking Behavior

When a check fails:
1. List the finding in the phase/stage completion message under "Observability
   Findings" with the rule ID and failed check.
2. Do NOT present "Continue to Next Stage" until resolved.
3. Present only "Request Changes" with what needs to change.
4. Log the finding in `aidlc-docs/audit/audit-log.md` as `EXTENSION_FINDING`.

If a rule does not apply (e.g., single-service project for OBS-03), mark it **N/A**.

---

## Applicability Question

Asked during pre-flight when this extension is loaded:

```
Should observability rules be enforced for this project?

A) Yes — enforce as blocking constraints (recommended for distributed systems and production services)
B) No — skip all observability rules (suitable for single-service PoCs)
```

Record the answer in `aidlc-docs/aidlc-state.md` under Extension Configuration.

---

## Design-Time Rules

Enforced during Mob Elaboration and the design stages of Mob Construction.

### OBS-01: SLI/SLO Definition

Every user-facing service has measurable service level indicators.

- [ ] SLIs defined per service: availability (success rate), latency (p99), throughput
- [ ] SLOs set as targets for each SLI (e.g., 99.9% success, p99 < 500ms)
- [ ] Error budget concept documented: what happens when SLO is breached
- [ ] SLIs derived from user-facing behavior, not infrastructure metrics
- [ ] *Defers to EGS §9.1 for latency targets and §8.1 for availability targets*

### OBS-02: Metric Taxonomy

Metrics follow a consistent naming and tagging scheme.

- [ ] Metric naming convention defined: `{service}.{component}.{metric}` (e.g., `orders.api.latency_ms`)
- [ ] Standard dimensions/tags: `service`, `environment`, `operation`, `status_code`
- [ ] Business metrics distinguished from technical metrics (e.g., `orders.created` vs `api.latency`)
- [ ] Cardinality limits documented: no unbounded tag values (no user IDs or request IDs as tags)

### OBS-03: Distributed Tracing Design

Cross-service requests are traceable end-to-end.

- [ ] Trace context propagation mechanism chosen (W3C Trace Context preferred, or platform-native trace header)
- [ ] All service boundaries (HTTP, queue, event) propagate trace context
- [ ] Trace sampling strategy defined (100% for errors, percentage for success)
- [ ] *Defers to EGS §5.1 for observability stack selection*

---

## Build-Time Rules

Enforced during Mob Construction implementation stages and Code Elevation.

### OBS-04: Correlation ID Propagation

Every request is traceable through all processing stages.

- [ ] Correlation ID generated at the entry point (API gateway, queue consumer, event handler)
- [ ] Correlation ID included in all log entries for the request
- [ ] Correlation ID propagated to downstream service calls (header, message attribute)
- [ ] Correlation ID returned to the caller in response headers
- [ ] *Defers to EGS §4.3 for mandatory log fields and Security OPS-01 for log structure*

### OBS-05: Structured Metric Emission

Code emits metrics at the right granularity.

- [ ] Request count, latency, and error count emitted per endpoint/operation
- [ ] Business events emitted as custom metrics (e.g., order placed, payment failed)
- [ ] Queue depth and processing latency emitted for async consumers
- [ ] No metric emission inside tight loops (aggregate, then emit)
- [ ] Metric library initialized once, not per-request

### OBS-06: Health Check Implementation

Health endpoints distinguish shallow from deep health.

- [ ] `/health` (shallow): returns 200 if the process is running (for load balancer)
- [ ] `/health/ready` (deep): validates downstream dependencies (database, cache, queues)
- [ ] Deep health check has a timeout (e.g., 3s) to prevent cascading slowness
- [ ] Health check responses include component status breakdown
- [ ] *Defers to EGS §5.2 for health check policy*

### OBS-07: Log Context Enrichment

Logs carry enough context to diagnose issues without reproducing them.

- [ ] Every log entry includes: correlation ID, operation name, duration (for completed operations)
- [ ] Error logs include: error type, error message, relevant entity IDs (not PII)
- [ ] Entry/exit logging at service boundaries (incoming request, outgoing call)
- [ ] No logging of request/response bodies in production (use DEBUG level, off by default)
- [ ] *Defers to EGS §4.3 for log format and Security OPS-01 for PII exclusion*

---

## Operational Readiness Rules

Flagged during Mob Elaboration, verified in implementation.

### OBS-08: Alerting Strategy

Alerts are actionable and tiered by severity.

- [ ] Alerts defined for SLO breach (error budget burn rate)
- [ ] Alert severity tiers: critical (page), warning (ticket), info (dashboard)
- [ ] Every alert has a linked runbook or investigation guide
- [ ] No alerts on raw thresholds without sustained duration (avoid flapping)
- [ ] *Defers to Security OPS-03 for security-specific alerting*

### OBS-09: Dashboard Readiness

Operational dashboards exist before production launch.

- [ ] Service dashboard includes: request rate, error rate, latency (p50/p95/p99), saturation
- [ ] Business dashboard includes: key business metrics from OBS-05
- [ ] Dependency dashboard includes: downstream latency and error rates
- [ ] Dashboard definitions stored as code (IaC templates, Terraform, Pulumi, or dashboard-as-code definitions)

---

## Enforcement by Ritual

| Ritual | Primary Rules | Secondary Rules |
|--------|--------------|-----------------|
| Mob Elaboration | OBS-01 through OBS-03 | OBS-08, OBS-09 (flag as requirements) |
| Mob Construction: Design stages | OBS-01 through OBS-03 | OBS-04 through OBS-07 (verify design supports them) |
| Mob Construction: Implementation | OBS-04 through OBS-07 | OBS-08, OBS-09 (verify operational readiness) |
| Code Elevation | All rules | Assess existing system against full checklist |

At each phase/stage completion, include an "Observability Compliance" section:
- List each applicable rule as ✅ compliant, ❌ non-compliant, or N/A
- Any ❌ is a blocking finding
