# Data Architecture Specification

**Part of:** [Axiom Master Tech Spec](spec-executive.md)

---

> **Scope:** This document specifies the Axiom data architecture, including the medallion pattern, Apache Iceberg configuration, schema definitions, data quality framework, streaming readiness, and operational policies (backup, retention, archive, DR). Domain-specific schemas (e.g., nuclear reactor data) are defined by downstream products.

| Property | Value |
|----------|-------|
| Version | 0.3 |
| Last Updated | 2026-03-26 |
| Status | Draft |

---

## Table of Contents

1. [Overview](#1-overview)
2. [Medallion Architecture](#2-medallion-architecture)
3. [Layer Specifications](#3-layer-specifications)
4. [Gold Layer Schemas](#4-gold-layer-schemas)
5. [Data Quality Framework](#5-data-quality-framework)
6. [Apache Iceberg Configuration](#6-apache-iceberg-configuration)
7. [Platform Comparison](#7-platform-comparison)
8. [Streaming Architecture](#8-streaming-architecture)
9. [Backup, Retention & Archive Policy](#9-backup-retention--archive-policy)

---

## 1. Overview

Axiom employs a medallion architecture (Bronze → Silver → Gold) built on:

- **Apache Iceberg** for time-travel capabilities and schema evolution
- **DuckDB** as the query engine
- **dbt** for transformations
- **Dagster** for orchestration

---

## 2. Medallion Architecture

### Layer Characteristics

| Layer | Purpose | Mutability | Typical Format |
|-------|---------|------------|----------------|
| **Bronze** | Raw ingestion | Append-only | Parquet (Iceberg) |
| **Silver** | Cleaned, validated | Upsert | Parquet (Iceberg) |
| **Gold** | Aggregated, business-ready | Materialized views | Parquet (Iceberg) |

---

## 3. Layer Specifications

### 3.1 Bronze Layer

Raw, unprocessed data exactly as received. Append-only to preserve complete history.

| Table | Source | Grain | Partitioning |
|-------|--------|-------|--------------|
| `interaction_log_raw` | RAG completions | Per completion | `date`, `facility_id` |
| `agent_state_raw` | Agent state transitions | Per transition | `date`, `facility_id` |

> **Note:** Domain-specific Bronze tables (e.g., sensor readings, operations logs, simulation outputs) are defined by downstream products. See [Data Platform PRD](../requirements/prd-data-platform.md) for the generic framework; axiom defines nuclear-specific Bronze schemas in its own Data Platform PRD.

**Multi-Tenant:** All tables partitioned by `org_id` and `system_id` for tenant isolation.

### 3.2 Silver Layer

Cleaned, validated, and deduplicated data. dbt transformations apply business rules.

| Table | Source | Transformations |
|-------|--------|-----------------|
| `interaction_log` | Bronze | Dedup, session linking, schema enforcement |
| `agent_state_transitions` | Bronze | FK validation, schema enforcement |

> **Note:** Domain-specific Silver tables are defined by downstream products.

### 3.3 Gold Layer

Business-ready, aggregated datasets optimized for analytics and dashboards.

| Table | Grain | Use Case |
|-------|-------|----------|
| `system_hourly_metrics` | Hour | Dashboard KPIs |
| `operational_kpis` | Day | Management reporting |
| `compliance_summary` | Day | Regulatory reporting |
| `interaction_analytics` | Day | RAG usage metrics |

> **Note:** Domain-specific Gold tables (e.g., fuel burnup, xenon dynamics for nuclear facilities) are defined by downstream product specs, not here.

---

## 4. Gold Layer Schemas

### 4.1 system_hourly_metrics

| Column | Type | Description |
|--------|------|-------------|
| `system_id` | string | System identifier |
| `hour` | timestamp | Hour bucket |
| `metrics` | jsonb | Domain-specific metric payload |
| `source` | enum | `measured` \| `modeled` |

> **Note:** The `metrics` JSONB column contains domain-specific fields. Downstream products define the schema (e.g., axiom defines `avg_power_kw`, `max_fuel_temp_c`).

### 4.2 log_entries (Unified Log)

Single table with `entry_type` discriminator for all log types.

| Column | Type | Description |
|--------|------|-------------|
| `entry_id` | uuid | Primary key |
| `system_id` | string | System identifier |
| `timestamp` | timestamp | Entry time |
| `entry_type` | string | Extensible type discriminator |
| `operator_id` | string | User who created |
| `content` | jsonb | Type-specific payload |

**Built-in Entry Types:**

| Type | Description |
|------|-------------|
| `status_check` | Periodic system status check |
| `startup` | System startup |
| `shutdown` | Normal shutdown |
| `emergency_stop` | Emergency shutdown |
| `maintenance` | Equipment issues |
| `general_note` | Miscellaneous |

> Domain-specific entry types (e.g., `radiation_survey`, `experiment_log`, `console_check` for nuclear) are registered by downstream products via extension configuration.

### 4.3 Model Corral Tables

**Reference:** [Model Corral Spec](spec-model-corral.md)

| Layer | Table | Description |
|-------|-------|-------------|
| Bronze | `model_registry_raw` | Raw model.yaml JSON as submitted |
| Silver | `model_registry_validated` | Schema-validated, enriched with lineage |
| Gold | `model_catalog` | Searchable model catalog |
| Gold | `model_lineage` | Parent-child relationships (derived → source) |
| Gold | `model_validation_metrics` | Aggregated validation statistics |

**Key relationships:**
- `model_lineage.child_model_id` → `model_catalog.model_id`
- `model_lineage.parent_model_id` → `model_catalog.model_id`
- Derived models (e.g., surrogates, reduced-order) have `training.source_model` referencing the source model they were trained from

### 4.4 Domain-Specific Tables

Domain-specific Bronze/Silver/Gold tables (e.g., digital twin run tracking, simulation outputs, domain sensor data) are defined by downstream products, not in this spec. Axiom provides the Iceberg/dbt/Dagster framework; downstream products register their schemas via the extension system.

See [Data Platform PRD § Domain Extension Points](../requirements/prd-data-platform.md#domain-extension-points) for the extension model.

---

## 5. Data Quality Framework

### Quality Tests

| Test Type | Layer | Example |
|-----------|-------|---------|
| **Not null** | Bronze | `sensor_id IS NOT NULL` |
| **Unique** | Silver | `entry_id` unique per table |
| **Referential** | Silver | `system_id` exists in `systems` |
| **Range** | Silver | Domain-specific value bounds |
| **Freshness** | Gold | Data < 1 hour old |
| **Custom** | Gold | Domain-specific consistency checks |

---

## 6. Apache Iceberg Configuration

### 6.1 Catalog Configuration

| Setting | Value | Rationale |
|---------|-------|-----------|
| Catalog type | REST | Standard API for multi-engine access |
| Metadata location | S3 (Ceph/Rook) | Durable, shared storage |
| Warehouse | `s3://axiom-lakehouse/` | All Iceberg data |

### 6.2 Partitioning Strategy

| Table Pattern | Partition Columns | Rationale |
|---------------|-------------------|-----------|
| Sensor data | `date`, `system_id` | Query by time + tenant |
| Logs | `date`, `entry_type` | Query by time + type |
| Simulations | `date`, `model_type` | Query by time + model |

### 6.3 Key Capabilities

- **Time-travel queries:** Query data as it existed at any point
- **Schema evolution:** Add/rename/drop columns without rewriting
- **Partition evolution:** Change partitioning without data movement
- **ACID transactions:** Concurrent reads and writes

---

## 7. Platform Comparison

### 7.1 Decision Summary

We chose open-source (Iceberg + DuckDB + dbt) over commercial platforms (Databricks, Snowflake).

### 7.2 Decision Rationale

| Factor | Open Lakehouse | Commercial Platform |
|--------|----------------|---------------------|
| **Cost** | Fixed infrastructure | Per-compute pricing |
| **Data sovereignty** | Full control | Vendor access |
| **Domain integration** | Native Python/HDF5 | Limited |
| **On-premise** | Supported | Cloud-primary |
| **Research integrity** | Open pipelines | Proprietary |
| **Workforce dev** | Industry-standard tools | Vendor-specific |

**Migration path:** Open formats (Iceberg, Parquet) ensure future migration feasibility.

### 7.3 INL DeepLynx Partnership Opportunity

**Status:** Exploratory — non-committal technical alignment

Idaho National Laboratory's [DeepLynx Nexus](https://github.com/idaholab/DeepLynx) is an open-source (MIT) digital engineering backbone developed for domain-specific projects. After codebase analysis, we've identified significant technology overlap and complementary capabilities.

**Technology Overlap:**
- Both use **DuckDB** for timeseries analytics
- Both target **domain-specific digital twins**
- Both provide **MCP servers** for AI agent integration

**Complementary Strengths:**

| Capability | DeepLynx | Axiom |
|------------|----------|-------|
| Ontology management | ✅ Mature (Class/Relationship model) | ⚠️ YAML schemas |
| Graph traversal | ✅ Native | ⚠️ Via JOINs |
| Real-time streaming | ⚠️ Batch webhooks | ✅ Kafka |
| Time-series analytics | ✅ DuckDB | ✅ DuckDB + Iceberg |
| ML/ROM workflows | ⚠️ Not focus | ✅ Native |
| AI agent tooling | ⚠️ Basic MCP | ✅ Full MCP |

**Potential Integration Approaches:**

1. **Data Exchange** (Low commitment): CSV/Parquet interchange for timeseries data
2. **MCP Interoperability** (Medium commitment): AI agents access both systems via unified tool spec
3. **Ontology Alignment** (Medium commitment): Share system ontology vocabulary
4. **Plugin Architecture** (High commitment): DeepLynx as optional ConfigurationPlugin for Axiom

**Reference:** Full technical analysis in [docs/research/deeplynx-assessment.md](../research/deeplynx-assessment.md)

---

## 8. Streaming Architecture

> **See:** [ADR-007: Streaming-First Architecture](../requirements/adr-007-streaming-first-architecture.md)

### 8.1 Design Principle

Build for streaming; use batch as fallback.

### 8.2 Event Schema

All events follow a common envelope:

```json
{
  "event_id": "uuid",
  "event_type": "sensor_reading | prediction | log_entry",
  "timestamp": "ISO8601",
  "source": { "facility": "facility-id", "system": "system-id" },
  "payload": { ... },
  "metadata": { "schema_version": "1.0" }
}
```

### 8.3 Latency Targets

| Data Type | Target Latency | Streaming Tech |
|-----------|----------------|----------------|
| Sensor readings | <1s | Kafka |
| Log entries | <5s | Kafka |
| Predictions | <100ms | Direct API |
| Aggregations | <1 min | Flink/Materialize |

---

## 9. Backup, Retention & Archive Policy

All data in the lakehouse must support configurable retention requirements and disaster recovery. This is the single canonical definition of operational policies for the Axiom data stack. Downstream products extend these policies for domain-specific regulatory requirements.

### 9.1 Retention Tiers

| Tier | Retention | Data Types | Storage |
|------|-----------|------------|---------|
| **Hot** | 90 days | Live sensor data, recent logs | Primary object storage (Ceph/Rook) |
| **Warm** | 2 years | Historical operations, default regulatory window | Primary object storage |
| **Cold** | 7 years | Audit trails, regulatory records (opt-in) | Glacier-tier / archive storage |
| **Archive** | Indefinite | Safety basis, licensing documents (opt-in) | Glacier-tier / offline |

**Configuration:** Retention tier is set per deployment via `[retention] policy` in `data-platform.toml`:
- `"standard"` (default): Hot + Warm tiers only (2-year max)
- `"regulatory"`: All four tiers active (7-year Cold + indefinite Archive)

Downstream products may define additional policy values for domain-specific requirements.

### 9.2 Backup Strategy

| Component | Frequency | Destination | Retention |
|-----------|-----------|-------------|-----------|
| PostgreSQL | Daily | S3 (Ceph/Rook) + offsite | 90 days |
| Iceberg metadata | Continuous | S3 | With data |
| Iceberg data | Via Iceberg | S3 | Per tier |
| Configuration | On change | Git + S3 | Indefinite |
| System logs | Daily | S3 + Glacier (if regulatory) | Per tier |
| Audit logs | Daily | S3 + Glacier (if regulatory) | Per tier |

### 9.3 Disaster Recovery

| Scenario | RTO | RPO | Recovery Method |
|----------|-----|-----|-----------------|
| Single node failure | <1 hour | 0 | K8s pod restart |
| Database corruption | <4 hours | <1 hour | Restore from backup |
| Full site loss | <24 hours | <1 day | Offsite restore |

### 9.4 Regulatory Compliance Extensions

The base Axiom policy provides the retention framework. Downstream products activate regulatory tiers as needed:

- **2-year warm tier:** Default for all deployments. Sufficient for most operational needs.
- **7-year cold tier:** Activated by `policy = "regulatory"`. Required for facilities subject to regulatory audit trails.
- **Indefinite archive:** Activated by `policy = "regulatory"`. For safety basis documents and licensing records.
- **Immutability:** All backups are append-only; no modification or deletion allowed.
- **Versioning:** Iceberg time-travel enables recovery of any point-in-time data within the retention window.
- **Audit trail:** All backup operations logged with tamper-evident verification (Hyperledger or HMAC chain, per deployment configuration).

### 9.5 Encryption

| Layer | Encryption | Key Management |
|-------|------------|----------------|
| At rest | AES-256 | OpenBao (production) / OS Keychain (dev) |
| In transit | TLS 1.3 | Auto-renewed certificates (cert-manager) |
| Backups | AES-256 | Separate backup keys |

**Key Rotation:** Quarterly for active encryption keys; archived keys retained for the lifetime of the data they protect.

---

## Related Documents

- [Executive Spec](spec-executive.md) — Master tech spec (§ Infrastructure Services for service inventory)
- [Model Corral Spec](spec-model-corral.md) — Model registry
- [Digital Twin Hosting Spec](spec-digital-twin-architecture.md) — DT execution
- [ADR-003: Lakehouse Architecture](../requirements/adr-003-lakehouse-iceberg-duckdb-superset.md) — Decision record
- [ADR-007: Streaming-First](../requirements/adr-007-streaming-first-architecture.md) — Decision record
