Metadata-Version: 2.4
Name: arc-training
Version: 4.2.2
Summary: Automatic Recovery Controller - Auto-detect and recover from neural network training failures
Home-page: https://github.com/aryankaushik/arc-training
Author: Aryan Kaushik
Author-email: Aryan Kaushik <a.kaushik0908@gmail.com>
License: AGPL-3.0-or-later
Project-URL: Homepage, https://github.com/a-kaushik2209/ARC
Project-URL: Documentation, https://github.com/a-kaushik2209/ARC
Project-URL: Repository, https://github.com/a-kaushik2209/ARC
Project-URL: Changelog, https://github.com/a-kaushik2209/ARC/blob/master/CHANGELOG.md
Keywords: deep-learning,pytorch,training,fault-tolerance,nan-recovery,checkpointing,machine-learning
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: torch>=1.9.0
Requires-Dist: numpy>=1.19.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.0; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: isort>=5.0; extra == "dev"
Requires-Dist: mypy>=0.900; extra == "dev"
Provides-Extra: lightning
Requires-Dist: pytorch-lightning>=1.5.0; extra == "lightning"
Provides-Extra: full
Requires-Dist: scipy>=1.7.0; extra == "full"
Requires-Dist: tqdm>=4.0.0; extra == "full"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# ARC

### Autonomous Recovery Controller for Neural Network Training

_Real-time fault tolerance that monitors, predicts, and recovers from training failures — automatically._

[![PyPI](https://img.shields.io/badge/PyPI-arc--training-blue?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/arc-training)
[![Python](https://img.shields.io/badge/Python-3.8+-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org)
[![PyTorch](https://img.shields.io/badge/PyTorch-EE4C2C?style=for-the-badge&logo=pytorch&logoColor=white)](https://pytorch.org)
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-green?style=for-the-badge)](https://www.gnu.org/licenses/agpl-3.0)

---

**3 lines of code** · **~18% GPU overhead** · **100% recovery on induced failures** · **20K–355M parameters validated**

[Quick Start](#quick-start) · [Architecture](#architecture) · [Benchmarks](#benchmarks)

---

## The Problem

Training neural networks is fragile. A single NaN gradient, an OOM spike, or an exploding loss at hour 47 of a 48-hour run can destroy days of compute. Engineers waste enormous time adding manual checkpointing, writing recovery scripts, and babysitting long runs.

**ARC eliminates this entirely.** It wraps your training loop with an autonomous controller that:

1. **Monitors** — Tracks multi-signal telemetry (loss trajectory, gradient norms, weight health, optimizer state integrity)
2. **Predicts** — Tracks gradient norm trends and detects exponential growth before failures become irreversible
3. **Recovers** — Automatically rolls back to the last healthy checkpoint and applies corrective measures (LR reduction, weight perturbation)

You keep training. ARC keeps it alive.

---

## Quick Start

### Installation

```bash
pip install arc-training
```

Or install from source:

```bash
git clone https://github.com/a-kaushik2209/ARC.git
cd ARC
pip install -e .
```

### 3-Line Integration

```python
from arc import Arc

controller = Arc(model, optimizer)

for batch in dataloader:
    loss = model(batch)
    action = controller.step(loss)       # monitor + protect

    if not action.rolled_back:           # normal path
        loss.backward()
        optimizer.step()
```

That's it. ARC handles NaN detection, gradient explosion recovery, checkpoint management, and learning rate adjustment — all behind `controller.step()`.

---

## Architecture

ARC is a modular multi-signal monitoring system:

```
arc/
├── core/            Self-healing engine with rollback + LR reduction
├── signals/         Multi-signal collectors (gradient, loss, weight, optimizer state)
├── features/        Feature extraction, normalization, and buffering
├── prediction/      Signal-based failure prediction (logistic regression + MLP)
├── intervention/    Recovery strategies (LR reduction, gradient clipping, weight perturbation)
├── checkpointing/   Checkpoint management with circular buffer
├── introspection/   Fisher Information, Hessian approximation, loss landscape analysis
├── physics/         Lyapunov stability analysis, FFT oscillation detection
├── uncertainty/     Conformal prediction for calibrated stability assessment
└── evaluation/      Benchmarking and validation harness
```

### Signal Pipeline

```
Training Step
     │
     ▼
┌─────────────────────────────────────────────────────────────┐
│  Signal Collectors                                          │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐  │
│  │ Gradient  │ │ Loss     │ │ Weight   │ │ Optimizer     │  │
│  │ Norm/Ent. │ │ Trend/Var│ │ Norm/NaN │ │ State Norm    │  │
│  └─────┬────┘ └─────┬────┘ └─────┬────┘ └──────┬────────┘  │
│        └──────┬──────┴──────┬─────┘             │           │
│               ▼             ▼                   ▼           │
│         Feature Extractor (12 features)                     │
│               │                                             │
│               ▼                                             │
│    ┌─────────────────────┐    ┌──────────────────────────┐  │
│    │  Heuristic Detector │    │  MLP Predictor           │  │
│    │  (instant response) │    │  (97.5% acc, 0 FP)       │  │
│    └─────────┬───────────┘    └────────────┬─────────────┘  │
│              └──────────┬─────────────────┘                 │
│                         ▼                                   │
│              Risk Assessment + Recovery Decision            │
└─────────────────────────┬───────────────────────────────────┘
                          │
              ┌───────────┴───────────┐
              │   HEALTHY             │──── Continue training
              │   WARNING             │──── Increase monitoring, prepare checkpoint
              │   FAILURE             │──── Rollback to checkpoint + corrective action
              └───────────────────────┘
```

---

## Failure Coverage

| Category    | Failure Type          | Detection | Recovery                     |
| ----------- | --------------------- | --------- | ---------------------------- |
| **Numeric** | NaN / Inf Loss        | Instant   | Rollback + LR reduction      |
| **Numeric** | Loss Explosion        | Instant   | Rollback + LR reduction      |
| **Numeric** | Gradient Explosion    | Instant   | Rollback + gradient clipping |
| **Numeric** | Weight Corruption     | Instant   | Rollback from checkpoint     |
| **Silent**  | Optimizer State Reset | Detected  | Rollback + state restoration |
| **Silent**  | Silent Weight Drift   | Detected  | Alert + optional rollback    |
| **Silent**  | LR Spike              | Instant   | Rollback + LR correction     |

---

## Benchmarks

> **All numbers below are from reproducible experiment scripts with fixed seeds.**

### Baseline Comparison (25 scenarios)

4 methods × 5 failure types × 5 seeds. Script: `experiments/baseline_comparison.py`

| Method            | Detection | Recovery | False Positives |
| :---------------- | :-------: | :------: | :-------------: |
| No Protection     |   52.0%   |   0.0%   |        0        |
| Gradient Clipping |   20.0%   |   0.0%   |        0        |
| Loss-Only Monitor |   80.0%   |  80.0%   |        0        |
| **Full ARC**      | **100%**  | **100%** |      **0**      |

### Failure Prediction (200 scenarios)

4 architectures × 5 failure types × 5 seeds × 2 labels, 5-fold CV. Script: `experiments/prediction_200_v2.py`

| Classifier         |     Accuracy     | Precision |  Recall   |        F1        |
| :----------------- | :--------------: | :-------: | :-------: | :--------------: |
| Logistic Reg (12f) |   95.5% ± 1.9%   |   100%    |   91.0%   |   0.953 ± 2.6%   |
| **MLP (12f)**      | **97.5% ± 2.2%** | **100%**  | **95.0%** | **0.974 ± 2.8%** |

### Ablation Study (35 scenarios)

7 failure types × 5 seeds. Script: `experiments/ablation_experiment.py`

| Configuration             | Detection | Δ from Full |
| :------------------------ | :-------: | :---------: |
| Full ARC (all components) |   85.7%   |     ---     |
| − Weight Health           |   85.7%   |    0.0%     |
| − Gradient Monitoring     |   85.7%   |    0.0%     |
| − Loss Monitoring         |   85.7%   |    0.0%     |
| − Optimizer State         |   71.4%   |   −14.3%    |
| Loss Only (baseline)      |   71.4%   |   −14.3%    |

> **Defense in depth**: Weight/gradient/loss provide redundant coverage (any one catches most failures). Optimizer state monitoring is uniquely valuable for silent failures.

### Overhead (measured, CPU)

Script: `experiments/overhead_measurement.py`

| Component           | Time (ms) | % of ARC Total |
| :------------------ | :-------: | :------------: |
| Gradient Norm       |   0.12    |      9.0%      |
| Weight Statistics   |   1.06    |     76.9%      |
| Loss Analysis       |   0.01    |      0.6%      |
| Checkpoint (amort.) |   0.13    |      9.6%      |
| Forecasting         |   0.06    |      4.1%      |
| **Total ARC**       | **1.38**  |    **100%**    |

| Model Scale | Parameters | ARC Overhead | Relative |
| :---------- | :--------: | :----------: | :------: |
| Small MLP   |    50K     |   0.86 ms    |   ~60%   |
| Medium CNN  |    288K    |   1.38 ms    |   ~10%   |
| Large CNN   |    2.5M    |   7.04 ms    |  ~9.5%   |

### Large Model Stress Test

Script: `experiments/validate_claims_phase2.py`

| Model        | Params | Failure Type     | ARC Recovery | Rollbacks |
| :----------- | :----- | :--------------- | :----------: | :-------: |
| NanoGPT      | 10M    | LR Spike (50×)   |      ✓       |     2     |
| ResNet-50    | 25.6M  | Loss Singularity |      ✓       |     1     |
| GPT-2 Small  | 50M    | NaN Bomb         |      ✓       |     4     |
| SD-UNet      | 60M    | Gradient Attack  |      ✓       |     4     |
| ViT-Base     | 86M    | Inf Nuke         |      ✓       |     1     |
| GPT-2 Medium | 117M   | NaN Bomb         |      ✓       |     3     |

---

## Theoretical Foundation

ARC integrates six mathematical frameworks, each experimentally validated:

| Framework                        | Purpose                                             | Validation                                               |
| :------------------------------- | :-------------------------------------------------- | :------------------------------------------------------- |
| **Fisher Information**           | Parameter importance weighting for recovery         | 11.5× separation ratio (important vs unimportant params) |
| **Lyapunov Stability**           | Online stability estimation from parameter velocity | 10× higher exponent under instability                    |
| **FFT Oscillation Detection**    | Periodic behaviour detection in training dynamics   | 6.9× power ratio at oscillation frequency                |
| **Conformal Prediction**         | Distribution-free coverage guarantees for stability | ≥99% empirical coverage at all target levels             |
| **Elastic Weight Consolidation** | Knowledge preservation during recovery              | 0.4% lower post-recovery loss                            |
| **Loss Landscape Analysis**      | Sharpness-based instability prediction              | 12.2× higher sharpness before failure                    |

---

## Known Limitations

ARC is honest about what it cannot do:

- **fp16 models**: Models loaded in fp16 require proper mixed precision setup (autocast + GradScaler). ARC monitors training — it does not manage dtype conversion
- **Scale ceiling**: Validated up to 355M parameters (GPT-2 Medium). Behaviour above this is not yet confirmed
- **First checkpoint**: No checkpoint exists before the first save — very early failures are unrecoverable
- **Data problems**: ARC cannot detect data corruption, label noise, or adversarial poisoning
- **Distributed training**: Multi-GPU (DDP/FSDP) is not yet supported
- **Non-PyTorch**: Only PyTorch is supported

---

## Citation

```bibtex
@article{kaushik2026arc,
  title   = {ARC: Autonomous Recovery Controller for Fault-Tolerant Neural Network Training},
  author  = {Kaushik, Aryan},
  year    = {2026},
  note    = {Maharaja Agrasen Institute of Technology, New Delhi}
}
```

---

**AGPL-3.0 License** · Copyright (c) 2026 Aryan Kaushik

_Built to make neural network training unkillable._

