Metadata-Version: 2.4
Name: adaptron
Version: 0.3.0
Summary: End-to-end LLM Fine-tuning Framework
Project-URL: Homepage, https://dhsitsolutions.com
Project-URL: Repository, https://github.com/GIS-DHSIT/Adaptron
Project-URL: Bug Tracker, https://github.com/GIS-DHSIT/Adaptron/issues
Project-URL: Developer, https://github.com/msg-dhsit
Author: Muthu Subramanian G
Maintainer: DHS IT Solutions
License-Expression: MIT
License-File: LICENSE
Keywords: fine-tuning,llm,machine-learning,nlp,training,transformers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: accelerate>=0.28; extra == 'all'
Requires-Dist: apscheduler>=3.10; extra == 'all'
Requires-Dist: bitsandbytes>=0.43; extra == 'all'
Requires-Dist: boto3>=1.34; extra == 'all'
Requires-Dist: cassandra-driver>=3.29; extra == 'all'
Requires-Dist: chromadb>=0.5; extra == 'all'
Requires-Dist: confluent-kafka>=2.3; extra == 'all'
Requires-Dist: datasets>=2.18; extra == 'all'
Requires-Dist: elasticsearch>=8.12; extra == 'all'
Requires-Dist: fastapi>=0.110; extra == 'all'
Requires-Dist: google-cloud-bigquery>=3.17; extra == 'all'
Requires-Dist: httpx>=0.27; extra == 'all'
Requires-Dist: huggingface-hub>=0.22; extra == 'all'
Requires-Dist: langchain-community>=0.2; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: openpyxl>=3.1; extra == 'all'
Requires-Dist: pandas>=2.2; extra == 'all'
Requires-Dist: peft>=0.10; extra == 'all'
Requires-Dist: pymongo>=4.6; extra == 'all'
Requires-Dist: pypdf>=4.0; extra == 'all'
Requires-Dist: python-docx>=1.1; extra == 'all'
Requires-Dist: redis>=5.0; extra == 'all'
Requires-Dist: sentence-transformers>=2.6; extra == 'all'
Requires-Dist: snowflake-connector-python>=3.7; extra == 'all'
Requires-Dist: spacy>=3.7; extra == 'all'
Requires-Dist: sqlalchemy>=2.0; extra == 'all'
Requires-Dist: tiktoken>=0.5.0; extra == 'all'
Requires-Dist: torch>=2.1; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Requires-Dist: trl>=0.8; extra == 'all'
Requires-Dist: uvicorn>=0.29; extra == 'all'
Requires-Dist: websockets>=12.0; extra == 'all'
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == 'api'
Requires-Dist: uvicorn>=0.29; extra == 'api'
Requires-Dist: websockets>=12.0; extra == 'api'
Provides-Extra: connectors
Requires-Dist: apscheduler>=3.10; extra == 'connectors'
Requires-Dist: boto3>=1.34; extra == 'connectors'
Requires-Dist: cassandra-driver>=3.29; extra == 'connectors'
Requires-Dist: confluent-kafka>=2.3; extra == 'connectors'
Requires-Dist: elasticsearch>=8.12; extra == 'connectors'
Requires-Dist: google-cloud-bigquery>=3.17; extra == 'connectors'
Requires-Dist: httpx>=0.27; extra == 'connectors'
Requires-Dist: pymongo>=4.6; extra == 'connectors'
Requires-Dist: redis>=5.0; extra == 'connectors'
Requires-Dist: snowflake-connector-python>=3.7; extra == 'connectors'
Provides-Extra: deploy
Requires-Dist: huggingface-hub>=0.22; extra == 'deploy'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.3; extra == 'dev'
Provides-Extra: ingest
Requires-Dist: langchain-community>=0.2; extra == 'ingest'
Requires-Dist: openpyxl>=3.1; extra == 'ingest'
Requires-Dist: pandas>=2.2; extra == 'ingest'
Requires-Dist: pypdf>=4.0; extra == 'ingest'
Requires-Dist: python-docx>=1.1; extra == 'ingest'
Requires-Dist: sqlalchemy>=2.0; extra == 'ingest'
Provides-Extra: pretrain
Requires-Dist: numpy>=1.24.0; extra == 'pretrain'
Requires-Dist: tiktoken>=0.5.0; extra == 'pretrain'
Requires-Dist: torch>=2.1; extra == 'pretrain'
Provides-Extra: rag
Requires-Dist: chromadb>=0.5; extra == 'rag'
Requires-Dist: sentence-transformers>=2.6; extra == 'rag'
Provides-Extra: train
Requires-Dist: accelerate>=0.28; extra == 'train'
Requires-Dist: bitsandbytes>=0.43; extra == 'train'
Requires-Dist: datasets>=2.18; extra == 'train'
Requires-Dist: peft>=0.10; extra == 'train'
Requires-Dist: torch>=2.1; extra == 'train'
Requires-Dist: transformers>=4.40; extra == 'train'
Requires-Dist: trl>=0.8; extra == 'train'
Provides-Extra: understand
Requires-Dist: sentence-transformers>=2.6; extra == 'understand'
Requires-Dist: spacy>=3.7; extra == 'understand'
Description-Content-Type: text/markdown

# Adaptron

**End-to-end LLM Fine-tuning Framework**

<!-- Badges placeholder -->
<!-- ![PyPI](https://img.shields.io/pypi/v/adaptron) -->
<!-- ![Python](https://img.shields.io/pypi/pyversions/adaptron) -->
<!-- ![License](https://img.shields.io/github/license/GIS-DHSIT/adaptron) -->

---

## Overview

Adaptron is a plugin-based framework that takes you from raw documents to a deployed, fine-tuned language model. It orchestrates six pipeline stages -- **Ingest, Understand, Synthesize, Train, Evaluate, Deploy** -- and exposes the entire workflow through a Python API, a CLI, a FastAPI backend, and a Next.js web UI.

## Features

- **Multi-format ingestion** -- PDF, DOCX, CSV, and SQL data sources
- **Semantic understanding** -- chunking, entity extraction, quality scoring, schema inference
- **Instruction synthesis** -- template-based training data generation
- **Multiple training strategies** -- QLoRA (Unsloth+PEFT), Full Fine-Tuning, Continual Pre-Training, Distillation, DPO alignment
- **Domain evaluation** -- automated scoring of fine-tuned models
- **One-click deployment** -- Ollama, GGUF export, HuggingFace Hub push
- **Training Strategy Wizard** -- answers seven questions, picks the best training mode and base model automatically
- **Playground** -- chat with deployed models, compare outputs side-by-side, toggle RAG augmentation
- **Plugin system** -- register custom ingesters, trainers, deployers, or any stage component
- **Event-driven pipeline** -- real-time progress via EventBus and WebSocket

## Quick Start

```bash
# Install with all optional dependencies
pip install -e ".[all]"

# Initialize a project
adaptron init --project-dir my-project
cd my-project

# Edit adaptron.yaml, then run the pipeline
adaptron run
```

## Installation

Requires **Python 3.11+**.

```bash
# Core only (config, CLI, pipeline orchestrator)
pip install -e .

# With specific extras
pip install -e ".[train]"       # torch, transformers, PEFT, TRL, bitsandbytes
pip install -e ".[ingest]"      # pypdf, python-docx, pandas, sqlalchemy
pip install -e ".[understand]"  # spacy, sentence-transformers
pip install -e ".[rag]"         # chromadb, sentence-transformers
pip install -e ".[api]"         # fastapi, uvicorn, websockets
pip install -e ".[deploy]"      # huggingface-hub

# Everything
pip install -e ".[all]"

# Development (pytest, ruff, coverage)
pip install -e ".[dev]"
```

## Usage

### Python API

```python
from adaptron.core.config import PipelineConfig, WizardAnswers

# Let the wizard pick the best strategy
answers = WizardAnswers(
    primary_goal="qa_docs",
    data_sources=["docs"],
    data_freshness="static",
    hardware="mid",
    timeline="medium",
    accuracy="professional",
    model_size="small",
)
config = PipelineConfig.from_wizard(answers)

print(config.base_model)       # e.g. Qwen/Qwen2.5-7B-Instruct
print(config.training_modes)   # e.g. ['qlora', 'rag']

# Or load from YAML
config = PipelineConfig.from_yaml("adaptron.yaml")
```

### CLI Commands

```bash
adaptron version              # Show version
adaptron init                 # Create adaptron.yaml with defaults
adaptron run                  # Execute the pipeline
adaptron wizard               # Interactive training strategy wizard
adaptron playground           # Chat with a finetuned model via Ollama
adaptron playground --rag     # Chat with RAG context augmentation
```

### Web UI

```bash
# Start the FastAPI backend
uvicorn adaptron.api.main:create_app --factory --reload

# In a separate terminal, start the Next.js frontend
cd web && npm install && npm run dev
```

Then open [http://localhost:3000](http://localhost:3000) to access the Wizard, Dashboard, and Playground.

## Architecture

```
                         adaptron.yaml
                              |
                              v
  +-------+  +------------+  +------------+  +-------+  +----------+  +--------+
  |Ingest | ->| Understand | ->| Synthesize | ->| Train | ->| Evaluate | ->| Deploy |
  +-------+  +------------+  +------------+  +-------+  +----------+  +--------+
   PDF,DOCX   Chunker,        Instruction     QLoRA,     Domain        Ollama,
   CSV,SQL    Entities,        templates      FullFT,    scoring       GGUF,
              Quality,                        CPT,DPO,                 HF Hub
              Schema                          Distill

  All stages are plugins registered in the global PluginRegistry.
  The PipelineOrchestrator runs them sequentially, emitting events
  via EventBus that the WebSocket API streams to the frontend.
```

## Plugin System

Every pipeline component is a plugin. Register your own with the `@register_plugin` decorator:

```python
from adaptron.core.registry import register_plugin
from adaptron.ingest.base import BaseIngester
from adaptron.ingest.models import DataSource, RawDocument

@register_plugin("ingester", "my_custom")
class MyCustomIngester(BaseIngester):
    def ingest(self, source: DataSource) -> list[RawDocument]:
        # Your custom ingestion logic
        ...

    def supported_types(self) -> list[str]:
        return ["my_custom"]
```

Retrieve plugins at runtime:

```python
from adaptron.core.registry import global_registry

ingester_cls = global_registry.get("ingest", "my_custom")
```

## Pipeline Stages

| Stage | Module | Plugins | Description |
|-------|--------|---------|-------------|
| **Ingest** | `adaptron.ingest` | `pdf`, `docx`, `csv`, `sql` | Extract text and metadata from data sources |
| **Understand** | `adaptron.understand` | `chunker`, `entities`, `quality`, `schema` | Semantic chunking, entity extraction, quality scoring, schema inference |
| **Synthesize** | `adaptron.synthesize` | `instruction` | Generate instruction-response training pairs from chunks |
| **Train** | `adaptron.train` | `qlora`, `full_ft`, `cpt`, `distill`, `alignment` (DPO) | Fine-tune or align the base model |
| **Evaluate** | `adaptron.evaluate` | `domain` | Score model outputs against domain-specific criteria |
| **Deploy** | `adaptron.deploy` | `ollama`, `gguf`, `huggingface` | Export and deploy the fine-tuned model |

## Playground

The playground lets you interact with deployed models:

- **Chat mode** -- streaming conversation with any Ollama-hosted model
- **RAG toggle** -- augment prompts with relevant context from ChromaDB
- **Comparison mode** -- send the same prompt to two models side-by-side (web UI)
- **CLI access** -- `adaptron playground --model adaptron-mymodel --rag`

## Configuration

Adaptron uses a YAML configuration file (`adaptron.yaml`):

```yaml
# Wizard answers -- drive automatic strategy selection
wizard:
  primary_goal: qa_docs          # qa_docs | erp_edw | report_gen | specialist
  data_sources:
    - docs                       # docs | erp | edw
  data_freshness: static         # static | monthly | daily | realtime
  hardware: mid                  # low | mid | high | cloud
  timeline: medium               # fast | medium | long | unlimited
  accuracy: professional         # professional | enterprise | mission
  model_size: small              # tiny | small | medium | large

# Manual overrides
overrides:
  epochs: 3
  learning_rate: 0.0002
  batch_size: 4
  lora_rank: 64
  max_seq_length: 2048
  quantization: Q4_K_M

data:
  input_dir: ./data
  output_dir: ./output

deploy:
  targets:
    - gguf
    - ollama
```

## Development

```bash
# Clone and install in dev mode
git clone <repo-url> && cd Adaptron
pip install -e ".[all,dev]"

# Run the test suite
pytest --tb=short -q

# Lint
ruff check adaptron/ tests/
```

## License

MIT License. See [LICENSE](LICENSE) for details.
