Metadata-Version: 2.4
Name: ai-prompt-forge
Version: 1.0.0
Summary: The Swiss Army knife for prompt engineering — build, test, version, and optimize prompts 10x faster.
Author: theihtisham
License: MIT
Project-URL: Homepage, https://github.com/user/promptforge
Project-URL: Repository, https://github.com/user/promptforge
Project-URL: Documentation, https://github.com/user/promptforge#readme
Keywords: prompt,engineering,llm,openai,anthropic,testing,templates
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: rich>=13.0
Requires-Dist: flask>=3.0
Requires-Dist: jinja2>=3.1
Requires-Dist: openai>=1.0
Requires-Dist: anthropic>=0.25
Requires-Dist: tiktoken>=0.7
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: httpx>=0.27
Requires-Dist: diskcache>=5.6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: markupsafe>=2.1
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: pytest-flask>=1.3; extra == "dev"
Dynamic: license-file

# PromptForge

**The Swiss Army knife for prompt engineering -- build, test, version, and optimize prompts 10x faster.**

[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-brightgreen.svg)](https://www.python.org/downloads/)
[![Pydantic v2](https://img.shields.io/badge/pydantic-v2-orange.svg)](https://docs.pydantic.dev/)
[![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![PyPI version](https://img.shields.io/pypi/v/ai-prompt-forge?style=for-the-badge&logo=pypi&color=3775A9)](https://pypi.org/project/ai-prompt-forge/)

---

## Why PromptForge?

Writing a good prompt is half the battle. Managing hundreds of prompts across projects, tracking which version performs best, estimating token costs before you hit "send" -- that is the other half, and it is where most teams waste time.

PromptForge gives you a single toolkit to:

- **Build** parameterized prompt templates with `{{variable}}` injection
- **Test** prompts with built-in A/B testing and batch evaluation
- **Version** every change with full diff and fork support
- **Score** output quality on coherence, relevance, safety, completeness, and clarity
- **Compress** prompts to use fewer tokens without losing meaning
- **Chain** prompts into multi-step pipelines with automatic output passing
- **Count** tokens and estimate costs across 16+ LLM models
- **Export** to JSON, YAML, or LangChain format

All backed by a file-based store (no database required) with support for OpenAI, Anthropic, Google Gemini, and local models (Ollama, LM Studio).

---

## Features

### Prompt Templates with Variable Injection

Define templates with `{{variable}}` placeholders. PromptForge auto-detects variables, validates required ones, and fills defaults.

### Full Version History

Every save creates a version snapshot. Diff any two versions, fork from a specific version, and track the evolution of your prompts over time.

### A/B Testing Framework

Compare two prompts head-to-head on the same inputs. PromptForge runs both, scores outputs on five quality dimensions, and declares a winner.

### Batch Testing

Run a suite of inputs through a single prompt and get back quality scores, latencies, token usage, and estimated cost per run.

### Chain-of-Thought Pipelines

Link prompts into multi-step pipelines where each step's output flows into the next as a `{{step_N_output}}` variable.

### Quality Scoring

Deterministic, heuristic-based scoring across five dimensions -- coherence, relevance, safety, completeness, and clarity -- with weighted overall score. No LLM call required.

### Prompt Compression

Reduce token count with configurable aggression (0.0 to 1.0) by removing filler phrases, collapsing whitespace, shortening verbose instructions, and compressing lists.

### Token Counting and Cost Estimation

Accurate token counts via tiktoken with pricing tables for GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Gemini 2.0 Flash, Llama 3.1, Mistral, and more. Compare costs across all models in one call.

### Multi-Provider LLM Engine

Unified interface for OpenAI, Anthropic, Google Gemini, and local models (Ollama / LM Studio). Switch providers with a single parameter.

### Export to JSON, YAML, and LangChain

Export individual prompts or your entire library. LangChain export generates `PromptTemplate`-compatible JSON with `input_variables`, `template_format`, and `metadata`.

---

## Installation

```bash
# Install from PyPI
pip install ai-prompt-forge

# Or from source
git clone https://github.com/theihtisham/ai-prompt-forge.git
cd ai-prompt-forge
pip install -e .

# With dev dependencies (pytest, coverage)
pip install -e ".[dev]"
```

### Requirements

- Python 3.10+
- API keys (set as environment variables):
  - `OPENAI_API_KEY` for OpenAI
  - `ANTHROPIC_API_KEY` for Anthropic
  - `GOOGLE_API_KEY` for Google Gemini
  - Local models need no key (uses `http://localhost:11434` by default)

---

## Quick Start

### Create and Render a Template

```python
from promptforge.models import PromptTemplate, PromptCategory

template = PromptTemplate(
    name="Code Reviewer",
    description="Reviews code for bugs and style issues",
    category=PromptCategory.CODING,
    template="""Review the following {{language}} code for bugs, style issues,
and performance problems. Be specific and actionable.

Code:
```{{language}}}
{{code}}
```

Focus on: {{focus_areas}}""",
)

# Render with variables
rendered = template.render({
    "language": "python",
    "code": "def add(a, b): return a + b",
    "focus_areas": "edge cases, type safety",
})
print(rendered)
```

### Save, Version, and Fork

```python
from promptforge.store import PromptStore

store = PromptStore()  # defaults to ~/.promptforge/data

# Save (creates version 1)
template = store.save_prompt(template)

# Update the template and save again (creates version 2)
template.template = "You are an expert {{language}} developer. Review:\n{{code}}"
template.version = 2
store.save_prompt(template)

# Diff versions
diff = store.diff_versions(template.id, old_ver=1, new_ver=2)
print(diff.summary)       # "+2 lines, -1 lines"
print(diff.additions)     # list of added lines
print(diff.deletions)     # list of removed lines

# Fork from version 1 into a new prompt
forked = store.fork_version(template.id, version=1, new_name="Simple Reviewer")
```

### List and Search Prompts

```python
# List all prompts
all_prompts = store.list_prompts()

# Filter by category
coding_prompts = store.list_prompts(category="coding")

# Filter by tag
tagged = store.list_prompts(tag="production")

# Full-text search
results = store.list_prompts(search="code review")
```

### Generate with LLM Providers

```python
from promptforge.engine import get_provider

# OpenAI
openai = get_provider("openai", model="gpt-4o")
response = openai.generate("Explain recursion in one paragraph.")
print(response.text)
print(f"Tokens: {response.total_tokens}, Latency: {response.latency_ms:.0f}ms")

# Anthropic
anthropic = get_provider("anthropic", model="claude-3-5-sonnet-20241022")
response = anthropic.generate("Explain recursion in one paragraph.", temperature=0.3)

# Google Gemini
google = get_provider("google", model="gemini-2.0-flash")
response = google.generate("Explain recursion in one paragraph.")

# Local (Ollama)
local = get_provider("local", model="llama3.1")
response = local.generate("Explain recursion in one paragraph.")
```

### A/B Testing

```python
from promptforge.testing import ABTester
from promptforge.models import ABTestConfig, ModelProvider

config = ABTestConfig(
    name="Review style comparison",
    prompt_a_id=prompt_a.id,
    prompt_b_id=prompt_b.id,
    test_inputs=[
        "Write a function to reverse a linked list",
        "Implement binary search in Python",
        "Create a REST API endpoint for user login",
    ],
    provider=ModelProvider.OPENAI,
    model="gpt-4o-mini",
)

tester = ABTester(store=store)
results, summary = tester.run_test(config)

print(f"Wins A: {summary.wins_a}, Wins B: {summary.wins_b}, Ties: {summary.ties}")
print(f"Recommendation: {summary.recommendation}")
```

### Batch Testing

```python
from promptforge.testing import BatchTester
from promptforge.models import BatchTestConfig, ModelProvider

config = BatchTestConfig(
    prompt_id=template.id,
    inputs=[
        "Write a sorting function",
        "Debug this null pointer exception",
        "Explain the difference between TCP and UDP",
    ],
    provider=ModelProvider.OPENAI,
    model="gpt-4o-mini",
)

tester = BatchTester(store=store)
result = tester.run_batch(config)

print(f"Success: {result.successful}/{result.total_inputs}")
print(f"Avg latency: {result.avg_latency_ms:.0f}ms")
print(f"Total tokens: {result.total_tokens_used}")
print(f"Estimated cost: ${result.estimated_cost_usd:.4f}")
```

### Chain Pipelines

```python
from promptforge.chain import ChainBuilder
from promptforge.models import ModelProvider

builder = ChainBuilder(store=store)

# Create a 3-step pipeline
pipeline = builder.create_pipeline(
    name="Research Pipeline",
    description="Research -> Summarize -> Format",
    step_prompt_ids=[research_prompt.id, summarize_prompt.id, format_prompt.id],
)

# Run it
result = builder.run_pipeline(
    pipeline_id=pipeline.id,
    initial_variables={"topic": "quantum computing"},
    provider=ModelProvider.OPENAI,
    model="gpt-4o-mini",
)

print(result.final_output)
print(f"Total tokens: {result.total_tokens_used}")
print(f"Steps completed: {len(result.step_results)}")
```

### Quality Scoring

```python
from promptforge.quality import QualityScorer

scorer = QualityScorer()

scores = scorer.score_all(
    output="To implement binary search, first sort the array...",
    prompt="Explain how binary search works",
)

print(f"Coherence:    {scores['coherence']:.2f}")
print(f"Relevance:    {scores['relevance']:.2f}")
print(f"Safety:       {scores['safety']:.2f}")
print(f"Completeness: {scores['completeness']:.2f}")
print(f"Clarity:      {scores['clarity']:.2f}")
print(f"Overall:      {scores['overall']:.2f}")
```

### Prompt Compression

```python
from promptforge.compression import PromptCompressor

compressor = PromptCompressor()

result = compressor.compress(
    """Please note that it is important to understand that in order to
    implement binary search, you should first make sure to sort the array
    prior to beginning the search algorithm. Keep in mind that the time
    complexity is O(log n).""",
    aggression=0.7,
)

print(result["compressed"])
print(f"Reduction: {result['reduction_pct']}%")
print(f"Steps applied: {', '.join(result['steps_applied'])}")
```

### Token Counting and Cost Estimation

```python
from promptforge.token_counter import count_tokens, estimate_cost, analyze_text, compare_models

# Count tokens
tokens = count_tokens("Explain quantum computing in detail.", model="gpt-4o")
print(f"Token count: {tokens}")

# Estimate cost
cost = estimate_cost(input_tokens=500, output_tokens=200, model="gpt-4o")
print(f"Estimated cost: ${cost:.6f}")

# Full analysis
analysis = analyze_text("Your long prompt here...", model="gpt-4o", estimated_output_tokens=500)
print(f"Input tokens: {analysis['input_tokens']}")
print(f"Context usage: {analysis['context_usage_pct']}%")
print(f"Total cost: ${analysis['total_cost_usd']:.6f}")

# Compare across all models
comparisons = compare_models("Your prompt text", estimated_output_tokens=500)
for c in comparisons:
    print(f"{c['model']:25s}  {c['total_tokens']:6d} tokens  ${c['total_cost_usd']:.6f}")
```

### Export

```python
from promptforge.exporter import PromptExporter
from promptforge.models import ExportFormat

exporter = PromptExporter(store=store)

# Export a single prompt
json_str = exporter.export_prompt(template, fmt=ExportFormat.JSON)
yaml_str = exporter.export_prompt(template, fmt=ExportFormat.YAML)
langchain_str = exporter.export_prompt(template, fmt=ExportFormat.LANGCHAIN)

# Export all prompts
all_json = exporter.export_all(fmt=ExportFormat.JSON)

# Export specific prompts by ID
selected = exporter.export_by_ids(["abc123", "def456"], fmt=ExportFormat.YAML)
```

---

## Architecture

```
promptforge/
├── models.py          # Pydantic data models (PromptTemplate, ABTestConfig, etc.)
├── store.py           # File-based persistence layer (~/.promptforge/data/)
├── engine.py          # LLM provider integrations (OpenAI, Anthropic, Google, Local)
├── quality.py         # Output quality scorer (coherence, relevance, safety, clarity)
├── testing.py         # A/B testing and batch testing frameworks
├── chain.py           # Chain-of-thought pipeline builder
├── compression.py     # Prompt compression engine
├── token_counter.py   # Token counting and cost estimation (tiktoken)
├── exporter.py        # Export to JSON, YAML, and LangChain format
└── __init__.py        # Package metadata
```

### Data Flow

```
PromptTemplate  ──render()──>  Rendered Prompt
       │                            │
       │                            ├──> engine.generate()  ──> LLMResponse
       │                            ├──> quality.score_all()──> QualityScore
       │                            ├──> compression.compress()
       │                            └──> token_counter.analyze_text()
       │
       ├──> store.save_prompt()    (creates version snapshot)
       ├──> store.diff_versions()  (compare two versions)
       ├──> store.fork_version()   (branch from a version)
       └──> exporter.export_prompt() (JSON / YAML / LangChain)
```

### Storage Layout

All data is stored under `~/.promptforge/data/` (configurable via `PROMPTFORGE_DATA_DIR`):

```
~/.promptforge/data/
├── prompts/              # Prompt templates (one JSON file per prompt)
├── versions/             # Version snapshots ({id}_v{N}.json)
├── tests/                # A/B test configs and batch results
└── chains/               # Chain pipelines and run results
```

---

## API Reference

### `promptforge.models`

| Class | Description |
|---|---|
| `PromptTemplate` | Core template model with `render()` and `extract_variables()` |
| `VariableDefinition` | Variable schema with name, default, required, and example |
| `PromptCategory` | Enum: coding, writing, analysis, creative, business, education, etc. |
| `ModelProvider` | Enum: openai, anthropic, google, local |
| `ExportFormat` | Enum: json, yaml, langchain |
| `QualityMetric` | Enum: coherence, relevance, safety, completeness, clarity |
| `VersionDiff` | Diff result between two prompt versions |
| `ABTestConfig` | A/B test configuration |
| `ABTestResult` | Single A/B test run result |
| `ABTestSummary` | Aggregated A/B test summary with recommendation |
| `BatchTestConfig` | Batch test configuration |
| `BatchTestResult` | Batch test result with cost and latency stats |
| `ChainPipeline` | Chain-of-thought pipeline definition |
| `ChainStep` | Single step in a chain pipeline |
| `ChainRunResult` | Result of running a chain pipeline |
| `TokenCount` | Token count result for a specific model |
| `QualityScore` | Quality scores with weighted overall |

### `promptforge.store.PromptStore`

| Method | Description |
|---|---|
| `save_prompt(prompt)` | Save a template (creates version snapshot) |
| `get_prompt(prompt_id)` | Retrieve a template by ID |
| `list_prompts(category, tag, search)` | List prompts with optional filters |
| `delete_prompt(prompt_id)` | Delete a template and all versions |
| `get_version(prompt_id, version)` | Get a specific version |
| `list_versions(prompt_id)` | List all version numbers |
| `diff_versions(prompt_id, old_ver, new_ver)` | Diff two versions |
| `fork_version(prompt_id, version, new_name)` | Fork from a specific version |
| `save_chain(chain)` | Save a chain pipeline |
| `get_chain(chain_id)` | Retrieve a chain pipeline |
| `list_chains()` | List all chain pipelines |

### `promptforge.engine`

| Function / Class | Description |
|---|---|
| `get_provider(provider, model, api_key)` | Factory function to create a provider |
| `OpenAIProvider` | OpenAI API integration |
| `AnthropicProvider` | Anthropic API integration |
| `GoogleProvider` | Google Gemini API integration |
| `LocalProvider` | Local model integration (Ollama, LM Studio) |
| `LLMResponse` | Standardized response with text, tokens, latency |

### `promptforge.quality.QualityScorer`

| Method | Description |
|---|---|
| `score_coherence(text)` | Score internal consistency and structure (0-1) |
| `score_relevance(output, prompt)` | Score relevance to the input prompt (0-1) |
| `score_safety(text)` | Score safety (penalizes harmful content) (0-1) |
| `score_completeness(output, prompt)` | Score how completely the prompt is addressed (0-1) |
| `score_clarity(text)` | Score readability and understandability (0-1) |
| `score_all(output, prompt)` | All five scores plus weighted overall |

### `promptforge.testing`

| Class | Method | Description |
|---|---|---|
| `ABTester` | `run_test(config, variables)` | Run A/B test, returns results and summary |
| `BatchTester` | `run_batch(config, variables)` | Run batch test, returns result with stats |

### `promptforge.chain.ChainBuilder`

| Method | Description |
|---|---|
| `create_pipeline(name, step_prompt_ids)` | Create a new pipeline with ordered steps |
| `add_step(pipeline_id, prompt_id)` | Add a step to an existing pipeline |
| `run_pipeline(pipeline_id, initial_variables)` | Execute pipeline sequentially |
| `list_pipelines()` | List all saved pipelines |

### `promptforge.compression.PromptCompressor`

| Method | Description |
|---|---|
| `compress(text, aggression)` | Compress prompt (aggression 0.0-1.0) |

### `promptforge.token_counter`

| Function | Description |
|---|---|
| `count_tokens(text, model)` | Count tokens for a given model |
| `estimate_cost(input_tokens, output_tokens, model)` | Estimate cost in USD |
| `analyze_text(text, model, estimated_output_tokens)` | Full token and cost analysis |
| `compare_models(text, estimated_output_tokens)` | Compare cost across all models |
| `get_supported_models()` | List all supported model names |

### `promptforge.exporter.PromptExporter`

| Method | Description |
|---|---|
| `export_prompt(prompt, fmt)` | Export a single prompt (json/yaml/langchain) |
| `export_all(fmt)` | Export all prompts |
| `export_by_ids(prompt_ids, fmt)` | Export selected prompts by ID |

---

## Supported Models

| Provider | Models |
|---|---|
| **OpenAI** | gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-4, gpt-4-32k, gpt-3.5-turbo |
| **Anthropic** | claude-3.5-sonnet, claude-3-opus, claude-3-haiku, claude-3.5-haiku |
| **Google** | gemini-1.5-pro, gemini-1.5-flash, gemini-2.0-flash |
| **Local** | llama-3.1-70b, llama-3.1-8b, mistral-large, mixtral-8x7b |

---

## Development

```bash
# Clone the repository
git clone https://github.com/theihtisham/ai-prompt-forge.git
cd ai-prompt-forge

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=promptforge --cov-report=term-missing
```

### Project Structure

```
12-prompt-forge/
├── pyproject.toml           # Build config, dependencies, and metadata
├── requirements.txt         # Pinned dependencies
├── LICENSE                  # MIT License
├── .gitignore
└── src/
    └── promptforge/
        ├── __init__.py      # Package metadata (v1.0.0)
        ├── models.py        # Pydantic data models and enums
        ├── store.py         # File-based storage layer
        ├── engine.py        # LLM provider integrations
        ├── quality.py       # Output quality scorer
        ├── testing.py       # A/B and batch testing
        ├── chain.py         # Chain-of-thought pipelines
        ├── compression.py   # Prompt compression engine
        ├── token_counter.py # Token counting and cost estimation
        └── exporter.py      # Multi-format export
```

---

## Contributing

Contributions are welcome. To contribute:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/my-feature`)
3. Write tests for your changes
4. Ensure all tests pass (`pytest`)
5. Commit with a descriptive message
6. Open a pull request

### Guidelines

- Follow the existing code style (type hints on all public APIs, Pydantic models for data)
- Add tests for new functionality
- Keep the public API backward-compatible
- Document new parameters and return types

---

## License

This project is licensed under the [MIT License](LICENSE).
