Metadata-Version: 2.4
Name: claude-token-reducer
Version: 1.4.0
Summary: Open-source local-first context compression and token reduction pipeline for Claude Code with hybrid retrieval (BM25 + vectors), reranking, and AST-aware chunking.
License: MIT License
        
        Copyright (c) 2026 Madhan230205
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: bm25,claude-code,context-compression,embeddings,hybrid-retrieval,local-first,open-source,rag,token-reduction,tree-sitter
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Requires-Dist: pydantic>=2.0.0
Requires-Dist: typer[all]>=0.12.0
Provides-Extra: ast
Requires-Dist: tree-sitter-c>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-cpp>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-go>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-java>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-javascript>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-python>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-ruby>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-rust>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter-typescript>=0.21.0; extra == 'ast'
Requires-Dist: tree-sitter>=0.21.0; extra == 'ast'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pyright>=1.1; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: full
Requires-Dist: token-reducer[ast,ml]; extra == 'full'
Provides-Extra: ml
Requires-Dist: faiss-cpu>=1.8.0; extra == 'ml'
Requires-Dist: hnswlib>=0.8.0; extra == 'ml'
Requires-Dist: numpy<2.0.0,>=1.26.0; extra == 'ml'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'ml'
Description-Content-Type: text/markdown

# Token Reducer

### Cut Claude API costs by 90%+ with intelligent context compression

[![Claude Code Plugin](https://img.shields.io/badge/Claude%20Code-Plugin-blueviolet?style=for-the-badge&logo=anthropic)](https://claude.ai)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg?style=for-the-badge)](LICENSE)
[![Release](https://img.shields.io/github/v/release/Madhan230205/token-reducer?style=for-the-badge)](https://github.com/Madhan230205/token-reducer/releases)
[![Python 3.11+](https://img.shields.io/badge/Python-3.11+-blue?style=for-the-badge&logo=python&logoColor=white)](https://python.org)
[![SQLite](https://img.shields.io/badge/SQLite-FTS5-003B57?style=for-the-badge&logo=sqlite)](https://sqlite.org)

**The open-source alternative to expensive context management tools.**

[Easy Install](#-easy-install) • [Features](#-features) • [Documentation](#-documentation) • [Contributing](#-contributing)

---

## The Problem

Every time you use Claude with a large codebase, you're paying for **thousands of tokens** that aren't relevant to your query. Most context management tools either:

- Send everything (expensive)
- Truncate blindly (loses important context)
- Require heavy Language Servers (slow, resource-intensive)

## The Solution

**Token Reducer** is a local-first, intelligent context compression pipeline that:

- Reduces tokens by 90-98% while preserving semantic relevance
- Runs entirely locally — no API calls, no data leaving your machine
- Works in milliseconds — faster than Language Server alternatives
- Understands code semantically — AST parsing, not just text matching

```
┌─────────────────┐     ┌───────────────┐     ┌──────────────────┐
│  Your Codebase  │────▶│ Token Reducer │────▶│  Compressed      │
│  (50,000 tokens)│     │   Pipeline    │     │  Context (500t)  │
└─────────────────┘     └───────────────┘     └──────────────────┘
                              │
                    ┌─────────┴─────────┐
                    │  - AST Chunking   │
                    │  - BM25 + Vector  │
                    │  - TextRank       │
                    │  - Import Graph   │
                    │  - 2-Hop Symbols  │
                    └───────────────────┘
```

---

## Easy Install

### Option 1 — Claude Code `/plugin` Command (Recommended)

**Step 1:** Register the marketplace (one-time setup):
```
/plugin marketplace add Madhan230205/token-reducer
```

This registers the marketplace as `Madhan230205-token-reducer`.

**Step 2:** Install:
```
/plugin install token-reducer@Madhan230205-token-reducer
```

For project-scoped install:
```
/plugin install token-reducer@Madhan230205-token-reducer --scope project
```

> **Already ran Step 1 before?** Just run `/plugin install token-reducer@Madhan230205-token-reducer` — no need to add the marketplace again.

---

### Option 2 — Git Clone (Manual)

```bash
# 1. Clone into your Claude plugins folder
git clone https://github.com/Madhan230205/token-reducer.git ~/.claude/plugins/token-reducer

# 2. Install dependencies (optional but recommended for best results)
pip install -r ~/.claude/plugins/token-reducer/requirements-optional.txt
```

> **Windows users**: Replace `~/.claude/plugins/` with `%USERPROFILE%\.claude\plugins\`

Then open `~/.claude/settings.json` and add:

```json
{
  "plugins": ["~/.claude/plugins/token-reducer"]
}
```

Restart Claude Code. Done.

---

**What `requirements-optional.txt` installs:**

| Package | Purpose |
|---------|---------|
| `sentence-transformers` | Neural embeddings for smarter retrieval |
| `hnswlib` / `faiss-cpu` | Fast approximate nearest-neighbor search |
| `tree-sitter` + language grammars | AST-based code chunking (Python, JS, TS, Go, Rust, Java, C/C++, Ruby) |

If you skip this step, Token Reducer still works using hash embeddings and regex chunking — no ML libraries required.

---

### Option 3 — Zero-Dependency Quick Start

No pip, no ML libs — runs immediately after cloning:

```bash
git clone https://github.com/Madhan230205/token-reducer.git
cd token-reducer
python scripts/context_pipeline.py run \
  --inputs ./src \
  --query "Find auth logic" \
  --embedding-backend hash \
  --db .cache/index.db
```

---

## Features

### Core Pipeline
- **Hybrid Retrieval** — BM25 + semantic vector search with intelligent fallback
- **AST-Based Chunking** — Tree-sitter parsing for Python, TypeScript, Go, Rust, Java, and more
- **TextRank Compression** — Graph-based sentence scoring for intelligent summarization
- **Sub-100ms Queries** — SQLite FTS5 + HNSW indexes for instant results
- **Local-First** — Everything runs on your machine, no external APIs

### LSP-Killer Features
- **Import Graph** — Automatically maps file dependencies without Language Server
- **2-Hop Symbol Expansion** — Auto "go-to-definition" for referenced functions
- **Diff Protocol** — SEARCH/REPLACE edit format with automatic application
- **Semantic Clustering** — Groups similar chunks to avoid redundancy

### Enterprise Ready
- **Fully Configurable** — 40+ tunable parameters in `settings.json`
- **Embedding Flexibility** — ML models or hash fallback (zero dependencies)
- **Query Caching** — Intelligent TTL-based caching for repeated queries
- **Session Memory** — Tracks context across conversation turns

---

## Documentation

### How It Works

```
Query → FTS(BM25) → (Vector fallback if needed) → Merge → Top 5 → Compress
```

Full pipeline:

```
PREPROCESS → INDEX → RETRIEVE → RE-RANK → COMPRESS → CONTEXT PACKET
```

### Basic Usage

```bash
# Index your codebase
python scripts/context_pipeline.py index --inputs ./src --db .cache/index.db

# Query with compression
python scripts/context_pipeline.py query \
  --query "How does authentication work?" \
  --db .cache/index.db \
  --json

# One-shot: index + query
python scripts/context_pipeline.py run \
  --inputs ./src \
  --query "Find the database connection logic" \
  --db .cache/index.db
```

### Configuration

All settings in `settings.json`:

```json
{
  "tokenReducer": {
    "chunkSizeWords": 220,
    "embeddingModel": "jinaai/jina-embeddings-v2-base-code",
    "hybridMode": "fallback",
    "astChunkingEnabled": true,
    "textRankEnabled": true,
    "lspFeatures": {
      "importGraphEnabled": true,
      "twoHopExpansionEnabled": true
    }
  }
}
```


Full Configuration Reference

| Setting | Default | Description |
|---------|---------|-------------|
| `chunkSizeWords` | 220 | Target words per chunk |
| `embeddingBackend` | "ml" | "ml" for neural, "hash" for zero-dep |
| `embeddingModel` | jina-v2-code | Code-optimized embeddings |
| `hybridMode` | "fallback" | "fallback" or "always" for vector |
| `astChunkingEnabled` | true | Use tree-sitter AST parsing |
| `textRankEnabled` | true | Graph-based sentence scoring |
| `importGraphEnabled` | true | Track file dependencies |
| `twoHopExpansionEnabled` | true | Auto-expand referenced symbols |
| `compressionWordBudget` | 350 | Max words in compressed output |

### Zero-Dependency Mode

Run without any ML libraries:

```bash
python scripts/context_pipeline.py run \
  --inputs ./src \
  --query "Find auth logic" \
  --embedding-backend hash \
  --db .cache/index.db
```

### Apply Code Edits

```bash
python scripts/apply_diff.py --input claude_response.txt --dir ./src
python scripts/apply_diff.py --input response.txt --dry-run
```

---

## Architecture

### Technology Stack

- **Storage**: SQLite with FTS5 + custom embeddings table
- **Chunking**: Tree-sitter AST parsing with regex fallback
- **Embeddings**: Jina Code v2 (or zero-dependency hash embeddings)
- **ANN Search**: HNSW via hnswlib (with FAISS fallback)
- **Compression**: TextRank + query-relevance scoring

### Repository Structure

```text
token-reducer/
├── .claude-plugin/plugin.json
├── .mcp.json
├── .env.example
├── settings.json
├── requirements-optional.txt
├── scripts/
├── hooks/
├── commands/
├── agents/
├── skills/
└── evals/
```

---

## Contributing

If anyone is interested in contributing, this project is open to contributions.
Please see [contribute.md](contribute.md) for contribution guidelines.

```bash
git clone https://github.com/Madhan230205/token-reducer.git
cd token-reducer
pip install -e ".[dev]"
python scripts/context_pipeline.py self-test
```

---

## License

MIT License — see [LICENSE](LICENSE) for details.

---

## Acknowledgments

- [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) for AST parsing
- [Sentence Transformers](https://www.sbert.net/) for embeddings
- [SQLite FTS5](https://sqlite.org/fts5.html) for blazing-fast text search
- [hnswlib](https://github.com/nmslib/hnswlib) for approximate nearest neighbors

---

**Star this repo if Token Reducer saves you money!**

[Report Bug](https://github.com/Madhan230205/token-reducer/issues) • [Request Feature](https://github.com/Madhan230205/token-reducer/issues) • [Discussions](https://github.com/Madhan230205/token-reducer/discussions)

