Metadata-Version: 2.4
Name: beyondbench
Version: 0.0.1
Summary: BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
Home-page: https://github.com/ctrl-gaurav/BeyondBench
Author: Aafiya Hussain, Zhenyu Bi, Swastik Roy, Priya Pitre, Meng Lu, Morteza Ziyadi, Xuan Wang
Author-email: Gaurav Srivastava <gks@vt.edu>
Maintainer-email: Gaurav Srivastava <gks@vt.edu>
License: MIT
Project-URL: Homepage, https://github.com/ctrl-gaurav/BeyondBench
Project-URL: Documentation, https://github.com/ctrl-gaurav/BeyondBench#readme
Project-URL: Repository, https://github.com/ctrl-gaurav/BeyondBench
Project-URL: Bug Tracker, https://github.com/ctrl-gaurav/BeyondBench/issues
Project-URL: Changelog, https://github.com/ctrl-gaurav/BeyondBench/blob/main/CHANGELOG.md
Project-URL: Paper, https://arxiv.org/abs/2509.24210
Keywords: language-models,evaluation,benchmark,reasoning,nlp,ai,machine-learning,llm-evaluation,computational-reasoning,contamination-resistant
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.5.0
Requires-Dist: transformers>=4.47.0
Requires-Dist: numpy>=2.0.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: tqdm>=4.67.0
Requires-Dist: tiktoken>=0.12.0
Requires-Dist: sentencepiece>=0.2.0
Requires-Dist: click>=8.1.0
Requires-Dist: rich>=14.0.0
Requires-Dist: colorama>=0.4.6
Requires-Dist: scipy>=1.14.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: jsonschema>=4.23.0
Requires-Dist: psutil>=6.0.0
Requires-Dist: tabulate>=0.9.0
Provides-Extra: openai
Requires-Dist: openai>=2.0.0; extra == "openai"
Provides-Extra: gemini
Requires-Dist: google-genai>=1.0.0; extra == "gemini"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40.0; extra == "anthropic"
Provides-Extra: vllm
Requires-Dist: vllm>=0.7.0; extra == "vllm"
Provides-Extra: all-apis
Requires-Dist: openai>=2.0.0; extra == "all-apis"
Requires-Dist: google-genai>=1.0.0; extra == "all-apis"
Requires-Dist: anthropic>=0.40.0; extra == "all-apis"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=6.0.0; extra == "dev"
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: isort>=5.13.0; extra == "dev"
Requires-Dist: flake8>=7.0.0; extra == "dev"
Requires-Dist: mypy>=1.8.0; extra == "dev"
Requires-Dist: pre-commit>=3.6.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=2.0.0; extra == "docs"
Requires-Dist: myst-parser>=3.0.0; extra == "docs"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.9.0; extra == "viz"
Requires-Dist: seaborn>=0.13.0; extra == "viz"
Requires-Dist: tabulate>=0.9.0; extra == "viz"
Provides-Extra: full
Requires-Dist: beyondbench[all-apis,dev,docs,viz]; extra == "full"
Requires-Dist: vllm>=0.7.0; extra == "full"
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

  BeyondBench Banner

[![Paper](https://img.shields.io/badge/📄_Paper-ArXiv%3A2509.24210-red?style=for-the-badge&logo=arxiv)](https://arxiv.org/abs/2509.24210)
[![Conference](https://img.shields.io/badge/🏆_ICLR-2026-blue?style=for-the-badge)](https://iclr.cc/)
[![PyPI](https://img.shields.io/pypi/v/beyondbench.svg?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/beyondbench/)
[![Python](https://img.shields.io/pypi/pyversions/beyondbench.svg?style=for-the-badge&logo=python&logoColor=white)](https://pypi.org/project/beyondbench/)
[![License](https://img.shields.io/badge/License-MIT-green.svg?style=for-the-badge)](LICENSE)
[![Stars](https://img.shields.io/github/stars/ctrl-gaurav/BeyondBench?style=for-the-badge&logo=github)](https://github.com/ctrl-gaurav/BeyondBench/stargazers)

*Contamination-Resistant Evaluation of Reasoning in Language Models*

**🏆 101+ Models Evaluated • 🧠 44 Reasoning Tasks • 🎯 117 Variations • 📊 >1015 Unique Instances**

[**🌟 Explore Leaderboard**](https://ctrl-gaurav.github.io/BeyondBench/) | [**📖 Read Paper**](https://arxiv.org/abs/2509.24210) | [**📦 PyPI**](https://pypi.org/project/beyondbench/) | [**📚 Documentation**](docs/DOCUMENTATION.md)

---

## 📢 Latest News

| Date | Update |
|------|--------|
| **Feb 2026** | v0.0.1 released — 44 tasks, 117 variations, 101+ models |
| **Jan 2026** | Paper accepted at **ICLR 2026** |
| **Jan 2026** | Interactive leaderboard website launched |
| **Sep 2025** | Paper submitted: [arXiv:2509.24210](https://arxiv.org/abs/2509.24210) |

---

## 💡 What is BeyondBench?

BeyondBench introduces a **revolutionary approach** to evaluating reasoning capabilities in language models without relying on traditional static benchmarks. Our system **dynamically generates** novel problems across **44 distinct reasoning tasks** with **117 variations**, ensuring that models cannot memorize solutions and must demonstrate **true reasoning abilities**.

### 🌟 Key Highlights

#### 🔄 **Dynamic Problem Generation**
- Problem space >10^15 unique instances
- Zero risk of data contamination
- Fresh problems on every evaluation

#### 🎯 **Three Difficulty Levels**
- **Easy**: 29 fundamental reasoning tasks
- **Medium**: 5 tasks with 49 variations
- **Hard**: 10 tasks with 68 variations

#### 🤖 **Multi-Backend Support**
- OpenAI, Gemini, Anthropic APIs
- vLLM for high-throughput local inference
- HuggingFace Transformers

#### 📊 **Comprehensive Metrics**
- Accuracy across difficulty levels
- Instruction-following compliance
- Token efficiency analysis

#### 🛡️ **Contamination-Resistant**
- No static benchmark memorization
- Novel problem generation
- Fair model comparison

#### ⚡ **Extensive Coverage**
- 101+ models evaluated
- Open-source and proprietary
- Regular updates with new models

---

## 🚀 Installation

### From PyPI

```bash
pip install beyondbench
```

### From Source

```bash
git clone https://github.com/ctrl-gaurav/BeyondBench.git
cd BeyondBench
pip install -e .
```

### With Optional Dependencies

```bash
# All API clients (OpenAI, Gemini, Anthropic)
pip install beyondbench[all-apis]

# vLLM support (requires CUDA)
pip install beyondbench[vllm]

# Everything
pip install beyondbench[full]
```

---

## ⚡ Quick Start

### Interactive Wizard

```bash
beyondbench
```

### Command Line

```bash
# Evaluate GPT-4o on the easy suite
beyondbench evaluate --model-id gpt-4o --api-provider openai --suite easy

# Evaluate a local model with vLLM
beyondbench evaluate --model-id meta-llama/Llama-3.2-3B-Instruct --backend vllm --suite all

# Evaluate Claude on hard tasks
beyondbench evaluate --model-id claude-sonnet-4-20250514 --api-provider anthropic --suite hard

# List available tasks
beyondbench list-tasks
```

### Python API

```python
from beyondbench import EvaluationEngine, ModelHandler, TaskRegistry

# Initialize model handler
model = ModelHandler(
    model_id="gpt-4o",
    api_provider="openai",
    api_key="your-api-key"
)

# Run evaluation
engine = EvaluationEngine(model_handler=model, output_dir="./results")
results = engine.run_evaluation(suite="easy", datapoints=100)

# Print results
print(f"Average Accuracy: {results['summary']['avg_accuracy']:.2%}")
```

---

## 🔌 Supported Backends

| Backend | Models | Features |
|---------|--------|----------|
| **OpenAI** | GPT-4o, GPT-4o-mini, GPT-5, GPT-5-mini | Reasoning effort control |
| **Gemini** | Gemini 2.5 Pro, Gemini 2.5 Flash | Thinking budget configuration |
| **Anthropic** | Claude Sonnet 4, Claude Opus 4 | Latest Claude models |
| **vLLM** | Any HuggingFace model | Batch processing, tensor parallelism |
| **Transformers** | Any HuggingFace model | CPU/GPU inference |

---

## 📊 Results

### 🏆 Leaderboard (Top Models)

 🏅 Rank
 🤖 Model
 📊 Overall
 🎯 Instruction Following

 🥇 GPT-5* 83.56% 96.15%
 🥈 GPT-5-Nano* 82.04% 93.58%
 🥉 GPT-5-Mini* 81.67% 94.23%
 4 o3* 80.36% 94.96%
 5 o4-Mini* 79.04% 95.30%

*Models marked with * use reasoning/thinking tokens. Full results for 101+ models available in the [paper](https://arxiv.org/abs/2509.24210) and on the [leaderboard](https://ctrl-gaurav.github.io/BeyondBench/).

### 🔍 Key Findings

- **Reasoning Gap**: Even top models show 20-30% performance drops on hard reasoning tasks
- **Scaling Effects**: Larger models generally perform better, but the relationship is not always linear
- **Instruction vs. Accuracy**: High accuracy does not guarantee perfect instruction-following

---

## 🧩 Task Suites

Easy Suite (29 Tasks)

| Category | Tasks |
|----------|-------|
| **Arithmetic** | sum, multiplication, subtraction, division, absolute_difference |
| **Statistics** | mean, median, mode |
| **Counting** | odd_count, even_count, count_negative, count_unique, count_greater_than_previous, count_palindromic, count_perfect_squares, count_multiples, local_maxima_count |
| **Extrema** | find_maximum, find_minimum, second_maximum, range, index_of_maximum, max_adjacent_difference, sum_of_max_indices |
| **Sequences** | sorting, longest_increasing_subsequence, alternating_sum, sum_of_digits |
| **Comparison** | comparison |

Medium Suite (5 Tasks, 49 Variations)

| Task | Variations |
|------|------------|
| **Fibonacci Sequence** | 6 (Tribonacci, Lucas numbers, modified recursive) |
| **Algebraic Sequence** | 10 (Polynomial, arithmetic, quadratic) |
| **Geometric Sequence** | 10 (Exponential, compound growth, factorial) |
| **Prime Sequence** | 11 (Prime gaps, twin primes, Sophie Germain) |
| **Complex Pattern** | 12 (Interleaved, conditional, multi-rule) |

Hard Suite (10 Tasks, 68 Variations)

| Task | Variations | Complexity |
|------|------------|------------|
| **Tower of Hanoi** | 6 | O(2^n) moves |
| **N-Queens** | 4 | NP-complete |
| **Graph Coloring** | 10 | NP-complete |
| **Boolean SAT** | 5 | NP-complete |
| **Sudoku** | 8 | Constraint satisfaction |
| **Cryptarithmetic** | 12 | Constraint satisfaction |
| **Matrix Chain** | 5 | Dynamic programming |
| **Modular Systems** | 5 | Number theory |
| **Constraint Optimization** | 5 | Operations research |
| **Logic Grid Puzzles** | 8 | Deductive reasoning |

---

## 📚 Documentation

- [**Full Documentation**](docs/DOCUMENTATION.md) — Complete API reference and configuration guide
- [**Usage Guide**](docs/USAGE.md) — Detailed usage examples for all backends

### Environment Variables

```bash
export OPENAI_API_KEY="sk-..."
export GEMINI_API_KEY="..."
export ANTHROPIC_API_KEY="sk-ant-..."
```

---

## 🤝 Contributing

We welcome contributions! See the [Contributing Guide](CONTRIBUTING.md) for details.

```bash
git clone https://github.com/ctrl-gaurav/BeyondBench.git
cd BeyondBench
pip install -e ".[dev]"
pre-commit install
pytest tests/ -v
```

### 🛠️ Ways to Contribute
- **🐛 Bug Reports**: Found an issue? [Report it here](https://github.com/ctrl-gaurav/BeyondBench/issues)
- **✨ Feature Requests**: Have ideas? [Share them here](https://github.com/ctrl-gaurav/BeyondBench/issues)
- **🔧 Code Contributions**: Submit PRs for improvements
- **📚 Documentation**: Help improve our docs
- **🤖 Model Submissions**: Suggest models for evaluation

---

## 📝 Citation

If you use BeyondBench in your research, please cite our paper (accepted at **ICLR 2026**):

```bibtex
@misc{srivastava2025beyondbenchbenchmarkfreeevaluationreasoning,
      title={BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models},
      author={Gaurav Srivastava and Aafiya Hussain and Zhenyu Bi and Swastik Roy and Priya Pitre and Meng Lu and Morteza Ziyadi and Xuan Wang},
      year={2025},
      eprint={2509.24210},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.24210},
}
```

---

## 📞 Contact & Support

- **📧 Email**: [gks@vt.edu](mailto:gks@vt.edu), [xuanw@vt.edu](mailto:xuanw@vt.edu)
- **🐛 Issues**: [GitHub Issues](https://github.com/ctrl-gaurav/BeyondBench/issues)
- **💬 Discussions**: [GitHub Discussions](https://github.com/ctrl-gaurav/BeyondBench/discussions)

---

## 📜 License

This project is licensed under the **MIT License** - see the [LICENSE](LICENSE) file for details.

---

## 🚀 Ready to Explore the Future of AI Evaluation?

**Made with ❤️ by the BeyondBench Team**

[![Virginia Tech](https://img.shields.io/badge/Virginia_Tech-CS_Department-maroon?style=flat&logo=data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZD0iTTEyIDJMMTMuMDkgOC4yNkwyMCA5TDEzLjA5IDE1Ljc0TDEyIDIyTDEwLjkxIDE1Ljc0TDQgOUwxMC45MSA4LjI2TDEyIDJaIiBmaWxsPSJjdXJyZW50Q29sb3IiLz4KPC9zdmc+)](https://cs.vt.edu/)
[![Amazon AGI](https://img.shields.io/badge/Amazon-AGI-orange?style=flat&logo=amazon)](https://www.amazon.science/)

*Advancing the frontier of AI reasoning evaluation, one benchmark at a time* 🌟

---

| 🏠 [**Home**](https://ctrl-gaurav.github.io/BeyondBench/) | 📊 [**Leaderboard**](https://ctrl-gaurav.github.io/BeyondBench/#leaderboard) | 📖 [**Paper**](https://arxiv.org/abs/2509.24210) | 💻 [**Code**](https://github.com/ctrl-gaurav/BeyondBench) |
|:---:|:---:|:---:|:---:|
| Main website | Interactive rankings | Research paper | Source code |

> **🎯 Transform your understanding of AI capabilities.** BeyondBench reveals what language models can truly reason about, beyond memorization. [**Start exploring now →**](https://ctrl-gaurav.github.io/BeyondBench/)

---

