Metadata-Version: 2.4
Name: checkagent
Version: 0.0.1a1
Summary: The open-source testing framework for AI agents
Project-URL: Homepage, https://github.com/xydac/checkagent
Project-URL: Repository, https://github.com/xydac/checkagent
Project-URL: Issues, https://github.com/xydac/checkagent/issues
Author: CheckAgent Contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,ai,evaluation,llm,pytest,testing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: click>=8.0
Requires-Dist: pluggy>=1.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pytest-asyncio>=0.23
Requires-Dist: pytest>=7.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: deepdiff>=7.0; extra == 'dev'
Requires-Dist: dirty-equals>=0.7; extra == 'dev'
Requires-Dist: jsonschema>=4.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: opentelemetry-api>=1.20; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest-xdist>=3.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: json-schema
Requires-Dist: jsonschema>=4.0; extra == 'json-schema'
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.20; extra == 'otel'
Provides-Extra: safety-ner
Requires-Dist: spacy>=3.7; extra == 'safety-ner'
Provides-Extra: structured
Requires-Dist: deepdiff>=7.0; extra == 'structured'
Requires-Dist: dirty-equals>=0.7; extra == 'structured'
Description-Content-Type: text/markdown

# CheckAgent

**The open-source testing framework for AI agents.**

*pytest-native · async-first · CI/CD-first · safety-aware*

[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org)

---

CheckAgent is a pytest plugin for testing AI agent workflows. It provides layered testing — from free, millisecond unit tests to LLM-judged evaluations with statistical rigor — so you can ship agents with the same confidence you ship traditional software.

## Why CheckAgent

- **pytest-native** — tests are `.py` files, assertions are `assert`, markers and fixtures are standard pytest
- **Async-first** — most agent frameworks are async; CheckAgent is too
- **Framework-agnostic** — works with LangChain, OpenAI Agents SDK, CrewAI, PydanticAI, Anthropic, or any Python callable
- **Cost-aware** — every test run tracks token usage and estimated cost, with budget limits
- **Safety built-in** — prompt injection, PII leakage, and tool misuse testing ships as core

## The Testing Pyramid

```
┌─────────────────────────────────────────────┐
│  Layer 4: JUDGE                             │
│  LLM-as-judge · statistical assertions      │
│  ⏱ Minutes  💰 $$$  📍 Nightly              │
├─────────────────────────────────────────────┤
│  Layer 3: EVAL                              │
│  Agent metrics · golden datasets            │
│  ⏱ Seconds  💰 $$   📍 On merge             │
├─────────────────────────────────────────────┤
│  Layer 2: REPLAY                            │
│  Record-and-replay · regression testing     │
│  ⏱ Seconds  💰 $    📍 On every PR          │
├─────────────────────────────────────────────┤
│  Layer 1: MOCK                              │
│  Deterministic unit tests · zero LLM cost   │
│  ⏱ Milliseconds  💰 Free  📍 On every commit│
└─────────────────────────────────────────────┘
```

## Quick Start

```bash
pip install checkagent
mkdir my-agent-tests && cd my-agent-tests
checkagent init
pytest tests/ -v
```

Or try the zero-config demo (no API keys needed):

```bash
pip install checkagent
checkagent demo
```

## Example Test

```python
import pytest
from checkagent import AgentInput, AgentRun, Step, ToolCall, assert_tool_called

# Your agent — any async function that calls LLMs and tools
async def booking_agent(query, *, llm, tools):
    plan = await llm.complete(query)
    event = await tools.call("create_event", {"title": "Meeting"})
    return AgentRun(
        input=AgentInput(query=query),
        steps=[Step(output_text=plan, tool_calls=[
            ToolCall(name="create_event", arguments={"title": "Meeting"}, result=event),
        ])],
        final_output=event,
    )

# Test with zero LLM cost, deterministic, milliseconds
@pytest.mark.agent_test(layer="mock")
async def test_booking(ap_mock_llm, ap_mock_tool):
    ap_mock_llm.on_input(contains="book").respond("Booking your meeting now.")
    ap_mock_tool.on_call("create_event").respond(
        {"confirmed": True, "event_id": "evt-123"}
    )

    result = await booking_agent(
        "Book a meeting", llm=ap_mock_llm, tools=ap_mock_tool
    )

    assert_tool_called(result, "create_event", title="Meeting")
    assert result.final_output["confirmed"] is True
```

## Features

| Feature | Status |
|---------|--------|
| Mock LLM & tool providers | ✅ Implemented |
| Streaming mock support | ✅ Implemented |
| Fault injection (timeouts, rate limits) | ✅ Implemented |
| Structured output assertions | ✅ Implemented |
| Multi-turn conversation testing | ✅ Implemented |
| Record-and-replay cassettes | ✅ Implemented |
| Evaluation metrics (task completion, tool correctness) | ✅ Implemented |
| Safety testing (prompt injection, PII leakage) | ✅ Implemented |
| LLM-as-judge with statistical assertions | ✅ Implemented |
| CI/CD quality gates (GitHub Action) | ✅ Implemented |
| Production trace import | ✅ Implemented |
| Cost tracking & budget limits | ✅ Implemented |
| Multi-agent trace & credit assignment | ✅ Implemented |

## Roadmap

All core milestones are implemented. See [ROADMAP.md](ROADMAP.md) for what's next:

- More framework adapters (AutoGen, DSPy, Marvin)
- Production trace import from OpenTelemetry/Langfuse
- VS Code extension for inline test results
- Community plugin ecosystem

## Contributing

We're building this in public and contributions are welcome from day one. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

Apache-2.0. See [LICENSE](LICENSE).
