---
name: agentbyte-eval
description: Guide for evaluating Agentbyte models, agents, and orchestrators with pointwise judges, local checks, reports, and pairwise comparison. Use this when implementing eval suites or comparing configurations.
---

Use this skill when a user wants to build evaluation suites for AgentByte agents, orchestrators, or raw model targets.

## Inputs To Confirm

- Whether the user needs pointwise scoring, local pass/fail checks, or pairwise comparison
- Which runtime is being evaluated: raw model, agent, or orchestrator
- Whether the evaluation target is the final answer or a specific agent's messages inside a multi-agent trajectory
- Whether the result should be a numeric score, a CI-style report, or a win/loss summary

## Core Eval Shapes

The eval package is built around four primary models:

- `EvalTask`: what to run
- `EvalTrajectory`: what happened at runtime
- `EvalScore`: numeric pointwise judgment
- `PairwiseResult`: relative A-vs-B judgment

Use the public package root:

```python
from agentbyte.eval import (
    AgentEvalTarget,
    EvalDatasetRecord,
    EvalRunner,
    EvalTask,
    ExactMatchJudge,
    PairwiseJudge,
    PairwiseRunner,
    save_eval_dataset,
    tasks_from_eval_dataset,
)
```

## Pattern: Pointwise Evaluation

Use pointwise evaluation when the question is "how good is this response?"

```python
import asyncio

from agentbyte.eval import AgentEvalTarget, EvalRunner, EvalTask, ExactMatchJudge


async def main(agent):
    tasks = [
        EvalTask(
            name="capital",
            input="What is the capital of France?",
            expected_output="Paris",
        ),
        EvalTask(
            name="addition",
            input="What is 17 + 25?",
            expected_output="42",
        ),
    ]

    runner = EvalRunner(judge=ExactMatchJudge(), parallel=True)
    scores = await runner.evaluate(AgentEvalTarget(agent), tasks)

    for score in scores:
        print(score.overall, score.reasoning)


asyncio.run(main(agent))
```

### Pointwise criteria precedence

`EvalRunner.evaluate()` resolves criteria in this order:

1. `criteria=` argument passed to `evaluate()`
2. `EvalTask.metadata["_criteria"]` (set manually or loaded from eval datasets)
3. Judge-level default behavior

This allows per-task dataset criteria to flow into scoring without custom wiring.

### Available pointwise targets

- `ModelEvalTarget`: wrap a raw chat-completion client
- `AgentEvalTarget`: wrap an `Agent`
- `OrchestratorEvalTarget`: wrap a `BaseOrchestrator`

### Available pointwise judges

- `ExactMatchJudge`
- `ContainsJudge`
- `FuzzyMatchJudge`
- `LLMEvalJudge`
- `CompositeJudge`

## Pattern: Local Checks And Reports

Use local checks when the question is "did the run satisfy required behavior?"

```python
import asyncio

from agentbyte.eval import (
    AgentEvalTarget,
    EvalRunner,
    EvalTask,
    ExactMatchJudge,
    LocalEvaluator,
    keyword_check,
    threshold_gate,
)


async def main(agent):
    task = EvalTask(
        name="weather",
        input="What's the weather in Paris?",
        expected_output="sunny",
    )

    runner = EvalRunner(judge=ExactMatchJudge(), parallel=False)
    scores = await runner.evaluate(AgentEvalTarget(agent), [task])
    trajectories = [score.trajectory for score in scores if score.trajectory is not None]

    report = await LocalEvaluator(
        threshold_gate(ExactMatchJudge(), threshold=7.0),
        keyword_check("sunny"),
    ).evaluate(trajectories)

    report.raise_for_status()


asyncio.run(main(agent))
```

### Built-in checks

- `tool_calls_present`
- `tool_call_args_match`
- `keyword_check`
- `@evaluator`
- `threshold_gate`

## Pattern: Pairwise Evaluation

Use pairwise evaluation when the question is "which of these two configurations is better?"

```python
import asyncio

from agentbyte.eval import (
    AgentEvalTarget,
    EvalTask,
    PairwiseJudge,
    PairwiseRunner,
    compare_pairwise,
)


async def main(baseline_agent, candidate_agent, judge_client):
    tasks = [
        EvalTask(name="capital", input="What is the capital of France?"),
        EvalTask(name="rewrite", input="Rewrite this note in a clearer tone."),
    ]

    judge = PairwiseJudge(client=judge_client, criteria=["accuracy", "helpfulness"])
    runner = PairwiseRunner(judge=judge, parallel=True)

    results = await runner.compare(
        AgentEvalTarget(baseline_agent),
        AgentEvalTarget(candidate_agent),
        tasks,
    )

    summary = compare_pairwise(results, "baseline", "candidate")
    print(summary)


asyncio.run(main(baseline_agent, candidate_agent, judge_client))
```

Pairwise output gives:

- `winner`: `"a"`, `"b"`, or `"tie"`
- `margin`: `0.0` to `1.0`
- `reasoning`
- both compared trajectories

## Pattern: Eval Dataset Integration (Dataset + Eval)

Use this when tasks come from dataset files rather than hand-authored `EvalTask` lists.

```python
from pathlib import Path

from agentbyte.dataset import LocalDatasetConfig, load_dataset
from agentbyte.eval import (
    EvalDatasetRecord,
    EvalRunner,
    ExactMatchJudge,
    save_eval_dataset,
    tasks_from_eval_dataset,
)

records = [
    EvalDatasetRecord(
        id="sql_1",
        scenario="No files selected",
        task="What is contract status for CW386232?",
        expected_response="Published.",
        eval_criteria=["accuracy", "no hallucination"],
        context={
            "reference_sql": "SELECT DISTINCT contract_status FROM contracts WHERE project_id='CW386232'",
            "reference_table": "contracts",
            "context_id": "CW386232",
        },
    )
]

save_eval_dataset(records, root=Path("/data"), dataset_id="asmd/sql_eval")

dataset = load_dataset("asmd/sql_eval", LocalDatasetConfig(local_root=Path("/data")))
tasks = tasks_from_eval_dataset(dataset)

# tasks[0].metadata includes:
# - "_criteria" from eval_criteria
# - "scenario"
# - flattened context keys like "reference_sql"

runner = EvalRunner(judge=ExactMatchJudge(), parallel=True)
# If evaluate(criteria=...) is omitted, runner uses task.metadata["_criteria"].
```

## Source-Scoped Extraction For Multi-Agent Evaluation

When evaluating orchestrators, the last message may belong to a reviewer or another downstream agent. Use `source_filter` to target a specific agent inside the flat trajectory.

```python
from agentbyte.eval import ExactMatchJudge, keyword_check

# Score only the writer's last assistant response
judge = ExactMatchJudge(
    answer_strategy="last_assistant",
    source_filter="writer",
)

# Check only reviewer comments
check = keyword_check(
    "approved",
    source_filter="reviewer",
    answer_strategy="last_assistant",
)
```

Rules:

- `source_filter` matches `message.source` exactly and case-sensitively
- no match returns an empty extracted answer
- `all_assistant` plus `source_filter` is the right pattern for "all writer messages" or "all reviewer comments"

## Answer Strategy Defaults

- Most pointwise judges default to `answer_strategy="last_non_empty"`
- `PairwiseJudge` defaults to `answer_strategy="last_assistant"`
- `keyword_check()` defaults to `answer_strategy="last_assistant"`

Choose explicitly when working with tool-heavy or orchestrated trajectories:

- `last_assistant`: best default for agent/orchestrator answer quality
- `last_non_empty`: useful when the last content may be the true answer
- `last_content`: use when non-assistant content should be scored directly
- `all_assistant`: use when all turns from the selected assistant(s) matter

## Recommended Construction Order

1. Define `EvalTask` objects.
2. Pick a target wrapper (`ModelEvalTarget`, `AgentEvalTarget`, `OrchestratorEvalTarget`).
3. Pick the judging mode:
   - pointwise judge
   - local checks/report
   - pairwise judge
4. Add `source_filter` if the target is multi-agent and only one agent's messages should count.
5. Prefer focused tests around extraction semantics, failure paths, and expected tool use.

## Guardrails

- Do not score raw tool output by accident in orchestrator flows; prefer `last_assistant` when evaluating agent-produced answers.
- Do not split orchestrator trajectories by source into separate pseudo-trajectories; use `source_filter` unless you are explicitly designing a stronger decomposition model.
- Do not use `PairwiseJudge.score()`; pairwise judges must be called through `compare()`.
- Prefer `keyword_check(answer_strategy="last_assistant")` when validating natural-language responses.
- Keep `EvalTask.input` string-based for the current public API; richer eval fixtures are deferred.
- Do not rely on malformed eval datasets: `tasks_from_eval_dataset()` requires `id` and `task` columns and raises `ValueError` when they are missing.
