---
name: agentbyte-eval
description: Guide for evaluating Agentbyte models, agents, and orchestrators with pointwise judges, local checks, reports, and pairwise comparison. Use this when implementing eval suites or comparing configurations.
---

Use this skill when a user wants to build evaluation suites for AgentByte agents, orchestrators, or raw model targets.

## Inputs To Confirm

- Whether the user needs pointwise scoring, local pass/fail checks, or pairwise comparison
- Which runtime is being evaluated: raw model, agent, or orchestrator
- Whether the evaluation target is the final answer or a specific agent's messages inside a multi-agent trajectory
- Whether the result should be a numeric score, a CI-style report, or a win/loss summary

## Core Eval Shapes

The eval package is built around four primary models:

- `EvalTask`: what to run
- `EvalTrajectory`: what happened at runtime
- `EvalScore`: numeric pointwise judgment
- `PairwiseResult`: relative A-vs-B judgment

Use the public package root:

```python
from agentbyte.eval import (
    AgentEvalTarget,
    EvalDatasetRecord,
    EvalRunner,
    EvalTask,
    ExactMatchJudge,
    PairwiseJudge,
    PairwiseRunner,
    save_eval_dataset,
    tasks_from_eval_dataset,
)
```

## Pattern: Pointwise Evaluation

Use pointwise evaluation when the question is "how good is this response?"

```python
import asyncio

from agentbyte.eval import AgentEvalTarget, EvalRunner, EvalTask, ExactMatchJudge


async def main(agent):
    tasks = [
        EvalTask(
            name="capital",
            input="What is the capital of France?",
            expected_output="Paris",
        ),
        EvalTask(
            name="addition",
            input="What is 17 + 25?",
            expected_output="42",
        ),
    ]

    runner = EvalRunner(judge=ExactMatchJudge(), parallel=True)
    scores = await runner.evaluate(AgentEvalTarget(agent), tasks)

    for score in scores:
        print(score.overall, score.reasoning)


asyncio.run(main(agent))
```

### Pointwise criteria precedence

`EvalRunner.evaluate()` resolves criteria in this order:

1. `criteria=` argument passed to `evaluate()`
2. `EvalTask.metadata["_criteria"]` (set manually or loaded from eval datasets)
3. Judge-level default behavior

This allows per-task dataset criteria to flow into scoring without custom wiring.

### Available pointwise targets

- `ModelEvalTarget`: wrap a raw chat-completion client
- `AgentEvalTarget`: wrap an `Agent`
- `OrchestratorEvalTarget`: wrap a `BaseOrchestrator`

### Available pointwise judges

- `ExactMatchJudge`
- `ContainsJudge`
- `FuzzyMatchJudge`
- `LLMEvalJudge`
- `CompositeJudge`

## Pattern: Local Checks And Reports

Use local checks when the question is "did the run satisfy required behavior?"

```python
import asyncio

from agentbyte.eval import (
    AgentEvalTarget,
    EvalRunner,
    EvalTask,
    ExactMatchJudge,
    LocalEvaluator,
    keyword_check,
    threshold_gate,
)


async def main(agent):
    task = EvalTask(
        name="weather",
        input="What's the weather in Paris?",
        expected_output="sunny",
    )

    runner = EvalRunner(judge=ExactMatchJudge(), parallel=False)
    scores = await runner.evaluate(AgentEvalTarget(agent), [task])
    trajectories = [score.trajectory for score in scores if score.trajectory is not None]

    report = await LocalEvaluator(
        threshold_gate(ExactMatchJudge(), threshold=7.0),
        keyword_check("sunny"),
    ).evaluate(trajectories)

    report.raise_for_status()


asyncio.run(main(agent))
```

### Built-in checks

- `tool_calls_present`
- `tool_call_args_match`
- `keyword_check`
- `@evaluator`
- `threshold_gate`

## Pattern: Pairwise Evaluation

Use pairwise evaluation when the question is "which of these two configurations is better?"

```python
import asyncio

from agentbyte.eval import (
    AgentEvalTarget,
    EvalTask,
    PairwiseJudge,
    PairwiseRunner,
    compare_pairwise,
)


async def main(baseline_agent, candidate_agent, judge_client):
    tasks = [
        EvalTask(name="capital", input="What is the capital of France?"),
        EvalTask(name="rewrite", input="Rewrite this note in a clearer tone."),
    ]

    judge = PairwiseJudge(client=judge_client, criteria=["accuracy", "helpfulness"])
    runner = PairwiseRunner(judge=judge, parallel=True)

    results = await runner.compare(
        AgentEvalTarget(baseline_agent),
        AgentEvalTarget(candidate_agent),
        tasks,
    )

    summary = compare_pairwise(results, "baseline", "candidate")
    print(summary)


asyncio.run(main(baseline_agent, candidate_agent, judge_client))
```

Pairwise output gives:

- `winner`: `"a"`, `"b"`, or `"tie"`
- `margin`: `0.0` to `1.0`
- `reasoning`
- both compared trajectories

## Pattern: Eval Dataset Integration (Dataset + Eval)

Use this when tasks come from dataset files rather than hand-authored `EvalTask` lists.

```python
from pathlib import Path

from agentbyte.dataset import LocalDatasetConfig, load_dataset
from agentbyte.eval import (
    EvalDatasetRecord,
    EvalRunner,
    ExactMatchJudge,
    save_eval_dataset,
    tasks_from_eval_dataset,
)

records = [
    EvalDatasetRecord(
        id="sql_1",
        scenario="No files selected",
        task="What is contract status for CW386232?",
        expected_response="Published.",
        eval_criteria=["accuracy", "no hallucination"],
        context={
            "reference_sql": "SELECT DISTINCT contract_status FROM contracts WHERE project_id='CW386232'",
            "reference_table": "contracts",
            "context_id": "CW386232",
        },
    )
]

save_eval_dataset(records, root=Path("/data"), dataset_id="asmd/sql_eval")

dataset = load_dataset("asmd/sql_eval", LocalDatasetConfig(local_root=Path("/data")))
tasks = tasks_from_eval_dataset(dataset)

# tasks[0].metadata includes:
# - "_criteria" from eval_criteria
# - "scenario"
# - flattened context keys like "reference_sql"

runner = EvalRunner(judge=ExactMatchJudge(), parallel=True)
# If evaluate(criteria=...) is omitted, runner uses task.metadata["_criteria"].
```

### Sharing Eval Datasets Through Dataset Publishing

If an eval dataset is authored locally and then needs to be reused by other
users or environments, publish the dataset artifact after saving it:

```python
from pathlib import Path

from agentbyte.dataset import publish_dataset
from agentbyte.dataset.config import LocalDatasetConfig
from agentbyte.dataset.publish_config import S3DatasetPublishConfig

publish_result = publish_dataset(
    "asmd/sql_eval",
    local_config=LocalDatasetConfig(local_root=Path("/data")),
    publish_config=S3DatasetPublishConfig(
        s3_bucket="my-bucket",
        s3_prefix="eval-datasets",
    ),
)

print(publish_result.destination_uri)
```

That keeps the eval dataset in the same canonical dataset layout, so other
users can later load it with `load_dataset(..., S3DatasetConfig(...))` and feed
it into `tasks_from_eval_dataset()`.

### Sharing Eval Datasets Through Dataset Publishing

If an eval dataset is authored locally and then needs to be reused by other
users or environments, publish the dataset artifact after saving it:

```python
from pathlib import Path

from agentbyte.dataset import publish_dataset
from agentbyte.dataset.config import LocalDatasetConfig
from agentbyte.dataset.publish_config import S3DatasetPublishConfig

publish_result = publish_dataset(
    "asmd/sql_eval",
    local_config=LocalDatasetConfig(local_root=Path("/data")),
    publish_config=S3DatasetPublishConfig(
        s3_bucket="my-bucket",
        s3_prefix="eval-datasets",
    ),
)

print(publish_result.destination_uri)
```

That keeps the eval dataset in the same canonical dataset layout, so other
users can later load it with `load_dataset(..., S3DatasetConfig(...))` and feed
it into `tasks_from_eval_dataset()`.

## Source-Scoped Extraction For Multi-Agent Evaluation

When evaluating orchestrators, the last message may belong to a reviewer or another downstream agent. Use `source_filter` to target a specific agent inside the flat trajectory.

```python
from agentbyte.eval import ExactMatchJudge, keyword_check

# Score only the writer's last assistant response
judge = ExactMatchJudge(
    answer_strategy="last_assistant",
    source_filter="writer",
)

# Check only reviewer comments
check = keyword_check(
    "approved",
    source_filter="reviewer",
    answer_strategy="last_assistant",
)
```

Rules:

- `source_filter` matches `message.source` exactly and case-sensitively
- no match returns an empty extracted answer
- `all_assistant` plus `source_filter` is the right pattern for "all writer messages" or "all reviewer comments"

## Answer Strategy Defaults

- Most pointwise judges default to `answer_strategy="last_non_empty"`
- `PairwiseJudge` defaults to `answer_strategy="last_assistant"`
- `keyword_check()` defaults to `answer_strategy="last_assistant"`

Choose explicitly when working with tool-heavy or orchestrated trajectories:

- `last_assistant`: best default for agent/orchestrator answer quality
- `last_non_empty`: useful when the last content may be the true answer
- `last_content`: use when non-assistant content should be scored directly
- `all_assistant`: use when all turns from the selected assistant(s) matter

## Recommended Construction Order

1. Define `EvalTask` objects.
2. Pick a target wrapper (`ModelEvalTarget`, `AgentEvalTarget`, `OrchestratorEvalTarget`).
3. Pick the judging mode:
   - pointwise judge
   - local checks/report
   - pairwise judge
4. Add `source_filter` if the target is multi-agent and only one agent's messages should count.
5. Prefer focused tests around extraction semantics, failure paths, and expected tool use.

## Guardrails

- Do not score raw tool output by accident in orchestrator flows; prefer `last_assistant` when evaluating agent-produced answers.
- Do not split orchestrator trajectories by source into separate pseudo-trajectories; use `source_filter` unless you are explicitly designing a stronger decomposition model.
- Do not use `PairwiseJudge.score()`; pairwise judges must be called through `compare()`.
- Prefer `keyword_check(answer_strategy="last_assistant")` when validating natural-language responses.
- Keep `EvalTask.input` string-based for the current public API; richer eval fixtures are deferred.
- Do not rely on malformed eval datasets: `tasks_from_eval_dataset()` requires `id` and `task` columns and raises `ValueError` when they are missing.
