> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronicle-labs.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluations
> Run agents against a task suite, wait for it to finish, and read the scores.
An evaluation runs each agent version against each task in a suite, in a fresh
copy of the world, and records a score per scorer. Each agent × task pair is a
**trial**. In the dashboard, evaluations are under **Backtests**.
Use a [configured SDK client](/api-reference/languages/overview#use-the-guide-examples).
The examples share that client and the IDs returned by earlier steps.
## Methods
| Method | HTTP | Does |
| ----------------------------------------------------------------- | ----------------------------------------- | --------------------------------------- |
| `evaluations.create(task_suite_id, agents, idempotency_key, ...)` | `POST /v1/backtests/jobs` | Start a run. |
| `evaluations.retrieve(run_id)` | `GET /v1/backtests/jobs/{jobId}` | Read status and progress. |
| `evaluations.wait(run_id, timeout)` | polls the above | Block until the run finishes. |
| `evaluations.stream(run_id)` | `GET .../stream` | Iterate progress events as they happen. |
| `evaluations.list(...)` | `GET /v1/backtests/jobs` | List runs. |
| `evaluations.cancel(run_id)` | `POST .../cancel` | Stop queued trials. |
| `evaluations.trials.list(run_id)` | `GET .../trials` | Every trial with its scores. |
| `evaluations.trials.retrieve(run_id, trial_id)` | `GET .../trials/{trialId}` | One trial with its steps and artifacts. |
| `evaluations.results(run_id)` | `GET /v1/task-suites/{suiteId}/eval-runs` | Pass/fail per task. |
## Start a run
You need a task suite ID and one or more agent versions. For a Worldsmith
world, the suite ID is `world.evaluation.task_suite_id`; wait for
`world.evaluation.status` to be `ready` first. Follow [Worlds](/api-reference/worldsmith)
to obtain `world`, and [connect the named agent version](/platform/agents)
before submitting. Registration alone does not configure a run command.
```python Python theme={null}
if world.evaluation is None or world.evaluation.status != "ready":
raise RuntimeError("The world's task setup is not ready")
suite_id = world.evaluation.task_suite_id
run = client.evaluations.create(
name="Invoice triage",
task_suite_id=suite_id,
agents=["billing-agent@1.0.0"],
concurrency=1,
idempotency_key="run-1",
)
print(run.id, run.status)
```
```typescript TypeScript theme={null}
if (world.evaluation?.status !== "ready") throw new Error("The world's task setup is not ready");
const suiteId = world.evaluation.taskSuiteId;
let run = await client.evaluations.create({
name: "Invoice triage",
taskSuiteId: suiteId,
agents: ["billing-agent@1.0.0"],
concurrency: 1,
idempotencyKey: "run-1",
});
console.log(run.id, run.status);
```
```javascript JavaScript theme={null}
if (world.evaluation?.status !== "ready") throw new Error("The world's task setup is not ready");
const suiteId = world.evaluation.taskSuiteId;
let run = await client.evaluations.create({
name: "Invoice triage",
taskSuiteId: suiteId,
agents: ["billing-agent@1.0.0"],
concurrency: 1,
idempotencyKey: "run-1",
});
console.log(run.id, run.status);
```
```go Go theme={null}
if world.Evaluation == nil || world.Evaluation.Status != "ready" { return fmt.Errorf("the world's task setup is not ready") }
suiteID := world.Evaluation.TaskSuiteID
run, err := client.Evaluations.Create(ctx, chronicle.EvaluationCreateParams{
Name: "Invoice triage",
TaskSuiteID: suiteID,
Agents: []string{"billing-agent@1.0.0"},
Concurrency: 1,
IdempotencyKey: "run-1",
})
if err != nil {
return err
}
fmt.Println(run.ID, run.Status)
```
```rust Rust theme={null}
let evaluation = world.evaluation.as_ref().filter(|e| e.status == "ready")
.ok_or("The world's task setup is not ready")?;
let suite_id = evaluation.task_suite_id.clone();
let run = client.evaluations().create(EvaluationCreateParams {
name: Some("Invoice triage".into()),
task_suite_id: suite_id.clone(),
agents: vec!["billing-agent@1.0.0".into()],
concurrency: Some(1),
idempotency_key: "run-1".into(),
..Default::default()
}).await?;
println!("{} {:?}", run.id, run.status);
```
| Parameter | Meaning |
| ------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `task_suite_id` | The suite to run. Its current tasks and scorers are frozen into a version at launch. |
| `agents` | One or more `name@version` references. Versions must be `current` or `stable` and have a run command. |
| `idempotency_key` | Retry with the same key and inputs to get the same run back. A new key starts a new run. |
| `name` | Shown in the dashboard. Defaults to the suite name. |
| `mode` | `suite` (default), `compare`, `regression`, or `replay`. `compare` treats the first agent as the baseline. |
| `concurrency` | Trials running at once, 1 to 32. Default 4. |
| `tasks` | Task IDs to run. Omit to run the whole suite. |
| `scorers` | Extra scorer IDs applied to every trial, on top of each task's own. |
| `environment_version_id` | Fallback world for tasks that don't specify one. |
Limits: 8 agents and 200 trials per run.
## Compare two versions
For a subsequent comparison, register `billing-agent@1.1.0`, configure its run
command, then pass both versions and `mode="compare"`. This starts a new run
on the same suite using the `suite_id` obtained above.
```python Python theme={null}
run = client.evaluations.create(
name="Handoff prompt v2",
task_suite_id=suite_id,
agents=["billing-agent@1.0.0", "billing-agent@1.1.0"],
mode="compare",
idempotency_key="compare-1",
)
```
```typescript TypeScript theme={null}
run = await client.evaluations.create({
name: "Handoff prompt v2",
taskSuiteId: suiteId,
agents: ["billing-agent@1.0.0", "billing-agent@1.1.0"],
mode: "compare",
idempotencyKey: "compare-1",
});
```
```javascript JavaScript theme={null}
run = await client.evaluations.create({
name: "Handoff prompt v2",
taskSuiteId: suiteId,
agents: ["billing-agent@1.0.0", "billing-agent@1.1.0"],
mode: "compare",
idempotencyKey: "compare-1",
});
```
```go Go theme={null}
run, err = client.Evaluations.Create(ctx, chronicle.EvaluationCreateParams{
Name: "Handoff prompt v2",
TaskSuiteID: suiteID,
Agents: []string{"billing-agent@1.0.0", "billing-agent@1.1.0"},
Mode: "compare",
IdempotencyKey: "compare-1",
})
if err != nil {
return err
}
```
```rust Rust theme={null}
let run = client.evaluations().create(EvaluationCreateParams {
name: Some("Handoff prompt v2".into()),
task_suite_id: suite_id.clone(),
agents: vec!["billing-agent@1.0.0".into(), "billing-agent@1.1.0".into()],
mode: Some("compare".into()),
idempotency_key: "compare-1".into(),
..Default::default()
}).await?;
```
To run a subset, pass `tasks=[...]` with task IDs from `tasks.list(task_suite_id)`.
## Wait for it
`wait` polls until the run is `succeeded`, `failed`, or `cancelled`. The
timeout only stops waiting; the run continues on the server, and you can wait
again with the same ID.
```python Python theme={null}
run = client.evaluations.wait(run.id, timeout=600)
print(run.status, run.completed_trials, "/", run.total_trials)
```
```typescript TypeScript theme={null}
const finished = await client.evaluations.wait(run.id, { timeout: 600_000 });
console.log(finished.status, finished.completedTrials, "/", finished.totalTrials);
```
```javascript JavaScript theme={null}
const finished = await client.evaluations.wait(run.id, { timeout: 600_000 });
console.log(finished.status, finished.completedTrials, "/", finished.totalTrials);
```
```go Go theme={null}
waitCtx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
run, err = client.Evaluations.Wait(waitCtx, run.ID)
if err != nil {
return err
}
fmt.Println(run.Status, run.CompletedTrials, "/", run.TotalTrials)
```
```rust Rust theme={null}
let run = client.evaluations().wait(&run.id, std::time::Duration::from_secs(600)).await?;
println!("{:?} {}/{}", run.status, run.completed_trials, run.total_trials);
```
For live progress instead of polling, `stream` yields events as they happen:
`trial_started`, `trial_finished`, `run_finished`.
```python Python theme={null}
for event in client.evaluations.stream(run.id):
print(event.type, event.trial_id, event.status)
```
```typescript TypeScript theme={null}
for await (const event of client.evaluations.stream(run.id)) {
console.log(event.type, event.trialId, event.status);
}
```
```javascript JavaScript theme={null}
for await (const event of client.evaluations.stream(run.id)) {
console.log(event.type, event.trialId, event.status);
}
```
```go Go theme={null}
stream, err := client.Evaluations.Stream(ctx, run.ID)
if err != nil {
return err
}
defer stream.Close()
for stream.Next() {
event := stream.Event()
fmt.Println(event.Type, event.TrialID, event.Status)
}
if err := stream.Err(); err != nil { return err }
```
```rust Rust theme={null}
let mut stream = client.evaluations().stream(&run.id).await?;
while let Some(event) = stream.next().await {
let event = event?;
println!("{:?} {:?} {:?}", event.r#type, event.trial_id, event.status);
}
```
If the stream disconnects, call `retrieve` to catch up; the run's state on the
server is the source of truth.
## Read the results
[Review the instruction and scorers](/product-explorer#agent-task) alongside
the result. The task describes the intended work; each trial records the tool
actions and the evidence its scorers evaluated.
`results` gives you pass/fail per task. A task passes when every scorer with a
threshold met it in every trial. A run that `succeeded` can still have failing
tasks: succeeded means the trials ran, not that the agent passed.
```python Python theme={null}
results = client.evaluations.results(run.id)
print(results.pass_rate)
for task in results.task_results:
print(task.task_id, task.title, "passed" if task.passed else "failed", task.failed_scorers)
```
```typescript TypeScript theme={null}
const results = await client.evaluations.results(run.id);
console.log(results.passRate);
for (const task of results.taskResults) {
console.log(task.taskId, task.title, task.passed ? "passed" : "failed", task.failedScorers);
}
```
```javascript JavaScript theme={null}
const results = await client.evaluations.results(run.id);
console.log(results.passRate);
for (const task of results.taskResults) {
console.log(task.taskId, task.title, task.passed ? "passed" : "failed", task.failedScorers);
}
```
```go Go theme={null}
results, err := client.Evaluations.Results(ctx, run.ID)
if err != nil {
return err
}
fmt.Println(results.PassRate)
for _, task := range results.TaskResults {
fmt.Println(task.TaskID, task.Title, task.Passed, task.FailedScorers)
}
```
```rust Rust theme={null}
let results = client.evaluations().results(&run.id).await?;
println!("{}", results.pass_rate);
for task in &results.task_results {
println!("{} {} {} {:?}", task.task_id, task.title, task.passed, task.failed_scorers);
}
```
In a comparison, `task_results` covers both agents together. Use trials to see
each agent separately.
[Open the illustrative score image](/images/product/evaluation-score-example.png).
[Explore the scorer view](/product-explorer#scores): the interactive example
uses illustrative pass/fail values to show how an issue-state check can pass
while the exact handoff check fails. Your evaluation supplies the real values
through `trials` and `results`.
## Inspect trials
Each trial has the agent, the task, a status, and a list of `scores`, one per
scorer. A trial's detail adds the instruction it received, its `steps`
(setup, agent, tool calls, scoring), and any `artifacts` it produced.
```python Python theme={null}
page = client.evaluations.trials.list(run.id)
if not page.items:
raise RuntimeError(f"Evaluation {run.id} has no trials yet; inspect its status")
selected_trial_id = page.items[0].id
for trial in page.auto_paging_iter():
print(trial.id, trial.agent, trial.task_id, trial.status, trial.error)
for score in trial.scores:
print(score.name, score.score, score.threshold, score.passed)
detail = client.evaluations.trials.retrieve(run.id, selected_trial_id)
for step in detail.steps:
print(step.kind, step.name, step.status, step.duration_ms)
```
```typescript TypeScript theme={null}
const page = await client.evaluations.trials.list(run.id);
const selectedTrial = page.items[0];
if (!selectedTrial) throw new Error(`Evaluation ${run.id} has no trials yet; inspect its status`);
for await (const trial of page) {
console.log(trial.id, trial.agent, trial.taskId, trial.status, trial.error);
for (const score of trial.scores) console.log(score.name, score.score, score.threshold, score.passed);
}
const detail = await client.evaluations.trials.retrieve(run.id, selectedTrial.id);
for (const step of detail.steps) console.log(step.kind, step.name, step.status, step.durationMs);
```
```javascript JavaScript theme={null}
const page = await client.evaluations.trials.list(run.id);
const selectedTrial = page.items[0];
if (!selectedTrial) throw new Error(`Evaluation ${run.id} has no trials yet; inspect its status`);
for await (const trial of page) {
console.log(trial.id, trial.agent, trial.taskId, trial.status, trial.error);
for (const score of trial.scores) console.log(score.name, score.score, score.threshold, score.passed);
}
const detail = await client.evaluations.trials.retrieve(run.id, selectedTrial.id);
for (const step of detail.steps) console.log(step.kind, step.name, step.status, step.durationMs);
```
```go Go theme={null}
page, err := client.Evaluations.Trials.List(ctx, run.ID, chronicle.PageParams{Limit: 100})
if err != nil { return err }
if len(page.Items) == 0 { return fmt.Errorf("evaluation %s has no trials yet; inspect its status", run.ID) }
selectedTrialID := page.Items[0].ID
for {
for _, trial := range page.Items {
fmt.Println(trial.ID, trial.Agent, trial.TaskID, trial.Status, trial.Error)
for _, score := range trial.Scores { fmt.Println(score.Name, score.Score, score.Threshold, score.Passed) }
}
if !page.HasNextPage() { break }
page, err = page.GetNextPage(ctx)
if err != nil { return err }
}
detail, err := client.Evaluations.Trials.Retrieve(ctx, run.ID, selectedTrialID)
if err != nil { return err }
for _, step := range detail.Steps { fmt.Println(step.Kind, step.Name, step.Status, step.DurationMs) }
```
```rust Rust theme={null}
let mut page = client.evaluations().trials().list(&run.id, PageParams { limit: Some(100), ..Default::default() }).await?;
let selected_trial_id = page.items.first().ok_or("No trials yet; inspect the evaluation status")?.id.clone();
loop {
for trial in &page.items {
println!("{} {} {} {:?} {:?}", trial.id, trial.agent, trial.task_id, trial.status, trial.error);
for score in &trial.scores { println!("{} {:?} {:?} {}", score.name, score.score, score.threshold, score.passed); }
}
if !page.has_next_page() { break; }
page = page.get_next_page().await?;
}
let detail = client.evaluations().trials().retrieve(&run.id, &selected_trial_id).await?;
for step in &detail.steps { println!("{:?} {} {:?} {:?}", step.kind, step.name, step.status, step.duration_ms); }
```
A score of `None`/`null` means the scorer didn't produce one, usually because
the trial errored. Read `trial.error` before treating it as a zero.
## Use results in CI
Continue with the `run` returned by [`evaluations.create`](#start-a-run). The
gate below waits for completion, rejects failed/cancelled runs and empty results,
and checks every task's `passed` value. In a comparison, that covers both agents.
Use it in your SDK program with the configured client; let errors produce a
nonzero process exit. In Go, propagate the returned error to `main` and exit
there, as in the [Go example](/api-reference/languages/go#run-an-evaluation).
```python Python theme={null}
run_id = run.id
finished = client.evaluations.wait(run_id, timeout=600)
if finished.status != "succeeded":
raise SystemExit(f"Evaluation {run_id}: {finished.status}")
results = client.evaluations.results(run_id)
if not results.task_results or any(not task.passed for task in results.task_results):
raise SystemExit("Evaluation did not pass every task")
print("Every task passed")
```
```typescript TypeScript theme={null}
const runId = run.id;
const completed = await client.evaluations.wait(runId, { timeout: 600_000 });
if (completed.status !== "succeeded") throw new Error(`Evaluation ${runId}: ${completed.status}`);
const outcome = await client.evaluations.results(runId);
if (!outcome.taskResults.length || outcome.taskResults.some((task) => !task.passed)) {
throw new Error("Evaluation did not pass every task");
}
console.log("Every task passed");
```
```javascript JavaScript theme={null}
const runId = run.id;
const completed = await client.evaluations.wait(runId, { timeout: 600_000 });
if (completed.status !== "succeeded") throw new Error(`Evaluation ${runId}: ${completed.status}`);
const outcome = await client.evaluations.results(runId);
if (!outcome.taskResults.length || outcome.taskResults.some((task) => !task.passed)) {
throw new Error("Evaluation did not pass every task");
}
console.log("Every task passed");
```
```go Go theme={null}
runID := run.ID
ciCtx, cancelCI := context.WithTimeout(ctx, 10*time.Minute)
defer cancelCI()
completed, err := client.Evaluations.Wait(ciCtx, runID)
if err != nil { return err }
if completed.Status != "succeeded" { return fmt.Errorf("evaluation %s: %s", runID, completed.Status) }
outcome, err := client.Evaluations.Results(ciCtx, runID)
if err != nil { return err }
if len(outcome.TaskResults) == 0 { return fmt.Errorf("evaluation returned no task results") }
for _, task := range outcome.TaskResults {
if !task.Passed { return fmt.Errorf("task %s failed: %v", task.TaskID, task.FailedScorers) }
}
fmt.Println("Every task passed")
```
```rust Rust theme={null}
let run_id = run.id.clone();
let completed = client.evaluations().wait(&run_id, Duration::from_secs(600)).await?;
if completed.status != "succeeded" { return Err(format!("evaluation {}: {}", run_id, completed.status).into()); }
let outcome = client.evaluations().results(&run_id).await?;
if outcome.task_results.is_empty() || outcome.task_results.iter().any(|task| !task.passed) {
return Err("Evaluation did not pass every task".into());
}
println!("Every task passed");
```
## Cancel
```python Python theme={null}
client.evaluations.cancel(run.id)
```
```typescript TypeScript theme={null}
await client.evaluations.cancel(run.id);
```
```javascript JavaScript theme={null}
await client.evaluations.cancel(run.id);
```
```go Go theme={null}
_, err = client.Evaluations.Cancel(ctx, run.ID)
if err != nil { return err }
```
```rust Rust theme={null}
client.evaluations().cancel(&run.id).await?;
```
Queued trials don't start. Finished trials keep their results. The run ends
as `cancelled`.
## Errors
| Status | Meaning |
| ------ | ---------------------------------------------------------- |
| `400` | Unknown task ID, agent reference, or environment version. |
| `403` | Your organization doesn't have sandbox access. |
| `409` | Idempotency key reused with a different request. |
| `429` | Over the concurrency or trial limit. |
| `503` | The evaluation runtime isn't available on this deployment. |