Evaluations

An evaluation runs each agent version against each task in a suite, in a fresh copy of the world, and records a score per scorer. Each agent × task pair is a trial. In the dashboard, evaluations are under Backtests. Use a configured SDK client. The examples share that client and the IDs returned by earlier steps.

Methods

Method	HTTP	Does
`evaluations.create(task_suite_id, agents, idempotency_key, ...)`	`POST /v1/backtests/jobs`	Start a run.
`evaluations.retrieve(run_id)`	`GET /v1/backtests/jobs/{jobId}`	Read status and progress.
`evaluations.wait(run_id, timeout)`	polls the above	Block until the run finishes.
`evaluations.stream(run_id)`	`GET .../stream`	Iterate progress events as they happen.
`evaluations.list(...)`	`GET /v1/backtests/jobs`	List runs.
`evaluations.cancel(run_id)`	`POST .../cancel`	Stop queued trials.
`evaluations.trials.list(run_id)`	`GET .../trials`	Every trial with its scores.
`evaluations.trials.retrieve(run_id, trial_id)`	`GET .../trials/{trialId}`	One trial with its steps and artifacts.
`evaluations.results(run_id)`	`GET /v1/task-suites/{suiteId}/eval-runs`	Pass/fail per task.

Start a run

You need a task suite ID and one or more agent versions. For a Worldsmith world, the suite ID is world.evaluation.task_suite_id; wait for world.evaluation.status to be ready first. Follow Worlds to obtain world, and connect the named agent version before submitting. Registration alone does not configure a run command.

Python

if world.evaluation is None or world.evaluation.status != "ready":
    raise RuntimeError("The world's task setup is not ready")
suite_id = world.evaluation.task_suite_id
run = client.evaluations.create(
    name="Invoice triage",
    task_suite_id=suite_id,
    agents=["billing-agent@1.0.0"],
    concurrency=1,
    idempotency_key="run-1",
)
print(run.id, run.status)

if (world.evaluation?.status !== "ready") throw new Error("The world's task setup is not ready");
const suiteId = world.evaluation.taskSuiteId;
let run = await client.evaluations.create({
  name: "Invoice triage",
  taskSuiteId: suiteId,
  agents: ["billing-agent@1.0.0"],
  concurrency: 1,
  idempotencyKey: "run-1",
});
console.log(run.id, run.status);

if (world.evaluation?.status !== "ready") throw new Error("The world's task setup is not ready");
const suiteId = world.evaluation.taskSuiteId;
let run = await client.evaluations.create({
  name: "Invoice triage",
  taskSuiteId: suiteId,
  agents: ["billing-agent@1.0.0"],
  concurrency: 1,
  idempotencyKey: "run-1",
});
console.log(run.id, run.status);

if world.Evaluation == nil || world.Evaluation.Status != "ready" { return fmt.Errorf("the world's task setup is not ready") }
suiteID := world.Evaluation.TaskSuiteID
run, err := client.Evaluations.Create(ctx, chronicle.EvaluationCreateParams{
    Name:           "Invoice triage",
    TaskSuiteID:    suiteID,
    Agents:         []string{"billing-agent@1.0.0"},
    Concurrency:    1,
    IdempotencyKey: "run-1",
})
if err != nil {
    return err
}
fmt.Println(run.ID, run.Status)

let evaluation = world.evaluation.as_ref().filter(|e| e.status == "ready")
    .ok_or("The world's task setup is not ready")?;
let suite_id = evaluation.task_suite_id.clone();
let run = client.evaluations().create(EvaluationCreateParams {
    name: Some("Invoice triage".into()),
    task_suite_id: suite_id.clone(),
    agents: vec!["billing-agent@1.0.0".into()],
    concurrency: Some(1),
    idempotency_key: "run-1".into(),
    ..Default::default()
}).await?;
println!("{} {:?}", run.id, run.status);

Parameter	Meaning
`task_suite_id`	The suite to run. Its current tasks and scorers are frozen into a version at launch.
`agents`	One or more `name@version` references. Versions must be `current` or `stable` and have a run command.
`idempotency_key`	Retry with the same key and inputs to get the same run back. A new key starts a new run.
`name`	Shown in the dashboard. Defaults to the suite name.
`mode`	`suite` (default), `compare`, `regression`, or `replay`. `compare` treats the first agent as the baseline.
`concurrency`	Trials running at once, 1 to 32. Default 4.
`tasks`	Task IDs to run. Omit to run the whole suite.
`scorers`	Extra scorer IDs applied to every trial, on top of each task’s own.
`environment_version_id`	Fallback world for tasks that don’t specify one.

Limits: 8 agents and 200 trials per run.

Compare two versions

For a subsequent comparison, register billing-agent@1.1.0, configure its run command, then pass both versions and mode="compare". This starts a new run on the same suite using the suite_id obtained above.

Python

run = client.evaluations.create(
    name="Handoff prompt v2",
    task_suite_id=suite_id,
    agents=["billing-agent@1.0.0", "billing-agent@1.1.0"],
    mode="compare",
    idempotency_key="compare-1",
)

run = await client.evaluations.create({
  name: "Handoff prompt v2",
  taskSuiteId: suiteId,
  agents: ["billing-agent@1.0.0", "billing-agent@1.1.0"],
  mode: "compare",
  idempotencyKey: "compare-1",
});

run = await client.evaluations.create({
  name: "Handoff prompt v2",
  taskSuiteId: suiteId,
  agents: ["billing-agent@1.0.0", "billing-agent@1.1.0"],
  mode: "compare",
  idempotencyKey: "compare-1",
});

run, err = client.Evaluations.Create(ctx, chronicle.EvaluationCreateParams{
    Name:           "Handoff prompt v2",
    TaskSuiteID:    suiteID,
    Agents:         []string{"billing-agent@1.0.0", "billing-agent@1.1.0"},
    Mode:           "compare",
    IdempotencyKey: "compare-1",
})
if err != nil {
    return err
}

let run = client.evaluations().create(EvaluationCreateParams {
    name: Some("Handoff prompt v2".into()),
    task_suite_id: suite_id.clone(),
    agents: vec!["billing-agent@1.0.0".into(), "billing-agent@1.1.0".into()],
    mode: Some("compare".into()),
    idempotency_key: "compare-1".into(),
    ..Default::default()
}).await?;

To run a subset, pass tasks=[...] with task IDs from tasks.list(task_suite_id).

Wait for it

wait polls until the run is succeeded, failed, or cancelled. The timeout only stops waiting; the run continues on the server, and you can wait again with the same ID.

Python

run = client.evaluations.wait(run.id, timeout=600)
print(run.status, run.completed_trials, "/", run.total_trials)

waitCtx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
run, err = client.Evaluations.Wait(waitCtx, run.ID)
if err != nil {
    return err
}
fmt.Println(run.Status, run.CompletedTrials, "/", run.TotalTrials)

For live progress instead of polling, stream yields events as they happen: trial_started, trial_finished, run_finished.

Python

for event in client.evaluations.stream(run.id):
    print(event.type, event.trial_id, event.status)

stream, err := client.Evaluations.Stream(ctx, run.ID)
if err != nil {
    return err
}
defer stream.Close()
for stream.Next() {
    event := stream.Event()
    fmt.Println(event.Type, event.TrialID, event.Status)
}
if err := stream.Err(); err != nil { return err }

let mut stream = client.evaluations().stream(&run.id).await?;
while let Some(event) = stream.next().await {
    let event = event?;
    println!("{:?} {:?} {:?}", event.r#type, event.trial_id, event.status);
}

If the stream disconnects, call retrieve to catch up; the run’s state on the server is the source of truth.

Read the results

Review the instruction and scorers alongside the result. The task describes the intended work; each trial records the tool actions and the evidence its scorers evaluated. results gives you pass/fail per task. A task passes when every scorer with a threshold met it in every trial. A run that succeeded can still have failing tasks: succeeded means the trials ran, not that the agent passed.

Python

results = client.evaluations.results(run.id)
print(results.pass_rate)
for task in results.task_results:
    print(task.task_id, task.title, "passed" if task.passed else "failed", task.failed_scorers)

const results = await client.evaluations.results(run.id);
console.log(results.passRate);
for (const task of results.taskResults) {
  console.log(task.taskId, task.title, task.passed ? "passed" : "failed", task.failedScorers);
}

const results = await client.evaluations.results(run.id);
console.log(results.passRate);
for (const task of results.taskResults) {
  console.log(task.taskId, task.title, task.passed ? "passed" : "failed", task.failedScorers);
}

results, err := client.Evaluations.Results(ctx, run.ID)
if err != nil {
    return err
}
fmt.Println(results.PassRate)
for _, task := range results.TaskResults {
    fmt.Println(task.TaskID, task.Title, task.Passed, task.FailedScorers)
}

let results = client.evaluations().results(&run.id).await?;
println!("{}", results.pass_rate);
for task in &results.task_results {
    println!("{} {} {} {:?}", task.task_id, task.title, task.passed, task.failed_scorers);
}

In a comparison, task_results covers both agents together. Use trials to see each agent separately.

Product scorer rows with an illustrative 1.0 pass for the Linear update and 0.0 fail for the exact Slack handoff — Illustrative scorer results, not an executed evaluation: the issue-state check passes and the required handoff text does not match.

Open the illustrative score image. Explore the scorer view: the interactive example uses illustrative pass/fail values to show how an issue-state check can pass while the exact handoff check fails. Your evaluation supplies the real values through trials and results.

Inspect trials

Each trial has the agent, the task, a status, and a list of scores, one per scorer. A trial’s detail adds the instruction it received, its steps (setup, agent, tool calls, scoring), and any artifacts it produced.

Python

page = client.evaluations.trials.list(run.id)
if not page.items:
    raise RuntimeError(f"Evaluation {run.id} has no trials yet; inspect its status")
selected_trial_id = page.items[0].id
for trial in page.auto_paging_iter():
    print(trial.id, trial.agent, trial.task_id, trial.status, trial.error)
    for score in trial.scores:
        print(score.name, score.score, score.threshold, score.passed)

detail = client.evaluations.trials.retrieve(run.id, selected_trial_id)
for step in detail.steps:
    print(step.kind, step.name, step.status, step.duration_ms)

const page = await client.evaluations.trials.list(run.id);
const selectedTrial = page.items[0];
if (!selectedTrial) throw new Error(`Evaluation ${run.id} has no trials yet; inspect its status`);
for await (const trial of page) {
  console.log(trial.id, trial.agent, trial.taskId, trial.status, trial.error);
  for (const score of trial.scores) console.log(score.name, score.score, score.threshold, score.passed);
}
const detail = await client.evaluations.trials.retrieve(run.id, selectedTrial.id);
for (const step of detail.steps) console.log(step.kind, step.name, step.status, step.durationMs);

const page = await client.evaluations.trials.list(run.id);
const selectedTrial = page.items[0];
if (!selectedTrial) throw new Error(`Evaluation ${run.id} has no trials yet; inspect its status`);
for await (const trial of page) {
  console.log(trial.id, trial.agent, trial.taskId, trial.status, trial.error);
  for (const score of trial.scores) console.log(score.name, score.score, score.threshold, score.passed);
}
const detail = await client.evaluations.trials.retrieve(run.id, selectedTrial.id);
for (const step of detail.steps) console.log(step.kind, step.name, step.status, step.durationMs);

page, err := client.Evaluations.Trials.List(ctx, run.ID, chronicle.PageParams{Limit: 100})
if err != nil { return err }
if len(page.Items) == 0 { return fmt.Errorf("evaluation %s has no trials yet; inspect its status", run.ID) }
selectedTrialID := page.Items[0].ID
for {
    for _, trial := range page.Items {
        fmt.Println(trial.ID, trial.Agent, trial.TaskID, trial.Status, trial.Error)
        for _, score := range trial.Scores { fmt.Println(score.Name, score.Score, score.Threshold, score.Passed) }
    }
    if !page.HasNextPage() { break }
    page, err = page.GetNextPage(ctx)
    if err != nil { return err }
}
detail, err := client.Evaluations.Trials.Retrieve(ctx, run.ID, selectedTrialID)
if err != nil { return err }
for _, step := range detail.Steps { fmt.Println(step.Kind, step.Name, step.Status, step.DurationMs) }

let mut page = client.evaluations().trials().list(&run.id, PageParams { limit: Some(100), ..Default::default() }).await?;
let selected_trial_id = page.items.first().ok_or("No trials yet; inspect the evaluation status")?.id.clone();
loop {
    for trial in &page.items {
        println!("{} {} {} {:?} {:?}", trial.id, trial.agent, trial.task_id, trial.status, trial.error);
        for score in &trial.scores { println!("{} {:?} {:?} {}", score.name, score.score, score.threshold, score.passed); }
    }
    if !page.has_next_page() { break; }
    page = page.get_next_page().await?;
}
let detail = client.evaluations().trials().retrieve(&run.id, &selected_trial_id).await?;
for step in &detail.steps { println!("{:?} {} {:?} {:?}", step.kind, step.name, step.status, step.duration_ms); }

A score of None/null means the scorer didn’t produce one, usually because the trial errored. Read trial.error before treating it as a zero.

Use results in CI

Continue with the run returned by evaluations.create. The gate below waits for completion, rejects failed/cancelled runs and empty results, and checks every task’s passed value. In a comparison, that covers both agents. Use it in your SDK program with the configured client; let errors produce a nonzero process exit. In Go, propagate the returned error to main and exit there, as in the Go example.

Python

run_id = run.id
finished = client.evaluations.wait(run_id, timeout=600)
if finished.status != "succeeded":
    raise SystemExit(f"Evaluation {run_id}: {finished.status}")
results = client.evaluations.results(run_id)
if not results.task_results or any(not task.passed for task in results.task_results):
    raise SystemExit("Evaluation did not pass every task")
print("Every task passed")

const runId = run.id;
const completed = await client.evaluations.wait(runId, { timeout: 600_000 });
if (completed.status !== "succeeded") throw new Error(`Evaluation ${runId}: ${completed.status}`);
const outcome = await client.evaluations.results(runId);
if (!outcome.taskResults.length || outcome.taskResults.some((task) => !task.passed)) {
  throw new Error("Evaluation did not pass every task");
}
console.log("Every task passed");

const runId = run.id;
const completed = await client.evaluations.wait(runId, { timeout: 600_000 });
if (completed.status !== "succeeded") throw new Error(`Evaluation ${runId}: ${completed.status}`);
const outcome = await client.evaluations.results(runId);
if (!outcome.taskResults.length || outcome.taskResults.some((task) => !task.passed)) {
  throw new Error("Evaluation did not pass every task");
}
console.log("Every task passed");

runID := run.ID
ciCtx, cancelCI := context.WithTimeout(ctx, 10*time.Minute)
defer cancelCI()
completed, err := client.Evaluations.Wait(ciCtx, runID)
if err != nil { return err }
if completed.Status != "succeeded" { return fmt.Errorf("evaluation %s: %s", runID, completed.Status) }
outcome, err := client.Evaluations.Results(ciCtx, runID)
if err != nil { return err }
if len(outcome.TaskResults) == 0 { return fmt.Errorf("evaluation returned no task results") }
for _, task := range outcome.TaskResults {
    if !task.Passed { return fmt.Errorf("task %s failed: %v", task.TaskID, task.FailedScorers) }
}
fmt.Println("Every task passed")

let run_id = run.id.clone();
let completed = client.evaluations().wait(&run_id, Duration::from_secs(600)).await?;
if completed.status != "succeeded" { return Err(format!("evaluation {}: {}", run_id, completed.status).into()); }
let outcome = client.evaluations().results(&run_id).await?;
if outcome.task_results.is_empty() || outcome.task_results.iter().any(|task| !task.passed) {
    return Err("Evaluation did not pass every task".into());
}
println!("Every task passed");

Cancel

Python

client.evaluations.cancel(run.id)

Queued trials don’t start. Finished trials keep their results. The run ends as cancelled.

Errors

Status	Meaning
`400`	Unknown task ID, agent reference, or environment version.
`403`	Your organization doesn’t have sandbox access.
`409`	Idempotency key reused with a different request.
`429`	Over the concurrency or trial limit.
`503`	The evaluation runtime isn’t available on this deployment.

Get started

Guides

SDKs

Worldsmith endpoints

Methods

Start a run

Compare two versions

Wait for it

Read the results

Inspect trials

Use results in CI

Cancel

Errors

​Methods

​Start a run

​Compare two versions

​Wait for it

​Read the results

​Inspect trials

​Use results in CI

​Cancel

​Errors

Methods

Start a run

Compare two versions

Wait for it

Read the results

Inspect trials

Use results in CI

Cancel

Errors