> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronicle-labs.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Tasks and scorers
> Create a task suite, add tasks and scorers, and publish a frozen version to evaluate.
A **task suite** holds **tasks**. A task has an instruction, an expected
outcome, the world it starts in, and **scorers** that grade the result. Editing
a suite changes the working copy; publishing freezes a version.
If you launched a Worldsmith world, it already made a suite for you. Its ID is
`world.evaluation.task_suite_id`. This page is for building suites yourself or
adding to a generated one.
Use a [configured SDK client](/api-reference/languages/overview#use-the-guide-examples).
The examples share that client and the IDs returned by earlier steps.
## Methods
| Method | HTTP | Does |
| -------------------------------------------------------------- | ------------------------------------------------ | ------------------------------------ |
| `task_suites.create(name, ...)` | `POST /v1/task-suites` | Create a suite. |
| `task_suites.list()` | `GET /v1/task-suites` | List suites. |
| `task_suites.retrieve(id)`, `.update(id, ...)`, `.archive(id)` | `GET` `PATCH` `DELETE /v1/task-suites/{suiteId}` | Read or change a suite. |
| `task_suites.publish(suite_id, label)` | `POST /v1/task-suites/{suiteId}/versions` | Freeze a version. |
| `task_suites.versions.list(suite_id)` | `GET .../versions` | List frozen versions. |
| `tasks.create(task_suite_id, ...)` | `POST /v1/task-suites/{suiteId}/tasks` | Add a task. |
| `tasks.list(task_suite_id)` | `GET /v1/task-suites/{suiteId}/tasks` | List tasks. |
| `tasks.retrieve(...)`, `.update(...)`, `.delete(...)` | `GET` `PATCH` `DELETE .../tasks/{taskId}` | Read or change a task. |
| `tasks.set_scorers(task_suite_id, task_id, scorers)` | `PUT .../tasks/{taskId}/scorers` | Replace a task's scorers. |
| `tasks.from_trace(task_suite_id, trace_id)` | `POST /v1/task-suites/{suiteId}/traces` | Create a task from a recorded trace. |
| `scorers.create(name, kind, ...)` | `POST /v1/scorers` | Create a scorer. |
| `scorers.list()` | `GET /v1/scorers` | List the scorer library. |
| `scorers.retrieve(id)`, `.update(id, ...)`, `.archive(id)` | `GET` `PUT` `DELETE /v1/scorers/{scorerId}` | Read or change a scorer. |
| `scorers.test(...)` | `POST /v1/scorers/test` | Run a scorer on sample input. |
## Choose the source task
Start with the ready duplicate-invoice world from [Worlds](/api-reference/worldsmith).
This walkthrough copies its two mutation scorers into a new task, then adds a
judge for the agent's summary. It checks real Linear and Slack changes as well
as the final answer.
```python Python theme={null}
if world.evaluation is None or world.evaluation.status != "ready":
raise RuntimeError("The example world's task setup must be ready")
source_ref = next((t for t in world.evaluation.tasks
if t.title == "Triage the duplicate invoice incident"), None)
if source_ref is None:
raise RuntimeError("Choose the duplicate-invoice example world")
source_task = client.tasks.retrieve(world.evaluation.task_suite_id, source_ref.task_id)
```
```typescript TypeScript theme={null}
if (world.evaluation?.status !== "ready") throw new Error("The example world's task setup must be ready");
const sourceRef = world.evaluation.tasks.find((t) => t.title === "Triage the duplicate invoice incident");
if (!sourceRef) throw new Error("Choose the duplicate-invoice example world");
const sourceTask = await client.tasks.retrieve(world.evaluation.taskSuiteId, sourceRef.taskId);
```
```javascript JavaScript theme={null}
if (world.evaluation?.status !== "ready") throw new Error("The example world's task setup must be ready");
const sourceRef = world.evaluation.tasks.find((t) => t.title === "Triage the duplicate invoice incident");
if (!sourceRef) throw new Error("Choose the duplicate-invoice example world");
const sourceTask = await client.tasks.retrieve(world.evaluation.taskSuiteId, sourceRef.taskId);
```
```go Go theme={null}
if world.Evaluation == nil || world.Evaluation.Status != "ready" {
return fmt.Errorf("the example world's task setup must be ready")
}
sourceTaskID := ""
for _, task := range world.Evaluation.Tasks {
if task.Title == "Triage the duplicate invoice incident" { sourceTaskID = task.TaskID; break }
}
if sourceTaskID == "" { return fmt.Errorf("choose the duplicate-invoice example world") }
sourceTask, err := client.Tasks.Retrieve(ctx, world.Evaluation.TaskSuiteID, sourceTaskID)
if err != nil { return err }
```
```rust Rust theme={null}
let evaluation = world.evaluation.as_ref().filter(|e| e.status == "ready")
.ok_or("The example world's task setup must be ready")?;
let source_ref = evaluation.tasks.iter().find(|t| t.title == "Triage the duplicate invoice incident")
.ok_or("Choose the duplicate-invoice example world")?;
let source_task = client.tasks().retrieve(&evaluation.task_suite_id, &source_ref.task_id).await?;
```
[Explore this task and its scorers](/product-explorer#agent-task). Its existing
scorers check that ENG-2 moved to In Progress and that the exact handoff text
was posted in #erp-alerts. They do not independently verify that the Salesforce
case remained open; inspect that state in the trial or add a separate assertion
before using it as a strict acceptance criterion.
## Build a suite
```python Python theme={null}
suite = client.task_suites.create(
name="Invoice triage",
description="Regression tests for the billing agent.",
)
```
```typescript TypeScript theme={null}
const suite = await client.taskSuites.create({
name: "Invoice triage",
description: "Regression tests for the billing agent.",
});
```
```javascript JavaScript theme={null}
const suite = await client.taskSuites.create({
name: "Invoice triage",
description: "Regression tests for the billing agent.",
});
```
```go Go theme={null}
suite, err := client.TaskSuites.Create(ctx, chronicle.TaskSuiteCreateParams{
Name: "Invoice triage",
Description: "Regression tests for the billing agent.",
})
if err != nil {
return err
}
```
```rust Rust theme={null}
let suite = client.task_suites().create(TaskSuiteCreateParams {
name: "Invoice triage".into(),
description: Some("Regression tests for the billing agent.".into()),
..Default::default()
}).await?;
```
Scorers live in a library and can be attached to many tasks. Two kinds:
`llm-judge` (a prompt) and `code` (Python or TypeScript).
```python Python theme={null}
judge = client.scorers.create(
name="Handoff clarity",
kind="llm-judge",
prompt=(
"Read the agent's final summary. Choose 'complete' only if it names "
"ENG-2, explains the invoice problem, and gives a next action. Do not "
"assume any tool action happened just because the agent says so."
),
choice_scores={"complete": 1.0, "incomplete": 0.0},
pass_threshold=1.0,
)
```
```typescript TypeScript theme={null}
const judge = await client.scorers.create({
name: "Handoff clarity",
kind: "llm-judge",
prompt:
"Read the agent's final summary. Choose 'complete' only if it names " +
"ENG-2, explains the invoice problem, and gives a next action. Do not " +
"assume any tool action happened just because the agent says so.",
choiceScores: { complete: 1.0, incomplete: 0.0 },
passThreshold: 1.0,
});
```
```javascript JavaScript theme={null}
const judge = await client.scorers.create({
name: "Handoff clarity",
kind: "llm-judge",
prompt:
"Read the agent's final summary. Choose 'complete' only if it names " +
"ENG-2, explains the invoice problem, and gives a next action. Do not " +
"assume any tool action happened just because the agent says so.",
choiceScores: { complete: 1.0, incomplete: 0.0 },
passThreshold: 1.0,
});
```
```go Go theme={null}
judge, err := client.Scorers.Create(ctx, chronicle.ScorerCreateParams{
Name: "Handoff clarity",
Kind: "llm-judge",
Prompt: "Read the agent's final summary. Choose 'complete' only if it names " +
"ENG-2, explains the invoice problem, and gives a next action. Do not " +
"assume any tool action happened just because the agent says so.",
ChoiceScores: map[string]float64{"complete": 1.0, "incomplete": 0.0},
PassThreshold: 1.0,
})
if err != nil {
return err
}
```
```rust Rust theme={null}
let judge = client.scorers().create(ScorerCreateParams {
name: "Handoff clarity".into(),
kind: "llm-judge".into(),
prompt: Some("Read the agent's final summary. Choose 'complete' only if it names \
ENG-2, explains the invoice problem, and gives a next action. Do not \
assume any tool action happened just because the agent says so.".into()),
choice_scores: Some([("complete".into(), 1.0), ("incomplete".into(), 0.0)].into()),
pass_threshold: Some(1.0),
..Default::default()
}).await?;
```
For a code scorer, pass `kind="code"`, `language="python"` or
`"typescript"`, and `code` containing a `handler(input, output, expected, metadata)`
that returns `{"score": 0..1}`. Python scorers can import
`chronicle_world_scorer` to check world changes; the scorers Worldsmith
generates are good examples.
A task that starts in a world needs that world's `environment_version_id`
(from `world.evaluation`). Tasks that don't touch a world can leave it out.
```python Python theme={null}
task = client.tasks.create(
task_suite_id=suite.id,
title="Triage the duplicate invoice",
instruction=(
"Read Linear issue ENG-2 and Salesforce case 00001001. Move ENG-2 to "
"In Progress. Post this exact handoff in #erp-alerts: "
"ENG-2 is in progress for order 7001; case 00001001 remains open. "
"Keep the case open and do not claim the bug is fixed. Finish with a summary that names ENG-2, explains the invoice problem, and gives a next action."
),
expected_outcome="ENG-2 is In Progress and #erp-alerts has a new handoff message.",
environment_version_id=world.evaluation.environment_version_id,
scorers=[*source_task.scorers, {"scorer_id": judge.id, "weight": "med", "pass_threshold": 1.0}],
)
```
```typescript TypeScript theme={null}
const task = await client.tasks.create({
taskSuiteId: suite.id,
title: "Triage the duplicate invoice",
instruction:
"Read Linear issue ENG-2 and Salesforce case 00001001. Move ENG-2 to " +
"In Progress. Post this exact handoff in #erp-alerts: " +
"ENG-2 is in progress for order 7001; case 00001001 remains open. " +
"Keep the case open and do not claim the bug is fixed. Finish with a summary that names ENG-2, explains the invoice problem, and gives a next action.",
expectedOutcome: "ENG-2 is In Progress and #erp-alerts has a new handoff message.",
environmentVersionId: world.evaluation.environmentVersionId,
scorers: [...sourceTask.scorers, { scorerId: judge.id, weight: "med", passThreshold: 1.0 }],
});
```
```javascript JavaScript theme={null}
const task = await client.tasks.create({
taskSuiteId: suite.id,
title: "Triage the duplicate invoice",
instruction:
"Read Linear issue ENG-2 and Salesforce case 00001001. Move ENG-2 to " +
"In Progress. Post this exact handoff in #erp-alerts: " +
"ENG-2 is in progress for order 7001; case 00001001 remains open. " +
"Keep the case open and do not claim the bug is fixed. Finish with a summary that names ENG-2, explains the invoice problem, and gives a next action.",
expectedOutcome: "ENG-2 is In Progress and #erp-alerts has a new handoff message.",
environmentVersionId: world.evaluation.environmentVersionId,
scorers: [...sourceTask.scorers, { scorerId: judge.id, weight: "med", passThreshold: 1.0 }],
});
```
```go Go theme={null}
task, err := client.Tasks.Create(ctx, chronicle.TaskCreateParams{
TaskSuiteID: suite.ID,
Title: "Triage the duplicate invoice",
Instruction: "Read Linear issue ENG-2 and Salesforce case 00001001. Move ENG-2 to " +
"In Progress. Post this exact handoff in #erp-alerts: " +
"ENG-2 is in progress for order 7001; case 00001001 remains open. " +
"Keep the case open and do not claim the bug is fixed. Finish with a summary that names ENG-2, explains the invoice problem, and gives a next action.",
ExpectedOutcome: "ENG-2 is In Progress and #erp-alerts has a new handoff message.",
EnvironmentVersionID: world.Evaluation.EnvironmentVersionID,
Scorers: append(append([]chronicle.ScorerBinding(nil), sourceTask.Scorers...),
chronicle.ScorerBinding{ScorerID: judge.ID, Weight: "med", PassThreshold: 1.0}),
})
if err != nil {
return err
}
```
```rust Rust theme={null}
let task = client.tasks().create(TaskCreateParams {
task_suite_id: suite.id.clone(),
title: "Triage the duplicate invoice".into(),
instruction: "Read Linear issue ENG-2 and Salesforce case 00001001. Move ENG-2 to \
In Progress. Post this exact handoff in #erp-alerts: \
ENG-2 is in progress for order 7001; case 00001001 remains open. \
Keep the case open and do not claim the bug is fixed. Finish with a summary that names ENG-2, explains the invoice problem, and gives a next action.".into(),
expected_outcome: Some("ENG-2 is In Progress and #erp-alerts has a new handoff message.".into()),
environment_version_id: world.evaluation.as_ref().map(|e| e.environment_version_id.clone()),
scorers: source_task.scorers.iter().cloned().chain(std::iter::once(
ScorerBinding { scorer_id: judge.id.clone(), weight: "med".into(), pass_threshold: Some(1.0) }
)).collect(),
..Default::default()
}).await?;
```
[Open the source task image](/images/product/worldsmith-task-focused.png).
A scorer binding takes a `scorer_id`, an optional `weight` (`low`, `med`,
`high`), and an optional `pass_threshold` that overrides the scorer's own.
Publishing freezes the tasks, their scorer code and prompts, weights, and
thresholds into a version that never changes.
```python Python theme={null}
version = client.task_suites.publish(suite.id, label="Baseline")
```
```typescript TypeScript theme={null}
const version = await client.taskSuites.publish(suite.id, { label: "Baseline" });
```
```javascript JavaScript theme={null}
const version = await client.taskSuites.publish(suite.id, { label: "Baseline" });
```
```go Go theme={null}
version, err := client.TaskSuites.Publish(ctx, suite.ID, chronicle.PublishParams{Label: "Baseline"})
if err != nil {
return err
}
fmt.Println(version.ID)
```
```rust Rust theme={null}
let version = client.task_suites().publish(&suite.id, Some("Baseline")).await?;
println!("{}", version.id);
```
Starting an evaluation publishes automatically if the working copy has
changed, so you only need this to name a version explicitly.
Now run it: [Evaluations](/api-reference/evaluations). [Explore scorer outcomes](/product-explorer#scores) to see why execution status and task correctness are separate.
## Edit a task
`tasks.update` changes any of `title`, `instruction`, `expected_outcome`,
`environment_version_id`. Fields you don't pass are left alone.
`tasks.set_scorers` replaces the whole scorer list, so include every scorer you
want to keep.
```python Python theme={null}
client.tasks.update(suite.id, task.id, title="Invoice triage and handoff")
# Reapply the complete binding list after editing a library scorer.
client.tasks.set_scorers(suite.id, task.id, scorers=task.scorers)
```
```typescript TypeScript theme={null}
await client.tasks.update(suite.id, task.id, { title: "Invoice triage and handoff" });
// Reapply the complete binding list after editing a library scorer.
await client.tasks.setScorers(suite.id, task.id, task.scorers);
```
```javascript JavaScript theme={null}
await client.tasks.update(suite.id, task.id, { title: "Invoice triage and handoff" });
// Reapply the complete binding list after editing a library scorer.
await client.tasks.setScorers(suite.id, task.id, task.scorers);
```
```go Go theme={null}
_, err = client.Tasks.Update(ctx, suite.ID, task.ID, chronicle.TaskUpdateParams{Title: "Invoice triage and handoff"})
if err != nil { return err }
// Reapply the complete binding list after editing a library scorer.
_, err = client.Tasks.SetScorers(ctx, suite.ID, task.ID, task.Scorers)
if err != nil { return err }
```
```rust Rust theme={null}
client.tasks().update(&suite.id, &task.id, TaskUpdateParams {
title: Some("Invoice triage and handoff".into()), ..Default::default()
}).await?;
// Reapply the complete binding list after editing a library scorer.
client.tasks().set_scorers(&suite.id, &task.id, task.scorers.clone()).await?;
```
If you edit a scorer in the library, call `set_scorers` again on the tasks that
use it so the next published version picks up the change.
## Create a task from a trace
Turn a real recorded interaction into a test. Chronicle snapshots the trace's
events and drafts a title, instruction, and expected outcome from them. Use a
real trace ID from [Events](/api-reference/telemetry#record-a-trace) in place
of ``. After creating the task, inspect its `events` and replace `` with
the last event the agent should receive as context. Later events are held back
as the reference answer.
```python Python theme={null}
trace_task = client.tasks.from_trace(suite.id, trace_id="", notes="Invoice triage regression")
client.tasks.update(
suite.id, trace_task.id,
instruction="Investigate the duplicate invoice report and start triage.",
cutoff_event_id="",
)
```
```typescript TypeScript theme={null}
const traceTask = await client.tasks.fromTrace(suite.id, { traceId: "", notes: "Invoice triage regression" });
await client.tasks.update(suite.id, traceTask.id, {
instruction: "Investigate the duplicate invoice report and start triage.",
cutoffEventId: "",
});
```
```javascript JavaScript theme={null}
const traceTask = await client.tasks.fromTrace(suite.id, { traceId: "", notes: "Invoice triage regression" });
await client.tasks.update(suite.id, traceTask.id, {
instruction: "Investigate the duplicate invoice report and start triage.",
cutoffEventId: "",
});
```
```go Go theme={null}
traceTask, err := client.Tasks.FromTrace(ctx, suite.ID, chronicle.TaskFromTraceParams{
TraceID: "", Notes: "Invoice triage regression",
})
if err != nil {
return err
}
_, err = client.Tasks.Update(ctx, suite.ID, traceTask.ID, chronicle.TaskUpdateParams{
Instruction: "Investigate the duplicate invoice report and start triage.",
CutoffEventID: "",
})
if err != nil { return err }
```
```rust Rust theme={null}
let trace_task = client.tasks().from_trace(&suite.id, TaskFromTraceParams {
trace_id: "".into(),
notes: Some("Invoice triage regression".into()),
}).await?;
client.tasks().update(&suite.id, &trace_task.id, TaskUpdateParams {
instruction: Some("Investigate the duplicate invoice report and start triage.".into()),
cutoff_event_id: Some("".into()),
..Default::default()
}).await?;
```
Everything after the cutoff is the reference answer. New events on the live
trace don't change the task. Add scorers and publish as usual.
## Test a scorer
Run a scorer on sample data before attaching it:
```python Python theme={null}
result = client.scorers.test(
kind="code",
language="python",
code="def handler(input, output, expected, metadata):\n return {'score': 1.0 if output == expected else 0.0}\n",
input="Return the issue ID.",
output="ENG-2",
expected="ENG-2",
)
if result.error:
raise RuntimeError(result.error)
print(result.score)
```
```typescript TypeScript theme={null}
const result = await client.scorers.test({
kind: "code",
language: "python",
code: "def handler(input, output, expected, metadata):\n return {'score': 1.0 if output == expected else 0.0}\n",
input: "Return the issue ID.",
output: "ENG-2",
expected: "ENG-2",
});
if (result.error) throw new Error(result.error);
console.log(result.score);
```
```javascript JavaScript theme={null}
const result = await client.scorers.test({
kind: "code",
language: "python",
code: "def handler(input, output, expected, metadata):\n return {'score': 1.0 if output == expected else 0.0}\n",
input: "Return the issue ID.",
output: "ENG-2",
expected: "ENG-2",
});
if (result.error) throw new Error(result.error);
console.log(result.score);
```
```go Go theme={null}
result, err := client.Scorers.Test(ctx, chronicle.ScorerTestParams{
Kind: "code",
Language: "python",
Code: "def handler(input, output, expected, metadata):\n return {'score': 1.0 if output == expected else 0.0}\n",
Input: "Return the issue ID.",
Output: "ENG-2",
Expected: "ENG-2",
})
if err != nil {
return err
}
if result.Error != "" { return fmt.Errorf("scorer failed: %s", result.Error) }
fmt.Println(result.Score)
```
```rust Rust theme={null}
let result = client.scorers().test(ScorerTestParams {
kind: "code".into(),
language: Some("python".into()),
code: Some("def handler(input, output, expected, metadata):\n return {'score': 1.0 if output == expected else 0.0}\n".into()),
input: "Return the issue ID.".into(),
output: "ENG-2".into(),
expected: Some("ENG-2".into()),
..Default::default()
}).await?;
if let Some(error) = &result.error { return Err(error.clone().into()); }
println!("{:?}", result.score);
```
`error` is set when the code failed to run. World-state scorers can't be tested
this way because there's no world; run them in an evaluation.