Skip to main content
A good brief describes the system you want to simulate: what the agent can access, its starting state, what the agent must do, and how you will grade the result. Use the duplicate-invoice workflow as a starting point: begin engineering triage and notify finance while the customer case stays open.

​
Include six details

Brief anatomy

​
A triage task with evidence and checks

System and agent role

Acme’s invoice workflow, with an operations agent handling a duplicate invoice.

Services

Slack, Linear, Salesforce, and SAP S/4HANA with shared people and customers.

Starting problem

Northwind was invoiced twice. Its case is open; the engineering issue is Todo.

Evidence

The order, case, issue, and finance discussion reference the same incident.

Required actions

Move the issue to In Progress and post a new finance handoff. Keep the case open.

Success checks

Grade the issue update and the new message separately. Add a check for each constraint you also want graded.

Worldsmith can build worlds using these four services:
  • Slack
  • Linear
  • Salesforce
  • SAP S/4HANA

​
Copy a complete brief

This asks for a variation of the built-in triage workflow. Review the generated record identifiers before running it. For the exact ENG-2 example shown in the screenshots, load the built-in world.
“Make a world to test my agent” leaves the workflow and success criteria unspecified. If Worldsmith asks questions, answer them and choose Continue, or tell it which details it should choose. Press Enter to send a brief and Shift+Enter for a new line.

​
Make the evidence findable

  • Cross-reference records: the Slack thread names the issue and case; the issue names the order.
  • Include facts that change the decision: who reported the problem, what was ruled out, and what remains unresolved.
  • Give dates when the agent needs to distinguish an older conclusion from newer evidence.
  • Start with one workflow. Add a similar incident or conflicting message after the simple version works.
Explore the connected records to see how ENG-2 points to SAP order 7001, Salesforce case 00001001, and #erp-alerts.

​
Make success observable

A change must differ from the starting state: an issue already In Progress does not prove the agent moved it. Use separate scorers when you want separate results for the issue update and Slack handoff. The built-in task’s first scorer checks ENG-2 → In Progress, with a 1.00 pass threshold. Its second checks a new message in #erp-alerts containing the exact handoff text in the task’s steps.
Python scorer for the duplicate-invoice task, checking the Linear issue state equals in_progress, with a 1.00 pass threshold

The built-in triage scorer checks an update to ENG-2's state. A score of 1.00 passes.

Open the scorer image. Grade constraints as well as changes. A handoff saying “the case remains open” does not verify the case’s actual status. The two built-in scorers do not independently check that constraint or the claim that the bug is fixed. To grade those requirements, select the task and ask Worldsmith to add the corresponding checks, then review them before launching. Use Add scorer to write a check yourself, or Generate scorers for the selected task. Scorers can be Python, TypeScript, or an LLM judge. World-state checks can cover new Slack messages, Linear state changes and comments, Salesforce case or opportunity updates, and SAP sales order fields. Use an LLM judge for the quality and accuracy of the agent’s written handoff. See Tasks and scorers.

​
Refine one thing at a time

Say what to change and what to preserve:
Open a task or scorer before asking about it. Worldsmith includes your selection with the next message: “Make this message scorer require the order and case identifiers shown in the task.” Edits are checked automatically and save to the draft until you launch a version.

​
Review before launch

  1. Starting state. The problem exists, and the changes the agent must make have not happened.
  2. Connected evidence. People, customers, issues, cases, and threads agree on the incident.
  3. Instructions. The task names the required actions, exact text where needed, and constraints.
  4. Scorers. Check the target record, expected change, required text, and pass threshold. Constraints need checks if they should affect the score.
  5. World prepared. No compile problems remain. Expand the notes and read the warnings.
Then launch the world and run an evaluation. World-state scorers need the trial’s captured world data; the scorer editor’s sample test does not supply it.

Build a world

Follow the built-in task through launch and evaluation.