Skip to main content
Run anvil eval <file> to execute a structured test suite against an agent route. Each suite is a TypeScript file that default-exports a suite object created with defineEvalSuite from anvil/eval. Anvil loads the file, runs every test case through your agent, evaluates all assertions, and prints a pass/fail report. Because anvil eval exits with code 1 when any case fails, you can plug it directly into CI pipelines.

​
Usage

anvil eval <file>
<file> is the path to a suite file that default-exports the result of defineEvalSuite(...). Relative and absolute paths are both accepted.

​
Assertions

anvil/eval ships two families of assertions.

​
Deterministic assertions

These assertions run synchronously against the agent’s recorded run and never make additional model calls:
FunctionWhat it checks
outputContains(substring)Final text output includes the given substring
outputMatches(re)Final text output matches the given RegExp
outputJson(schema)Final text is valid JSON that satisfies a Zod schema
toolCalled(name)At least one tool call with the given name appears in the run
maxCost(usd)result.totalCostUsd does not exceed the given dollar amount
maxIterations(n)The agent completed within n agentic iterations

​
LLM-as-judge assertion

judge({ client, rubric, model? }) scores the agent’s final output against a free-text rubric using a model. The judge prompt is strict: the model must respond with {"pass": boolean, "reason": string}. Judge calls run through the same LlmClient instance you provide — so they are traced and cost-tracked exactly like any other model call, and their spend is visible in the dashboard.

​
defineEvalSuite

Import defineEvalSuite from anvil/eval and use it as the default export of your suite file. It is a typed identity function — its only job is to give you autocomplete and type safety: 
import { defineEvalSuite } from 'anvil-sdk/eval';

export default defineEvalSuite({ /* EvalSuite */ });
A complete EvalSuite has three fields:
FieldTypeDescription
namestringHuman-readable suite name printed in the report header
runnerEvalRunnerAsync function that turns input messages into an EvalRunContext
casesEvalCase[]Array of test cases, each with a name, input, and assert list
Use agentRunner from anvil/eval to build a runner from your agent configuration without wiring up the streaming loop yourself.

​
Minimal eval suite example

// evals/chat.eval.ts
import { agentRunner, defineEvalSuite, judge, outputContains, toolCalled, maxCost } from 'anvil-sdk/eval';
import { LlmClient } from 'anvil-sdk/llm';
import { myTools } from '../server/tools/index.js';

const client = new LlmClient({ /* your driver config */ });

export default defineEvalSuite({
  name: 'Chat agent — basic scenarios',
  runner: agentRunner({
    client,
    tools: myTools,
    model: 'gpt-4o-mini',
    system: 'You are a helpful assistant.',
  }),
  cases: [
    {
      name: 'greets the user',
      input: 'Hello!',
      assert: [
        outputContains('hello'),
        maxCost(0.001),
      ],
    },
    {
      name: 'calls the weather tool',
      input: "What's the weather in Paris?",
      assert: [
        toolCalled('get_weather'),
        outputContains('Paris'),
      ],
    },
    {
      name: 'answer quality — judge',
      input: 'Explain recursion in one sentence.',
      assert: [
        judge({
          client,
          rubric: 'The explanation must mention that a function calls itself and must be a single sentence.',
        }),
      ],
    },
  ],
});
Run it: 
anvil eval evals/chat.eval.ts

​
Example report output

Chat agent — basic scenarios
  PASS  greets the user
  PASS  calls the weather tool
  FAIL  answer quality — judge
        ✗ judge: The explanation must mention... — The output was two sentences.

[anvil] eval: 2 passed, 1 failed
The process exits with code 1 because one case failed. A fully passing run looks like: 
Chat agent — basic scenarios
  PASS  greets the user
  PASS  calls the weather tool
  PASS  answer quality — judge

[anvil] eval: 3 passed, 0 failed
Exit code is 0.

​
CI integration

# .github/workflows/ci.yml
- name: Run agent evals
  run: npx anvil eval evals/chat.eval.ts
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
Judge model calls are routed through the same LlmClient you provide to judge(...). This means they are traced, and their token usage and cost appear in the Anvil dashboard alongside the agent calls they evaluate.
The suite file must default-export the return value of defineEvalSuite(...). If the default export is missing or does not have a cases array, anvil eval prints an error and exits with code 1 without running any cases.