anvil eval: Run Deterministic and Judge-Based Eval Suites

Run anvil eval <file> to execute a structured test suite against an agent route. Each suite is a TypeScript file that default-exports a suite object created with defineEvalSuite from anvil/eval. Anvil loads the file, runs every test case through your agent, evaluates all assertions, and prints a pass/fail report. Because anvil eval exits with code 1 when any case fails, you can plug it directly into CI pipelines.

Usage

anvil eval <file>

<file> is the path to a suite file that default-exports the result of defineEvalSuite(...). Relative and absolute paths are both accepted.

Assertions

anvil/eval ships two families of assertions.

Deterministic assertions

These assertions run synchronously against the agent’s recorded run and never make additional model calls:

Function	What it checks
`outputContains(substring)`	Final text output includes the given substring
`outputMatches(re)`	Final text output matches the given `RegExp`
`outputJson(schema)`	Final text is valid JSON that satisfies a Zod schema
`toolCalled(name)`	At least one tool call with the given name appears in the run
`maxCost(usd)`	`result.totalCostUsd` does not exceed the given dollar amount
`maxIterations(n)`	The agent completed within `n` agentic iterations

LLM-as-judge assertion

judge({ client, rubric, model? }) scores the agent’s final output against a free-text rubric using a model. The judge prompt is strict: the model must respond with {"pass": boolean, "reason": string}. Judge calls run through the same LlmClient instance you provide — so they are traced and cost-tracked exactly like any other model call, and their spend is visible in the dashboard.

`defineEvalSuite`

Import defineEvalSuite from anvil/eval and use it as the default export of your suite file. It is a typed identity function — its only job is to give you autocomplete and type safety:

import { defineEvalSuite } from 'anvil-sdk/eval';

export default defineEvalSuite({ /* EvalSuite */ });

A complete EvalSuite has three fields:

Field	Type	Description
`name`	`string`	Human-readable suite name printed in the report header
`runner`	`EvalRunner`	Async function that turns input messages into an `EvalRunContext`
`cases`	`EvalCase[]`	Array of test cases, each with a `name`, `input`, and `assert` list

Use agentRunner from anvil/eval to build a runner from your agent configuration without wiring up the streaming loop yourself.

Minimal eval suite example

// evals/chat.eval.ts
import { agentRunner, defineEvalSuite, judge, outputContains, toolCalled, maxCost } from 'anvil-sdk/eval';
import { LlmClient } from 'anvil-sdk/llm';
import { myTools } from '../server/tools/index.js';

const client = new LlmClient({ /* your driver config */ });

export default defineEvalSuite({
  name: 'Chat agent — basic scenarios',
  runner: agentRunner({
    client,
    tools: myTools,
    model: 'gpt-4o-mini',
    system: 'You are a helpful assistant.',
  }),
  cases: [
    {
      name: 'greets the user',
      input: 'Hello!',
      assert: [
        outputContains('hello'),
        maxCost(0.001),
      ],
    },
    {
      name: 'calls the weather tool',
      input: "What's the weather in Paris?",
      assert: [
        toolCalled('get_weather'),
        outputContains('Paris'),
      ],
    },
    {
      name: 'answer quality — judge',
      input: 'Explain recursion in one sentence.',
      assert: [
        judge({
          client,
          rubric: 'The explanation must mention that a function calls itself and must be a single sentence.',
        }),
      ],
    },
  ],
});

Run it:

anvil eval evals/chat.eval.ts

Example report output

Chat agent — basic scenarios
  PASS  greets the user
  PASS  calls the weather tool
  FAIL  answer quality — judge
        ✗ judge: The explanation must mention... — The output was two sentences.

[anvil] eval: 2 passed, 1 failed

The process exits with code 1 because one case failed. A fully passing run looks like:

Chat agent — basic scenarios
  PASS  greets the user
  PASS  calls the weather tool
  PASS  answer quality — judge

[anvil] eval: 3 passed, 0 failed

Exit code is 0.

CI integration

# .github/workflows/ci.yml
- name: Run agent evals
  run: npx anvil eval evals/chat.eval.ts
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Judge model calls are routed through the same LlmClient you provide to judge(...). This means they are traced, and their token usage and cost appear in the Anvil dashboard alongside the agent calls they evaluate.

The suite file must default-export the return value of defineEvalSuite(...). If the default export is missing or does not have a cases array, anvil eval prints an error and exits with code 1 without running any cases.

​Usage

​Assertions

​Deterministic assertions

​LLM-as-judge assertion

​defineEvalSuite

​Minimal eval suite example

​Example report output

​CI integration

Usage

Assertions

Deterministic assertions

LLM-as-judge assertion

`defineEvalSuite`

Minimal eval suite example

Example report output

CI integration