> ## Documentation Index
> Fetch the complete documentation index at: https://anvil.thatdevguy.in/llms.txt
> Use this file to discover all available pages before exploring further.

# anvil eval: Run Deterministic and Judge-Based Eval Suites

> Run a defineEvalSuite test file against an agent route with deterministic and LLM-as-judge assertions; exits non-zero on failure for CI use.

Run `anvil eval <file>` to execute a structured test suite against an agent route. Each suite is a TypeScript file that default-exports a suite object created with `defineEvalSuite` from `anvil/eval`. Anvil loads the file, runs every test case through your agent, evaluates all assertions, and prints a pass/fail report. Because `anvil eval` exits with code `1` when any case fails, you can plug it directly into CI pipelines.

## Usage

```bash theme={null}
anvil eval <file>
```

`<file>` is the path to a suite file that default-exports the result of `defineEvalSuite(...)`. Relative and absolute paths are both accepted.

## Assertions

`anvil/eval` ships two families of assertions.

### Deterministic assertions

These assertions run synchronously against the agent's recorded run and never make additional model calls:

| Function                    | What it checks                                                |
| --------------------------- | ------------------------------------------------------------- |
| `outputContains(substring)` | Final text output includes the given substring                |
| `outputMatches(re)`         | Final text output matches the given `RegExp`                  |
| `outputJson(schema)`        | Final text is valid JSON that satisfies a Zod schema          |
| `toolCalled(name)`          | At least one tool call with the given name appears in the run |
| `maxCost(usd)`              | `result.totalCostUsd` does not exceed the given dollar amount |
| `maxIterations(n)`          | The agent completed within `n` agentic iterations             |

### LLM-as-judge assertion

`judge({ client, rubric, model? })` scores the agent's final output against a free-text rubric using a model. The judge prompt is strict: the model must respond with `{"pass": boolean, "reason": string}`. Judge calls run through the same `LlmClient` instance you provide — so they are **traced and cost-tracked** exactly like any other model call, and their spend is visible in the dashboard.

## `defineEvalSuite`

Import `defineEvalSuite` from `anvil/eval` and use it as the default export of your suite file. It is a typed identity function — its only job is to give you autocomplete and type safety:

```typescript theme={null}
import { defineEvalSuite } from 'anvil-sdk/eval';

export default defineEvalSuite({ /* EvalSuite */ });
```

A complete `EvalSuite` has three fields:

| Field    | Type         | Description                                                         |
| -------- | ------------ | ------------------------------------------------------------------- |
| `name`   | `string`     | Human-readable suite name printed in the report header              |
| `runner` | `EvalRunner` | Async function that turns input messages into an `EvalRunContext`   |
| `cases`  | `EvalCase[]` | Array of test cases, each with a `name`, `input`, and `assert` list |

Use `agentRunner` from `anvil/eval` to build a runner from your agent configuration without wiring up the streaming loop yourself.

## Minimal eval suite example

```typescript theme={null}
// evals/chat.eval.ts
import { agentRunner, defineEvalSuite, judge, outputContains, toolCalled, maxCost } from 'anvil-sdk/eval';
import { LlmClient } from 'anvil-sdk/llm';
import { myTools } from '../server/tools/index.js';

const client = new LlmClient({ /* your driver config */ });

export default defineEvalSuite({
  name: 'Chat agent — basic scenarios',
  runner: agentRunner({
    client,
    tools: myTools,
    model: 'gpt-4o-mini',
    system: 'You are a helpful assistant.',
  }),
  cases: [
    {
      name: 'greets the user',
      input: 'Hello!',
      assert: [
        outputContains('hello'),
        maxCost(0.001),
      ],
    },
    {
      name: 'calls the weather tool',
      input: "What's the weather in Paris?",
      assert: [
        toolCalled('get_weather'),
        outputContains('Paris'),
      ],
    },
    {
      name: 'answer quality — judge',
      input: 'Explain recursion in one sentence.',
      assert: [
        judge({
          client,
          rubric: 'The explanation must mention that a function calls itself and must be a single sentence.',
        }),
      ],
    },
  ],
});
```

Run it:

```bash theme={null}
anvil eval evals/chat.eval.ts
```

## Example report output

```text theme={null}
Chat agent — basic scenarios
  PASS  greets the user
  PASS  calls the weather tool
  FAIL  answer quality — judge
        ✗ judge: The explanation must mention... — The output was two sentences.

[anvil] eval: 2 passed, 1 failed
```

The process exits with code `1` because one case failed.

A fully passing run looks like:

```text theme={null}
Chat agent — basic scenarios
  PASS  greets the user
  PASS  calls the weather tool
  PASS  answer quality — judge

[anvil] eval: 3 passed, 0 failed
```

Exit code is `0`.

## CI integration

```yaml theme={null}
# .github/workflows/ci.yml
- name: Run agent evals
  run: npx anvil eval evals/chat.eval.ts
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

<Note>
  Judge model calls are routed through the same `LlmClient` you provide to `judge(...)`. This means they are traced, and their token usage and cost appear in the Anvil dashboard alongside the agent calls they evaluate.
</Note>

<Warning>
  The suite file **must** default-export the return value of `defineEvalSuite(...)`. If the default export is missing or does not have a `cases` array, `anvil eval` prints an error and exits with code `1` without running any cases.
</Warning>