LlmClient: Provider-Agnostic Model Calls in Anvil JS

LlmClient is the provider-agnostic model client at the heart of Anvil’s agent system. It owns driver dispatch, transient retries, fallback chains, cost accumulation, and structured-output enforcement — so your agent handlers call generate, generateObject, or stream without worrying about provider-specific details. Import it from anvil/llm:

import { LlmClient, AnthropicDriver } from 'anvil-sdk/llm';

Constructor

const client = new LlmClient(options: LlmClientOptions);

ModelDriver[]

required

One or more driver instances that handle model calls. Each driver advertises which models it supports via driver.supports(model). At least one driver is required. Add multiple drivers to cover different model families.

string

The model used when a generate, generateObject, or stream call omits the model field. If omitted here too, the call throws unless the request specifies a model explicitly.

string[]

An ordered list of model IDs to try after the primary model fails with a transient error (RetryableModelError). The client walks the list left-to-right, retrying maxRetries times per model before advancing. Each fallback model must be supported by one of the registered drivers.

number

Attempts per model before moving to the next in the fallback chain. Defaults to 2.

number

Base backoff delay in milliseconds between retry attempts. The actual wait before each attempt is retryBaseMs * attemptNumber. Defaults to 50. Set to 0 in tests to eliminate wait time.

(event: TraceEvent) => void

Observability hook called after every model call — successful or failed. The TraceEvent carries the model name, provider, attempt number, token usage, cost, and whether the call hit the fallback chain. Use this to feed your own metrics or logging pipeline.

Drivers

Provider SDKs are optional peer dependencies loaded lazily at runtime — the Anvil kernel never pulls them in unless you instantiate the matching driver.

Anthropic
OpenAI
Gemini

import { AnthropicDriver } from 'anvil-sdk/llm';

new AnthropicDriver({ apiKey: process.env.ANTHROPIC_API_KEY })

Requires @anthropic-ai/sdk as a peer dependency. Supports all claude-* models (e.g. claude-opus-4-8, claude-sonnet-4-5, claude-haiku-3-5). Uses the top-level system parameter and adaptive thinking when thinking: true is set on the request.

import { LlmClient, AnthropicDriver } from 'anvil-sdk/llm';

const client = new LlmClient({
  drivers: [new AnthropicDriver({ apiKey: process.env.ANTHROPIC_API_KEY })],
  defaultModel: 'claude-opus-4-8',
});

import { OpenAIDriver } from 'anvil-sdk/llm';

new OpenAIDriver({ apiKey: process.env.OPENAI_API_KEY })

Requires openai as a peer dependency. Supports gpt-* models (gpt-4o, gpt-4o-mini) and o-series reasoning models (o1, o3, o4-mini). Uses the Chat Completions API with structured output via response_format.json_schema.

import { LlmClient, OpenAIDriver } from 'anvil-sdk/llm';

const client = new LlmClient({
  drivers: [new OpenAIDriver({ apiKey: process.env.OPENAI_API_KEY })],
  defaultModel: 'gpt-4o',
});

import { GeminiDriver } from 'anvil-sdk/llm';

new GeminiDriver({ apiKey: process.env.GEMINI_API_KEY })

Requires @google/generative-ai as a peer dependency. Supports all gemini-* models (e.g. gemini-2.5-flash, gemini-2.5-pro). Gemini function calls carry no IDs; the driver synthesizes stable IDs from the function name so the tool loop works correctly.

import { LlmClient, GeminiDriver } from 'anvil-sdk/llm';

const client = new LlmClient({
  drivers: [new GeminiDriver({ apiKey: process.env.GEMINI_API_KEY })],
  defaultModel: 'gemini-2.5-flash',
});

Methods

`client.generate(req)`

Make a single-turn model call and return the complete result.

const result = await client.generate({
  model: 'claude-opus-4-8',  // optional if defaultModel is set
  system: 'You are a helpful assistant.',
  messages: [{ role: 'user', content: 'What is the capital of France?' }],
});

console.log(result.text);        // "Paris"
console.log(result.usage);       // { inputTokens: 14, outputTokens: 6 }
console.log(result.costUsd);     // 0.000042

string

The model’s text response.

string

The model ID that produced the response (may differ from the request if the fallback chain was used).

string

The provider name, e.g. "anthropic", "openai", "gemini".

{ inputTokens: number, outputTokens: number }

Token counts for this call.

number | undefined

Estimated USD cost from the built-in pricing table. undefined if pricing data for the model isn’t registered.

ToolCall[] | undefined

Tool invocations the model requested, when tools were provided and stopReason is "tool_use".

`client.generateObject(req, schema, opts?)`

Request structured output validated against a Zod schema. On validation failure the client automatically re-prompts the model with the validation error, up to maxRepairs times.

import { z } from 'zod';

const SentimentSchema = z.object({
  sentiment: z.enum(['positive', 'neutral', 'negative']),
  confidence: z.number().min(0).max(1),
});

const result = await client.generateObject(
  {
    system: 'Classify the sentiment of the user message.',
    messages: [{ role: 'user', content: 'This product is amazing!' }],
  },
  SentimentSchema,
  { maxRepairs: 2, name: 'sentiment_analysis' },
);

console.log(result.object.sentiment);    // "positive"
console.log(result.object.confidence);  // 0.97

GenerateRequest

required

The same request shape as generate. Do not set responseFormat manually — generateObject sets it automatically from the Zod schema.

z.ZodType<T>

required

The Zod schema to validate the response against.

number

Maximum number of repair attempts after a validation failure. Defaults to 2. Each attempt re-prompts the model with the Zod error message.

string

Optional name for the JSON schema advertised to the model. Helps some providers choose a better response format.

`client.stream(req)`

Stream tokens from the model as they arrive. Yields StreamEvent objects.

for await (const event of client.stream({
  messages: [{ role: 'user', content: 'Tell me a short story.' }],
})) {
  if (event.type === 'text') process.stdout.write(event.text);
  if (event.type === 'done') console.log('\nUsage:', event.result.usage);
}

StreamEvent

A text fragment from the model. Concatenate these to build the full response.

StreamEvent

The final event, carrying the complete GenerateResult with usage, cost, and stop reason.

Streaming does not retry mid-stream — bytes may already be on the wire. The client only falls back to the next model in the chain if the stream fails before yielding any events.

Fallback chain example

import { LlmClient, AnthropicDriver, OpenAIDriver } from 'anvil-sdk/llm';

const client = new LlmClient({
  drivers: [
    new AnthropicDriver({ apiKey: process.env.ANTHROPIC_API_KEY }),
    new OpenAIDriver({ apiKey: process.env.OPENAI_API_KEY }),
  ],
  defaultModel: 'claude-opus-4-8',
  fallback: ['gpt-4o'],
  maxRetries: 2,
  onTrace: (event) => {
    if (event.fallback) {
      console.warn(`Fell back to ${event.model} after primary model failed`);
    }
  },
});

The client tries claude-opus-4-8 up to two times. If both attempts throw a RetryableModelError (rate limit, 5xx, connection error) it moves to gpt-4o and tries that up to two times. Non-retryable errors (bad request, auth failure) stop the chain immediately.

Install only the provider SDKs you need. Adding AnthropicDriver to your drivers list does not pull in @anthropic-ai/sdk until the first request that routes through it.

Get Started

Routing

MCP & Tools

Agent Routes

Observability

Safety & Durability

Memory & RAG

Background Tasks

LlmClient: Provider-Agnostic Model Calls in Anvil JS

Constructor

Drivers

Methods

`client.generate(req)`

`client.generateObject(req, schema, opts?)`

`client.stream(req)`

Fallback chain example

​Constructor

​Drivers

​Methods

​client.generate(req)

​client.generateObject(req, schema, opts?)

​client.stream(req)

​Fallback chain example

Constructor

Drivers

Methods

`client.generate(req)`

`client.generateObject(req, schema, opts?)`

`client.stream(req)`

Fallback chain example