> ## Documentation Index
> Fetch the complete documentation index at: https://anvil.thatdevguy.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Semantic Cache: Skip Redundant LLM Calls by Similarity

> Use SemanticCache to skip redundant LLM calls when a semantically equivalent prompt has already been answered, cutting latency and token spend.

LLMs are expensive. When users ask questions that are semantically equivalent — "What's your return policy?" and "How do I return an item?" — there is no reason to run two separate model calls. `SemanticCache` stores responses keyed on embedding similarity rather than exact string matching, so a new prompt that is close enough to a cached one returns the stored answer instantly without touching the model.

## How it works

When you call `cache.get(prompt)`, `SemanticCache` embeds the prompt and queries the backing vector store for the nearest stored entry. If the top result's cosine similarity exceeds the configured `threshold` (default `0.95`) and the entry has not expired, it's a cache hit and the stored response is returned. On a miss, your code calls the LLM as normal, then stores the response with `cache.set()` for future hits.

The `cache.wrap()` convenience method handles the full get-or-produce flow in one call.

## Setup

```ts theme={null}
import { SemanticCache, HashEmbedder, SqliteVectorStore } from 'anvil-sdk/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),          // swap for a semantic embedder in production
  store: await SqliteVectorStore.open(), // persists hits across restarts
  threshold: 0.92,                       // tune for your use case
  ttlMs: 1000 * 60 * 60 * 24,           // 24-hour TTL
});
```

### Constructor options

<Accordion title="SemanticCacheOptions">
  | Option      | Type          | Default             | Description                                                                                                        |
  | ----------- | ------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ |
  | `embedder`  | `Embedder`    | — (required)        | Embeds prompts for similarity comparison                                                                           |
  | `threshold` | `number`      | `0.95`              | Minimum cosine similarity for a cache hit. Lower values cast a wider net; higher values require near-exact matches |
  | `store`     | `VectorStore` | `MemoryVectorStore` | Backing store for cached `(embedding, response)` pairs                                                             |
  | `ttlMs`     | `number`      | none                | Entry lifetime in milliseconds. Entries older than this are treated as misses                                      |
</Accordion>

<Note>
  The default `store` is an in-process `MemoryVectorStore`. Use `SqliteVectorStore` or another persistent store to keep the cache across process restarts.
</Note>

## API

### `cache.get(prompt, opts?)`

Looks up a prompt in the cache. Returns a `CacheLookup` object.

```ts theme={null}
const result = await cache.get('What is your return policy?');

if (result.hit) {
  console.log('Cache hit! Score:', result.score);
  return result.value; // the stored response string
}
```

`CacheLookup` fields:

| Field   | Type      | Description                                                         |                                            |
| ------- | --------- | ------------------------------------------------------------------- | ------------------------------------------ |
| `hit`   | `boolean` | `true` if a sufficiently similar entry was found and is still fresh |                                            |
| `value` | \`string  | undefined\`                                                         | The cached response on a hit               |
| `score` | \`number  | undefined\`                                                         | The cosine similarity of the nearest entry |

The optional `opts` object accepts:

| Option         | Type          | Description                                      |
| -------------- | ------------- | ------------------------------------------------ |
| `trace`        | `TraceHandle` | Records a `cache` span in the Anvil dashboard    |
| `parentSpanId` | `string`      | Parent span for nested trace trees               |
| `now`          | `number`      | Override the current timestamp (useful in tests) |

### `cache.set(prompt, value)`

Stores a response string paired with the prompt's embedding. Call this after a model response to prime future cache hits.

```ts theme={null}
const response = await callLLM(prompt);
await cache.set(prompt, response);
```

### `cache.wrap(prompt, produce, opts?)`

The recommended entry point. Checks the cache, calls `produce()` on a miss, stores the new response, and returns a result object.

```ts theme={null}
const { value, cached } = await cache.wrap(
  userPrompt,
  () => callYourLLM(userPrompt),
  { trace: ctx.trace },
);

console.log(cached ? 'Served from cache' : 'Fresh LLM response');
```

The return value has:

| Field    | Type      | Description                                |
| -------- | --------- | ------------------------------------------ |
| `value`  | `string`  | The response (cached or freshly generated) |
| `cached` | `boolean` | `true` if the response came from the cache |

## Integrating with an agent

Wrap the model call inside `defineAgent` with `cache.wrap()` to cache agent responses by their user query:

```ts theme={null}
import { defineAgent } from 'anvil-sdk/agent';
import { SemanticCache, HashEmbedder } from 'anvil-sdk/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),
  threshold: 0.93,
  ttlMs: 60 * 60 * 1000, // 1 hour
});

export default defineAgent({
  name: 'faq-agent',
  model: 'gpt-4o-mini',
  system: 'You answer frequently asked questions about our product.',

  async run(ctx) {
    const userMessage = ctx.messages.at(-1)?.content as string;

    const { value, cached } = await cache.wrap(
      userMessage,
      () => ctx.complete().then((r) => r.text),
      { trace: ctx.trace },
    );

    ctx.setHeader('x-cache', cached ? 'HIT' : 'MISS');
    return { text: value };
  },
});
```

## Tuning the threshold

The right `threshold` depends on how similar "equivalent" prompts need to be for your use case.

| Threshold | Behavior                                                                       |
| --------- | ------------------------------------------------------------------------------ |
| `0.99`    | Near-exact matches only. Safe for factual Q\&A with precise phrasing.          |
| `0.95`    | Default. Catches minor wording variations like "What's" vs "What is".          |
| `0.90`    | Catches paraphrases and common synonyms. May produce occasional false hits.    |
| `0.80`    | Aggressive caching. Good for narrow-domain FAQs; avoid for open-ended queries. |

<Warning>
  Using a `HashEmbedder` in production limits cache effectiveness because it tracks vocabulary overlap, not semantic meaning. Use a semantic provider embedder (OpenAI `text-embedding-3-small`, Gemini `text-embedding-004`) for accurate similarity scoring.
</Warning>

## Observability

When you pass a `trace` handle to `cache.get()` or `cache.wrap()`, Anvil records a `cache` span that includes:

* `hit` — whether the lookup was a cache hit
* `score` — the cosine similarity of the nearest entry
* `threshold` — the configured threshold

These spans appear in the Anvil dashboard timeline so you can track cache hit rates alongside latency and cost.