> ## Documentation Index
> Fetch the complete documentation index at: https://anvil.thatdevguy.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Semantic Cache: Skip Redundant LLM Calls by Similarity

> Use SemanticCache to skip redundant LLM calls when a semantically equivalent prompt has already been answered, cutting latency and token spend.

LLMs are expensive. When users ask questions that are semantically equivalent — "What's your return policy?" and "How do I return an item?" — there is no reason to run two separate model calls. `SemanticCache` stores responses keyed on embedding similarity rather than exact string matching, so a new prompt that is close enough to a cached one returns the stored answer instantly without touching the model.

## How it works

When you call `cache.get(prompt)`, `SemanticCache` embeds the prompt and queries the backing vector store for the nearest stored entry. If the top result's cosine similarity exceeds the configured `threshold` (default `0.95`) and the entry has not expired, it's a cache hit and the stored response is returned. On a miss, your code calls the LLM as normal, then stores the response with `cache.set()` for future hits.

The `cache.wrap()` convenience method handles the full get-or-produce flow in one call.

## Setup

```ts theme={null}
import { SemanticCache, HashEmbedder, SqliteVectorStore } from 'anvil-sdk/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),          // swap for a semantic embedder in production
  store: await SqliteVectorStore.open(), // persists hits across restarts
  threshold: 0.92,                       // tune for your use case
  ttlMs: 1000 * 60 * 60 * 24,           // 24-hour TTL
});
```

### Constructor options

<Accordion title="SemanticCacheOptions">
  | Option | Type | Default | Description |
  | - | - | - | - |
  | `embedder` | `Embedder` | — (required) | Embeds prompts for similarity comparison |
  | `threshold` | `number` | `0.95` | Minimum cosine similarity for a cache hit. Lower values cast a wider net; higher values require near-exact matches |
  | `store` | `VectorStore` | `MemoryVectorStore` | Backing store for cached `(embedding, response)` pairs |
  | `ttlMs` | `number` | none | Entry lifetime in milliseconds. Entries older than this are treated as misses |
</Accordion>

<Note>
  The default `store` is an in-process `MemoryVectorStore`. Use `SqliteVectorStore` or another persistent store to keep the cache across process restarts.
</Note>

## API

### `cache.get(prompt, opts?)`

Looks up a prompt in the cache. Returns a `CacheLookup` object.

```ts theme={null}
const result = await cache.get('What is your return policy?');

if (result.hit) {
  console.log('Cache hit! Score:', result.score);
  return result.value; // the stored response string
}
```

`CacheLookup` fields:

| Field | Type | Description | |
| - | - | - | - |
| `hit` | `boolean` | `true` if a sufficiently similar entry was found and is still fresh | |
| `value` | \`string | undefined\` | The cached response on a hit |
| `score` | \`number | undefined\` | The cosine similarity of the nearest entry |

The optional `opts` object accepts:

| Option | Type | Description |
| - | - | - |
| `trace` | `TraceHandle` | Records a `cache` span in the Anvil dashboard |
| `parentSpanId` | `string` | Parent span for nested trace trees |
| `now` | `number` | Override the current timestamp (useful in tests) |

### `cache.set(prompt, value)`

Stores a response string paired with the prompt's embedding. Call this after a model response to prime future cache hits.

```ts theme={null}
const response = await callLLM(prompt);
await cache.set(prompt, response);
```

### `cache.wrap(prompt, produce, opts?)`

The recommended entry point. Checks the cache, calls `produce()` on a miss, stores the new response, and returns a result object.

```ts theme={null}
const { value, cached } = await cache.wrap(
  userPrompt,
  () => callYourLLM(userPrompt),
  { trace: ctx.trace },
);

console.log(cached ? 'Served from cache' : 'Fresh LLM response');
```

The return value has:

| Field | Type | Description |
| - | - | - |
| `value` | `string` | The response (cached or freshly generated) |
| `cached` | `boolean` | `true` if the response came from the cache |

## Integrating with an agent

Wrap the model call inside `defineAgent` with `cache.wrap()` to cache agent responses by their user query:

```ts theme={null}
import { defineAgent } from 'anvil-sdk/agent';
import { SemanticCache, HashEmbedder } from 'anvil-sdk/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),
  threshold: 0.93,
  ttlMs: 60 * 60 * 1000, // 1 hour
});

export default defineAgent({
  name: 'faq-agent',
  model: 'gpt-4o-mini',
  system: 'You answer frequently asked questions about our product.',

  async run(ctx) {
    const userMessage = ctx.messages.at(-1)?.content as string;

    const { value, cached } = await cache.wrap(
      userMessage,
      () => ctx.complete().then((r) => r.text),
      { trace: ctx.trace },
    );

    ctx.setHeader('x-cache', cached ? 'HIT' : 'MISS');
    return { text: value };
  },
});
```

## Tuning the threshold

The right `threshold` depends on how similar "equivalent" prompts need to be for your use case.

| Threshold | Behavior |
| - | - |
| `0.99` | Near-exact matches only. Safe for factual Q\&A with precise phrasing. |
| `0.95` | Default. Catches minor wording variations like "What's" vs "What is". |
| `0.90` | Catches paraphrases and common synonyms. May produce occasional false hits. |
| `0.80` | Aggressive caching. Good for narrow-domain FAQs; avoid for open-ended queries. |

<Warning>
  Using a `HashEmbedder` in production limits cache effectiveness because it tracks vocabulary overlap, not semantic meaning. Use a semantic provider embedder (OpenAI `text-embedding-3-small`, Gemini `text-embedding-004`) for accurate similarity scoring.
</Warning>

## Observability

When you pass a `trace` handle to `cache.get()` or `cache.wrap()`, Anvil records a `cache` span that includes:

* `hit` — whether the lookup was a cache hit
* `score` — the cosine similarity of the nearest entry
* `threshold` — the configured threshold

These spans appear in the Anvil dashboard timeline so you can track cache hit rates alongside latency and cost.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.