Semantic Cache: Skip Redundant LLM Calls by Similarity

LLMs are expensive. When users ask questions that are semantically equivalent — “What’s your return policy?” and “How do I return an item?” — there is no reason to run two separate model calls. SemanticCache stores responses keyed on embedding similarity rather than exact string matching, so a new prompt that is close enough to a cached one returns the stored answer instantly without touching the model.

How it works

When you call cache.get(prompt), SemanticCache embeds the prompt and queries the backing vector store for the nearest stored entry. If the top result’s cosine similarity exceeds the configured threshold (default 0.95) and the entry has not expired, it’s a cache hit and the stored response is returned. On a miss, your code calls the LLM as normal, then stores the response with cache.set() for future hits. The cache.wrap() convenience method handles the full get-or-produce flow in one call.

Setup

import { SemanticCache, HashEmbedder, SqliteVectorStore } from 'anvil-sdk/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),          // swap for a semantic embedder in production
  store: await SqliteVectorStore.open(), // persists hits across restarts
  threshold: 0.92,                       // tune for your use case
  ttlMs: 1000 * 60 * 60 * 24,           // 24-hour TTL
});

Constructor options

SemanticCacheOptions

Option	Type	Default	Description
`embedder`	`Embedder`	— (required)	Embeds prompts for similarity comparison
`threshold`	`number`	`0.95`	Minimum cosine similarity for a cache hit. Lower values cast a wider net; higher values require near-exact matches
`store`	`VectorStore`	`MemoryVectorStore`	Backing store for cached `(embedding, response)` pairs
`ttlMs`	`number`	none	Entry lifetime in milliseconds. Entries older than this are treated as misses

The default store is an in-process MemoryVectorStore. Use SqliteVectorStore or another persistent store to keep the cache across process restarts.

API

`cache.get(prompt, opts?)`

Looks up a prompt in the cache. Returns a CacheLookup object.

const result = await cache.get('What is your return policy?');

if (result.hit) {
  console.log('Cache hit! Score:', result.score);
  return result.value; // the stored response string
}

CacheLookup fields:

Field	Type	Description
`hit`	`boolean`	`true` if a sufficiently similar entry was found and is still fresh
`value`	`string	undefined`	The cached response on a hit
`score`	`number	undefined`	The cosine similarity of the nearest entry

The optional opts object accepts:

Option	Type	Description
`trace`	`TraceHandle`	Records a `cache` span in the Anvil dashboard
`parentSpanId`	`string`	Parent span for nested trace trees
`now`	`number`	Override the current timestamp (useful in tests)

`cache.set(prompt, value)`

Stores a response string paired with the prompt’s embedding. Call this after a model response to prime future cache hits.

const response = await callLLM(prompt);
await cache.set(prompt, response);

`cache.wrap(prompt, produce, opts?)`

The recommended entry point. Checks the cache, calls produce() on a miss, stores the new response, and returns a result object.

const { value, cached } = await cache.wrap(
  userPrompt,
  () => callYourLLM(userPrompt),
  { trace: ctx.trace },
);

console.log(cached ? 'Served from cache' : 'Fresh LLM response');

The return value has:

Field	Type	Description
`value`	`string`	The response (cached or freshly generated)
`cached`	`boolean`	`true` if the response came from the cache

Integrating with an agent

Wrap the model call inside defineAgent with cache.wrap() to cache agent responses by their user query:

import { defineAgent } from 'anvil-sdk/agent';
import { SemanticCache, HashEmbedder } from 'anvil-sdk/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),
  threshold: 0.93,
  ttlMs: 60 * 60 * 1000, // 1 hour
});

export default defineAgent({
  name: 'faq-agent',
  model: 'gpt-4o-mini',
  system: 'You answer frequently asked questions about our product.',

  async run(ctx) {
    const userMessage = ctx.messages.at(-1)?.content as string;

    const { value, cached } = await cache.wrap(
      userMessage,
      () => ctx.complete().then((r) => r.text),
      { trace: ctx.trace },
    );

    ctx.setHeader('x-cache', cached ? 'HIT' : 'MISS');
    return { text: value };
  },
});

Tuning the threshold

The right threshold depends on how similar “equivalent” prompts need to be for your use case.

Threshold	Behavior
`0.99`	Near-exact matches only. Safe for factual Q&A with precise phrasing.
`0.95`	Default. Catches minor wording variations like “What’s” vs “What is”.
`0.90`	Catches paraphrases and common synonyms. May produce occasional false hits.
`0.80`	Aggressive caching. Good for narrow-domain FAQs; avoid for open-ended queries.

Using a HashEmbedder in production limits cache effectiveness because it tracks vocabulary overlap, not semantic meaning. Use a semantic provider embedder (OpenAI text-embedding-3-small, Gemini text-embedding-004) for accurate similarity scoring.

Observability

When you pass a trace handle to cache.get() or cache.wrap(), Anvil records a cache span that includes:

hit — whether the lookup was a cache hit
score — the cosine similarity of the nearest entry
threshold — the configured threshold

These spans appear in the Anvil dashboard timeline so you can track cache hit rates alongside latency and cost.

Get Started

Routing

MCP & Tools

Agent Routes

Observability

Safety & Durability

Memory & RAG

Background Tasks

Semantic Cache: Skip Redundant LLM Calls by Similarity

How it works

Setup

Constructor options

API

`cache.get(prompt, opts?)`

`cache.set(prompt, value)`

`cache.wrap(prompt, produce, opts?)`

Integrating with an agent

Tuning the threshold

Observability

​How it works

​Setup

​Constructor options

​API

​cache.get(prompt, opts?)

​cache.set(prompt, value)

​cache.wrap(prompt, produce, opts?)

​Integrating with an agent

​Tuning the threshold

​Observability

How it works

Setup

Constructor options

API

`cache.get(prompt, opts?)`

`cache.set(prompt, value)`

`cache.wrap(prompt, produce, opts?)`

Integrating with an agent

Tuning the threshold

Observability