Skip to main content
LLMs are expensive. When users ask questions that are semantically equivalent — “What’s your return policy?” and “How do I return an item?” — there is no reason to run two separate model calls. SemanticCache stores responses keyed on embedding similarity rather than exact string matching, so a new prompt that is close enough to a cached one returns the stored answer instantly without touching the model.

​
How it works

When you call cache.get(prompt), SemanticCache embeds the prompt and queries the backing vector store for the nearest stored entry. If the top result’s cosine similarity exceeds the configured threshold (default 0.95) and the entry has not expired, it’s a cache hit and the stored response is returned. On a miss, your code calls the LLM as normal, then stores the response with cache.set() for future hits. The cache.wrap() convenience method handles the full get-or-produce flow in one call.

​
Setup

import { SemanticCache, HashEmbedder, SqliteVectorStore } from 'anvil/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),          // swap for a semantic embedder in production
  store: await SqliteVectorStore.open(), // persists hits across restarts
  threshold: 0.92,                       // tune for your use case
  ttlMs: 1000 * 60 * 60 * 24,           // 24-hour TTL
});

​
Constructor options

OptionTypeDefaultDescription
embedderEmbedder— (required)Embeds prompts for similarity comparison
thresholdnumber0.95Minimum cosine similarity for a cache hit. Lower values cast a wider net; higher values require near-exact matches
storeVectorStoreMemoryVectorStoreBacking store for cached (embedding, response) pairs
ttlMsnumbernoneEntry lifetime in milliseconds. Entries older than this are treated as misses
The default store is an in-process MemoryVectorStore. Use SqliteVectorStore or another persistent store to keep the cache across process restarts.

​
API

​
cache.get(prompt, opts?)

Looks up a prompt in the cache. Returns a CacheLookup object. 
const result = await cache.get('What is your return policy?');

if (result.hit) {
  console.log('Cache hit! Score:', result.score);
  return result.value; // the stored response string
}
CacheLookup fields:
FieldTypeDescription
hitbooleantrue if a sufficiently similar entry was found and is still fresh
valuestring | undefinedThe cached response on a hit
scorenumber | undefinedThe cosine similarity of the nearest entry
The optional opts object accepts:
OptionTypeDescription
traceTraceHandleRecords a cache span in the Anvil dashboard
parentSpanIdstringParent span for nested trace trees
nownumberOverride the current timestamp (useful in tests)

​
cache.set(prompt, value)

Stores a response string paired with the prompt’s embedding. Call this after a model response to prime future cache hits. 
const response = await callLLM(prompt);
await cache.set(prompt, response);

​
cache.wrap(prompt, produce, opts?)

The recommended entry point. Checks the cache, calls produce() on a miss, stores the new response, and returns a result object. 
const { value, cached } = await cache.wrap(
  userPrompt,
  () => callYourLLM(userPrompt),
  { trace: ctx.trace },
);

console.log(cached ? 'Served from cache' : 'Fresh LLM response');
The return value has:
FieldTypeDescription
valuestringThe response (cached or freshly generated)
cachedbooleantrue if the response came from the cache

​
Integrating with an agent

Wrap the model call inside defineAgent with cache.wrap() to cache agent responses by their user query: 
import { defineAgent } from 'anvil/agent';
import { SemanticCache, HashEmbedder } from 'anvil/rag';

const cache = new SemanticCache({
  embedder: new HashEmbedder(),
  threshold: 0.93,
  ttlMs: 60 * 60 * 1000, // 1 hour
});

export default defineAgent({
  name: 'faq-agent',
  model: 'gpt-4o-mini',
  system: 'You answer frequently asked questions about our product.',

  async run(ctx) {
    const userMessage = ctx.messages.at(-1)?.content as string;

    const { value, cached } = await cache.wrap(
      userMessage,
      () => ctx.complete().then((r) => r.text),
      { trace: ctx.trace },
    );

    ctx.setHeader('x-cache', cached ? 'HIT' : 'MISS');
    return { text: value };
  },
});

​
Tuning the threshold

The right threshold depends on how similar “equivalent” prompts need to be for your use case.
ThresholdBehavior
0.99Near-exact matches only. Safe for factual Q&A with precise phrasing.
0.95Default. Catches minor wording variations like “What’s” vs “What is”.
0.90Catches paraphrases and common synonyms. May produce occasional false hits.
0.80Aggressive caching. Good for narrow-domain FAQs; avoid for open-ended queries.
Using a HashEmbedder in production limits cache effectiveness because it tracks vocabulary overlap, not semantic meaning. Use a semantic provider embedder (OpenAI text-embedding-3-small, Gemini text-embedding-004) for accurate similarity scoring.

​
Observability

When you pass a trace handle to cache.get() or cache.wrap(), Anvil records a cache span that includes:
  • hit — whether the lookup was a cache hit
  • score — the cosine similarity of the nearest entry
  • threshold — the configured threshold
These spans appear in the Anvil dashboard timeline so you can track cache hit rates alongside latency and cost.