Caching & Usage Metering

import { InMemoryCacheAdapter } from 'vern-llm';

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  cache: new InMemoryCacheAdapter(), // swap for your own Redis/Upstash adapter
});

const result = await llm.cachedCall({
  cacheKey: `cv:${cvId}`,
  ttl: 3600,
  fn: () => llm.call({ systemPrompt, userContent }),
  reserveUsage: () => quota.reserve(userId),
  refundUsage: () => quota.refund(userId),
});

ttl is in seconds, not milliseconds. ttl: 3600 caches for one hour. This differs from most other timing options in the library (timeoutMs, baseDelayMs), which are milliseconds, so it's worth double-checking if you're copying a value across.

If refundUsage itself throws, that failure is logged and swallowed. The original error from fn (or the cache write) is what propagates, so a broken refund path never masks the real failure.

How the cache/reserve/refund flow works

On a cache miss, cachedCall runs in this order:

reserveUsage(), if provided.
fn(), the actual LLM call.
On success, write the result to the cache (a failed cache write is logged and swallowed; the result is still returned).
On any failure in steps 1 or 2, call refundUsage(), then rethrow the original error.

refundUsage runs on a failure from either reserveUsage or fn, since both are inside the same try block. If reserveUsage itself throws (e.g. the user is already out of quota), refundUsage still fires, refunding a reservation that was never actually made. Make sure your refundUsage implementation is safe to call in that case (e.g. a no-op if there's nothing to refund, rather than an unconditional decrement).

A cache hit is determined by cached !== null. If your cache adapter can legitimately store null as a cached value, that entry will be treated as a miss and recomputed every time.

Deleting cached entries

Cache invalidation is application-specific, so VernLLM does not automatically decide when a cached response becomes stale. When your application knows that a cached response should no longer be used, you can remove the entry manually:

await llm.deleteCache(`cv:${cvId}`);

`cachedLLMCall`: cached + retried in one call

cachedCall is a thin cache wrapper; it doesn't apply retry/timeout/circuit-breaker behavior on its own. If fn is always going to be () => llm.call(...), cachedLLMCall wires that up for you:

const result = await llm.cachedLLMCall({
  cacheKey: `cv:${cvId}`,
  ttl: 3600,
  call: { systemPrompt, userContent }, // same shape as call()'s params
  reserveUsage: () => quota.reserve(userId),
  refundUsage: () => quota.refund(userId),
});

Choosing a cache key

There's no automatic key derivation from the call's contents. cacheKey is entirely up to you, which means it's also easy to get wrong: if you reuse a key across calls with a different systemPrompt, schema, or model, you'll get a cached result that doesn't match what the new call would have produced.

A safer pattern is to fold the things that affect the output into the key itself:

const cacheKey = `cv:${cvId}:${model}:v2`; // bump the version suffix when the prompt/schema changes

Custom cache adapter

import type { CacheAdapter } from 'vern-llm';

class UpstashCacheAdapter implements CacheAdapter {
  async get(key: string) {
    /* ... */
  }

  async set(key: string, value: unknown, ttl: number) {
    /* ... */
  }

  async delete(key: string) {
    /* ... */
  }
}

The built-in InMemoryCacheAdapter treats ttl as seconds and supports deletion. Custom adapters should keep the same ttl convention (rather than milliseconds) so behavior stays consistent across adapters. Implement delete if your storage backend supports explicit cache invalidation.