Tiered Caching

InMemoryCacheAdapter is fast but per-process; a Redis/Upstash-backed adapter is shared across processes but adds network latency on every lookup. TieredCacheAdapter gets both: it checks a fast local cache first, and only falls through to the shared store on a local miss.

Usage

import { InMemoryCacheAdapter, TieredCacheAdapter, VernLLM } from 'vern-llm';

const cache = new TieredCacheAdapter(
  new InMemoryCacheAdapter(), // L1: fast, per-process
  new UpstashCacheAdapter(), // L2: shared across processes, slower
);

const llm = new VernLLM({ client: openai, model: 'gpt-4o', cache });

How lookups flow

async get(key) {
  const l1 = await this.l1.get(key);
  if (l1.hit) return l1; // fast path, no L2 round trip

  const l2 = await this.l2.get(key);
  if (l2.hit) await this.l1.set(key, l2.value, l1Ttl); // backfill so next lookup is fast

  return l2;
}

• L1 hit: returns immediately, no L2 call at all.
• L1 miss, L2 hit: value is written back into L1 before returning, so the next lookup for that key on this process hits L1 and skips L2 entirely.
• Both miss: returns a miss, same as any other adapter.

set() writes to both tiers at once (in parallel, via Promise.all), so a fresh write is immediately visible to both a local get() and any other process reading L2 directly.

Controlling L1's TTL separately

By default, TieredCacheAdapter uses the same ttl for both tiers. Pass a third constructor argument to give L1 a shorter lifetime than L2, useful when L1 is memory-constrained and you'd rather it churn faster than the shared store:

const cache = new TieredCacheAdapter(
  new InMemoryCacheAdapter(),
  new UpstashCacheAdapter(),
  60, // L1 entries expire after 60s regardless of the ttl passed to cachedCall
);

Combining with resolveKey-based matching

The simplest way to combine tiering with normalized or semantic matching is to put the matching adapter directly in one of the tier slots. Putting it at L1 is usually the right call, since L1 is checked first and its resolveKey is the one TieredCacheAdapter will use:

import { NormalizedCacheAdapter, TieredCacheAdapter } from 'vern-llm';

const cache = new TieredCacheAdapter(
  new NormalizedCacheAdapter(), // L1: fast, normalizes keys, resolveKey lives here
  new UpstashCacheAdapter(), // L2: shared, exact match on the already-normalized key
);

cachedCall calls cache.resolveKey(key) once. TieredCacheAdapter forwards that to L1's NormalizedCacheAdapter, and the resolved (normalized) key is what's then used for both the L1 and L2 lookups.

If you'd rather keep the tiered pair fully exact-match internally and layer matching on top instead, wrapping still works the same way it always did, put the matching adapter outermost, with the tiered pair as its inner store:

import { InMemoryCacheAdapter, NormalizedCacheAdapter, TieredCacheAdapter } from 'vern-llm';

const tiered = new TieredCacheAdapter(new InMemoryCacheAdapter(), new UpstashCacheAdapter());
const cache = new NormalizedCacheAdapter(tiered); // resolveKey lives here, storage happens in `tiered`

Both patterns land on the same resolved key by the time get/set reach the tiers, they're functionally equivalent for this case. The same pattern works for a custom semantic adapter: pass it as a tier directly, or wrap the tiered pair with it, rather than trying to make TieredCacheAdapter itself embedding-aware.