Caching

VernLLM ships InMemoryCacheAdapter, NormalizedCacheAdapter, and TieredCacheAdapter out of the box, plus a CacheAdapter interface for bringing your own (Redis, Upstash, or anything else):

interface CacheAdapter<T = unknown> {
  get(key: string): Promise<{ hit: boolean; value: T | null }>;
  set(key: string, value: T, ttl: number): Promise<void>;
  delete?(key: string): Promise<void>;
  /** Optional: for adapters that match on something other than exact string equality. */
  resolveKey?(key: string): Promise<string>;
}

upstash-cache-adapter.ts

import type { CacheAdapter } from 'vern-llm';

class UpstashCacheAdapter implements CacheAdapter {
  async get(key: string) {
    const raw = await redis.get(key);
    if (raw === null) return { hit: false, value: null };
    return { hit: true, value: JSON.parse(raw) };
  }

  async set(key: string, value: unknown, ttl: number) {
    await redis.set(key, JSON.stringify(value), { ex: ttl });
  }

  async delete(key: string) {
    await redis.del(key);
  }
}

const llm = new VernLLM({ client: openai, model: 'gpt-4o', cache: new UpstashCacheAdapter() });

The key thing to get right: hit should be true whenever the key existed in your underlying store, even if the stored value itself is null. hit should only be false when nothing was found for that key.

This adapter matches cacheKey exactly, same as InMemoryCacheAdapter. If you want lookups based on something other than exact string equality, implement resolveKey as well, see Non-exact matching with resolveKey in Core.

The built-in InMemoryCacheAdapter treats ttl as seconds, supports deletion, and accepts an optional maxSize limit to prevent unbounded memory growth. When the limit is reached, the oldest entries are removed first. Custom adapters should keep the same ttl convention (seconds, not milliseconds) so behavior stays consistent across adapters.

If get, set, or delete throws (a Redis connection drop, for example), VernLLM does not fail the call: a broken get is treated as a miss and falls through to a real provider call, and a broken set/delete is swallowed. Either way the failure is reported via logger.warn, not silently ignored, so configure a logger if you want visibility into a misbehaving adapter.

See Caching for the full behavior: concurrent-miss coalescing, resolveKey for fuzzy/semantic matching, the built-in NormalizedCacheAdapter and TieredCacheAdapter, and how failures in a custom adapter are handled.