Multi Provider Fallback

import Anthropic from '@anthropic-ai/sdk';
import OpenAI from 'openai';
import { VernLLM, fromAnthropic } from 'vern-llm';

const llm = new VernLLM({
  client: new OpenAI({ apiKey: process.env.OPENAI_API_KEY }),
  model: 'gpt-4o',
  name: 'openai',
  circuitBreaker: true,

  fallback: [
    {
      client: fromAnthropic(new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY })),
      model: 'claude-sonnet-5',
      name: 'anthropic',
      circuitBreaker: true,
    },
  ],

  onEvent: (event) => {
    if (event.kind === 'fallback') {
      console.warn(`falling over ${event.from} -> ${event.to}`, { error: event.error });
    }
  },
});

const result = await llm.call({ userContent: 'Summarize the earnings call.' });

fallback declares an ordered list of backup targets, tried after the primary and after each earlier target, once its own retries are exhausted or abandoned. Order is the whole policy: VernLLM never reorders, scores, or health checks targets to decide which one goes first. If the primary is listed first, it is tried first, every time, regardless of latency, cost, or recent history.

Declaring targets

interface FallbackTarget {
  client: LLMClient;
  model: string;
  name?: string; // default `fallback[${index}]`

  // Every field below falls back to the parent instance's own option
  // when omitted. circuitBreaker and rateLimit are never inherited.
  maxRetries?: number;
  timeoutMs?: number;
  chunkIdleTimeoutMs?: number;
  baseDelayMs?: number;
  defaultMaxTokens?: number;
  defaultTemperature?: number | null;
  defaultReasoningEffort?: 'minimal' | 'low' | 'medium' | 'high';
  defaultBudgetTokens?: number;
  nonRetryableStatus?: number[];
  circuitBreaker?: boolean | CircuitBreakerOptions;
  rateLimit?: RateLimitOptions;
}

fallback accepts either a single FallbackTarget or an array. Each target needs its own client and model, everything else is optional. Retry and timeout knobs left unset inherit the parent instance's own resolved value, so a target only needs to specify what's actually different about it:

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  maxRetries: 2,
  timeoutMs: 15_000,

  fallback: {
    client: fromGemini(gemini),
    model: 'gemini-3.1-flash-lite',
    // maxRetries inherits the primary's: 2. timeoutMs is overridden
    // below instead of inheriting the primary's 15000.
    timeoutMs: 30_000,
    defaultReasoningEffort: 'low', // this target's own default, independent of the primary's
    defaultBudgetTokens: 4096,
  },
});

circuitBreaker and rateLimit are the exception: they are never inherited from the parent instance. Each target's breaker and limiter, if any, are entirely its own. See

Circuit Breaker and Rate Limiting for their individual behavior.

Every target is independent

Each target, primary and every fallback, gets its own CallExecutor under the hood: its own retry loop, its own per-attempt timeout, its own circuit breaker if configured, and its own rate limiter if configured. Tripping one target's breaker has no effect on any other target's.

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  circuitBreaker: { threshold: 5, cooldownMs: 30_000 },

  fallback: {
    client: fromAnthropic(anthropic),
    model: 'claude-sonnet-5',
    circuitBreaker: { threshold: 3, cooldownMs: 15_000 },
  },
});

An open target breaker is treated as a target failure for fallback purposes. If fallbackOn returns 'next', the chain moves to the next target without waiting for that target's cooldown.

For the full circuit-breaker lifecycle, including cooldowns, half-open trials, per-model isolation, and state inspection, see Circuit Breaker.

Deciding whether to move on

fallbackOn runs once per failed target, after that target's own retries are exhausted or abandoned early, and returns 'next' or 'stop'. 'retry' is never a valid return here, retrying already happened inside the target.

type FallbackOn = (error: LLMError, context: { isLastTarget: boolean }) => 'next' | 'stop';

The default, defaultFallbackOn, is exported so a custom policy can wrap rather than replace it:

import { defaultFallbackOn } from 'vern-llm';

const fallbackOn: FallbackOn = (error, context) => {
  if (error.type === 'circuit_open' && context.isLastTarget) {
    alerting.page('every configured provider is currently unavailable');
  }
  return defaultFallbackOn(error, context);
};

A circuit_open error therefore normally moves the call to the next configured target. This is particularly useful when each target has its own breaker: a provider that is already known to be unhealthy can be skipped immediately rather than consuming another provider attempt.

See Error Handling for the full LLMError type and code reference this table draws from.

Streaming: open failures only

Fallback only applies to a stream failing to open. Once at least one chunk has been delivered to the caller, a mid-stream failure is terminal for that call, VernLLM does not fall over to a different provider partway through:

const { chunks, finalResult } = await llm.call({
  userContent: '...',
  stream: true,
});

If openai's stream never opens (a 5xx, a timeout before the first chunk), the chain falls over to the next target exactly like a non-streaming failure would. If it opens and delivers a few chunks before dying, finalResult rejects directly and no other target is tried.

Knowing which target answered

onUsage reports provider and usedFallback on every successful TokenUsage:

onUsage: (usage) => {
  metrics.increment('llm.call', { provider: usage.provider });
  if (usage.usedFallback) {
    metrics.increment('llm.fallback_used', { provider: usage.provider });
  }
};

For a non-streaming call, meta is an optional out-parameter that's populated synchronously once call() resolves, so provider identity is available on the same line as the result without reaching back into onUsage:

import type { CallMeta } from 'vern-llm';

const meta: { current?: CallMeta } = {};
const result = await llm.call({ userContent: '...', meta });

meta.current;
// {
//   provider: 'anthropic',
//   model: 'claude-sonnet-5',
//   fallbackIndex: 0,      // -1 if the primary answered
//   usedFallback: true,
//   attempts: 1,           // attempts against the target that answered, including the successful one
// }

meta is ignored for stream: true. call() must return { chunks, finalResult } before the real outcome, and so the answering target, is known. Use onUsage for streaming calls instead.

Observing the chain

onEvent reports a fallback event whenever the chain moves to the next target:

{
  kind: 'fallback',
  requestId: string,
  from: string,
  to: string,
  fromIndex: number,
  toIndex: number,
  error: LLMError,
  elapsedMs: number,
}

onEvent: (event) => {
  if (event.kind === 'fallback') {
    metrics.increment('llm.fallback', { from: event.from, to: event.to });
  }
};

The error is the normalized error that caused the chain to move on. This can be a normal provider failure such as a timeout, or circuit_open when the target was already blocked by its circuit breaker.

See Event kinds for the rest of the onEvent union this composes with.

When every target fails

call() throws FallbackExhaustedError, which extends LLMError, so isLLMError and any instanceof LLMError check still passes. It carries attempts, a snapshot of every target's own error, in order:

import { FallbackExhaustedError, isLLMError } from 'vern-llm';

try {
  await llm.call({ userContent: '...' });
} catch (err) {
  if (err instanceof FallbackExhaustedError) {
    for (const attempt of err.attempts) {
      console.error(`${attempt.provider} failed`, {
        type: attempt.error.type,
        code: attempt.error.code,
      });
    }
  } else if (isLLMError(err)) {
    // A single target (or fallbackOn stopped early) threw its own error
    // directly.
  }
}

A lone target with no fallback configured throws exactly what it threw before this option existed, its own normalized LLMError, never wrapped in FallbackExhaustedError. The same is true whenever fallbackOn returns 'stop' on the very first failure: only one target was ever tried, so there's nothing to aggregate.

cachedCall

cachedCall composes with fallback automatically, no extra wiring needed. Fallback lives inside call(), which cachedCall already wraps, so the whole chain caches under one key and the successful result, however far down the chain it came from, is what gets stored. In-flight coalescing for concurrent misses on the same cacheKey covers the full chain too. See Caching.

Options reference

See Configuration for how fallback sits alongside every other constructor option, and

Provider Fallback Patterns in Guides for putting this to use effectively.