Provider Fallback Patterns
Ordering targets, sizing per-target retries, and reading a fallback chain's signals
Provider Fallback covers the fallback option itself,
target inheritance, fallbackOn, and every event it emits. This guide walks through putting it to
use: choosing an order, sizing each target's own resilience knobs, and turning the signals fallback
emits into something actionable instead of noise.
Every snippet below assumes the same setup: an OpenAI primary, and Anthropic and Groq clients
wrapped for fallback.
import Anthropic from '@anthropic-ai/sdk';
import OpenAI from 'openai';
import { VernLLM, fromAnthropic, fromOpenAI, fromOpenAICompatible } from 'vern-llm';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, maxRetries: 0 });
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const groq = new OpenAI({
apiKey: process.env.GROQ_API_KEY,
baseURL: 'https://api.groq.com/openai/v1',
maxRetries: 0,
});Every client below has its own SDK-level maxRetries: 0. That's independent of VernLLM's own
maxRetries: without it, the raw client would silently retry internally before VernLLM's retry
loop or fallbackOn ever sees the failure, which would undercut the "fails fast" and "falls over
immediately" behavior every example below is demonstrating.
Order by what you'd actually accept, not by preference
The order you declare is the order VernLLM tries, every time, with no scoring or health-checking in between. That means the list itself is a statement about tradeoffs you've already made, not just a ranked wishlist:
const llm = new VernLLM({
client: fromOpenAI(openai), // fastest, cheapest, what you'd pick if nothing were down
model: 'gpt-4o',
name: 'openai',
circuitBreaker: true,
fallback: [
{
// A close substitute: similar quality, similar latency, different
// infrastructure. The obvious next try when openai is unavailable.
client: fromAnthropic(anthropic),
model: 'claude-sonnet-5',
name: 'anthropic',
circuitBreaker: true,
},
{
// A genuinely different tradeoff: cheaper and faster, but you'd only
// accept its answers when nothing better is reachable. Listed last
// on purpose.
client: fromOpenAICompatible(groq),
model: 'llama-3.3-70b',
name: 'groq-last-resort',
},
],
});Don't list a materially worse or cheaper model ahead of a better one just because it happens to respond faster on average. VernLLM will try it first every time your primary is down, even for a request where that substitution isn't acceptable. If quality matters more than speed for a given call, put the closer substitute first and accept the extra latency on the rare path where it's needed.
Why fallback never picks a provider for you
It can be tempting to want VernLLM to route to "whichever provider is healthiest right now" or
"whichever is cheapest this month." It deliberately doesn't, and understanding why makes the
fallback list easier to reason about:
- The list is the whole policy. VernLLM never reorders it based on latency, cost, or recent
failures. If you want dynamic routing, that's a decision your application makes by rebuilding the
list (or swapping which
VernLLMinstance you call), not something to configure on VernLLM itself. fallbackOndecides "was this worth trying elsewhere," not "who should answer." It classifies the failure that just happened; it never inspects the other targets to decide which one to try next. The next target is always just the next one in the list.- Every target you declare is one VernLLM will actually contact. There's no provider registry or
auto-discovery step. If a client and model aren't in
fallback, VernLLM will never call them.
This keeps a fallback chain auditable: reading the fallback array tells you the complete set of
providers a given call could ever reach, in the exact order it could reach them.
Give each target retry room proportional to how much you trust it
maxRetries, timeoutMs, and baseDelayMs are per target and inherit from the parent instance
when omitted. Don't just let every target inherit the primary's numbers uncritically, think about
how much time you're willing to spend on a target before moving on:
const llm = new VernLLM({
client: fromOpenAI(openai),
model: 'gpt-4o',
maxRetries: 2, // give the primary a real chance: this is who you actually want answering
timeoutMs: 20_000,
fallback: [
{
client: fromAnthropic(anthropic),
model: 'claude-sonnet-5',
// A close substitute still deserves a couple of retries; a
// transient failure here shouldn't skip straight to the last resort.
maxRetries: 2,
},
{
client: fromOpenAICompatible(groq),
model: 'llama-3.3-70b',
// The last resort: fail fast, don't burn wall-clock time retrying
// a target you only reach when everything else is already down.
maxRetries: 0,
timeoutMs: 8_000,
},
],
});Every retry against a target is a real request against that provider's own limits. A target with a
generous maxRetries and no circuitBreaker can end up absorbing a lot of wall-clock time before
the chain ever reaches the next one. See Backoff and
jitter for how the per-attempt delay is computed.
Give every target its own circuit breaker
A target without circuitBreaker retries from scratch, and gets checked from scratch, on every
single call, even one that's currently down. That's usually the wrong default for anything beyond a
last-resort target you rarely expect to reach:
const llm = new VernLLM({
client: fromOpenAI(openai),
model: 'gpt-4o',
circuitBreaker: { threshold: 5, cooldownMs: 30_000 },
fallback: {
client: fromAnthropic(anthropic),
model: 'claude-sonnet-5',
// Independent from the primary's breaker: tripping this one has zero
// effect on the primary, and vice versa.
circuitBreaker: { threshold: 5, cooldownMs: 30_000 },
},
});Once a target's breaker opens, every call against it fails immediately with
LLMError('circuit_open') instead of paying for a timeout, and fallbackOn treats that exactly
like any other failure: it moves straight on to the next target. This is what makes a longer
fallback chain cheap to keep configured even when the earlier targets are actually down for a
while, you pay the full retry/timeout cost once, then the breaker keeps every subsequent call fast
until the cooldown elapses.
getCircuitState() only reflects the primary's breaker. For a snapshot of every target's state at
once, call getCircuitStates() instead, it returns each target's provider name, chain index, and
circuit state in one array. For real-time notification the moment a target's circuit transitions,
watch the circuit_state event on onEvent, it carries provider so you can tell targets apart.
See circuit_state.
Turn onEvent's fallback event into an alert, not just a log line
A single fallback event is normal, transient noise is exactly what fallback exists to absorb. A
sustained pattern of falling all the way to your last-resort target is worth paging on, before
your users notice it as degraded quality rather than an outage:
const llm = new VernLLM({
client: fromOpenAI(openai),
model: 'gpt-4o',
name: 'openai',
fallback: [
{ client: fromAnthropic(anthropic), model: 'claude-sonnet-5', name: 'anthropic' },
{ client: fromOpenAICompatible(groq), model: 'llama-3.3-70b', name: 'groq-last-resort' },
],
onEvent: (event) => {
if (event.kind !== 'fallback') return;
metrics.increment('llm.fallback', { from: event.from, to: event.to });
if (event.to === 'groq-last-resort') {
// Reaching the last resort means both better options just failed.
alerting.page(`llm chain degraded to last resort, last failure: ${event.error.type}`);
}
},
});Pair this with TokenUsage.usedFallback on onUsage for a second, complementary signal: onEvent
tells you a fallover happened, onUsage tells you which provider actually answered a given
request, useful for billing or quality dashboards that care about the outcome rather than the path
that got there.
onUsage: (usage) => {
if (usage.usedFallback) {
metrics.increment('llm.answered_by_fallback', { provider: usage.provider });
}
};Handle FallbackExhaustedError distinctly from a single-provider failure
Catching FallbackExhaustedError specifically, rather than only isLLMError, lets you show a
different message (and log a richer signal) for "every provider we tried is down" versus "the one
provider we tried returned something we couldn't use":
import { FallbackExhaustedError, isLLMError } from 'vern-llm';
try {
return await llm.call({ userContent: message });
} catch (err) {
if (err instanceof FallbackExhaustedError) {
console.error('every configured provider failed', {
attempts: err.attempts.map((a) => ({ provider: a.provider, type: a.error.type })),
});
return {
message: "We're having trouble reaching any provider right now, please try again shortly.",
};
}
if (isLLMError(err) && err.type === 'validation') {
// A single target answered but its response didn't validate.
// fallbackOn already decided this wasn't worth trying elsewhere.
return { message: 'Something went wrong processing that request.' };
}
throw err;
}Don't retry a caught FallbackExhaustedError yourself in a tight loop. Every target already
exhausted its own retries and, if configured, tripped its own breaker; an immediate re-call just
repeats the same chain against providers that are still cooling down. Wait a meaningful amount of
time, or rely on the circuit breakers you configured per target to recover on their own once each
target's cooldownMs elapses.
Rotating API keys is the same mechanism, no separate feature needed
A "fallback target" doesn't have to be a different provider. Two targets on the same provider
with different API keys, using the default fallbackOn (which treats a rate-limited or otherwise
failing target as 'next'), gives you key rotation for free:
const llm = new VernLLM({
client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_KEY_PRIMARY, maxRetries: 0 })),
model: 'gpt-4o',
name: 'openai-key-a',
// 429 is retryable by default, which would burn a full backoff cycle
// against a key that's already over its limit. Treating it as
// non-retryable here means the first 429 moves straight to key b.
nonRetryableStatus: [429],
fallback: {
client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_KEY_SECONDARY, maxRetries: 0 })),
model: 'gpt-4o',
name: 'openai-key-b',
},
});When openai-key-a gets rate limited, nonRetryableStatus skips straight to fallbackOn instead
of retrying against the same over-limit key, and the default policy moves on to openai-key-b
rather than waiting out a backoff cycle that can't help. No separate feature or wrapper is needed,
it's the exact same ordered-list mechanism this whole guide has been about.
Test the chain by breaking the primary on purpose
Before relying on a fallback chain in production, verify it actually falls over the way you expect, rather than discovering the behavior for the first time during a real outage. Point the primary at an invalid endpoint or an intentionally wrong API key and confirm the call still succeeds:
const llm = new VernLLM({
client: fromOpenAI(
new OpenAI({
apiKey: 'intentionally-invalid',
baseURL: 'https://invalid.test',
maxRetries: 0,
}),
),
model: 'gpt-4o',
maxRetries: 0, // fail fast for the test, don't wait out real backoff
timeoutMs: 3_000,
fallback: {
client: fromAnthropic(anthropic), // a real, working client
model: 'claude-sonnet-5',
},
});
const result = await llm.call({ userContent: 'ping', jsonMode: false });
// Should succeed, answered by the fallback target.This also exercises the exact wire request your fallback target will receive under a real outage,
including tools and jsonSchema, since VernLLM builds the same request shape for every target.