Tuning Rate Limits

Rate Limiting covers the rateLimit option itself, the three buckets, queueing, and the events it emits. This guide walks through putting it to use: sizing it against numbers a provider actually gives you, sharing one budget across several routes, and deciding what to do when a call gives up waiting.

Start from your plan's real numbers, not a guess

Every major provider dashboard publishes requests-per-minute and tokens-per-minute limits per model per tier. Set rateLimit a little under those, not at them exactly, so VernLLM's own request overhead and estimation slack still land you under the real ceiling:

// Provider dashboard says: gpt-4o, tier 2 → 5,000 RPM / 800,000 TPM
const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: {
    requestsPerMinute: 4_500, // ~10% headroom
    tokensPerMinute: 720_000, // ~10% headroom
    maxConcurrent: 100,
  },
});

maxConcurrent isn't published anywhere, it's not a rate the provider enforces, it's a knob for your own outbound connection pool and downstream memory. Size it against how many concurrent requests your process (or the connection pool underneath your HTTP client) can actually sustain, not the provider's RPM number.

One instance, one budget, many callers

rateLimit is per VernLLM instance, not global. If several routes or background jobs in your app share one provider API key, route them all through the same instance so they share one budget, instead of creating a new VernLLM (and therefore a fresh, unaware set of buckets) per call site:

// shared.ts, imported everywhere that talks to this provider
export const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: { requestsPerMinute: 4_500, tokensPerMinute: 720_000, maxConcurrent: 100 },
});

import { llm } from './shared.js';

export async function summarizeHandler(req: Request) {
  return llm.call({ userContent: req.body.text });
}

Two separate VernLLM instances pointed at the same provider account have two separate, unaware buckets between them, and can together exceed the account's real limit even though each instance individually stays under its own configured number.

If you run multiple providers behind one logical service, fallback targets each get their own rateLimit, sized against that provider's own numbers; it's never inherited from the primary. See

Provider Fallback for declaring several targets on one instance.

Choosing maxQueueMs for your call site

maxQueueMs is the most important knob to actually think about, because its right value depends on what's waiting on the other end of the call:

// An interactive chat request: a user is staring at a spinner. Fail fast
// and show them something rather than let them wait out a long queue.
const chatLlm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: { requestsPerMinute: 500, maxQueueMs: 3_000 },
});

// A background batch job processing a queue of documents overnight.
// Nobody's watching a spinner; let it wait out the rate limit instead
// of failing and needing its own retry-with-backoff logic bolted on.
const batchLlm = new VernLLM({
  client: openai,
  model: 'gpt-4o-mini',
  rateLimit: { requestsPerMinute: 500, maxQueueMs: 0 }, // wait indefinitely
});

For the interactive case, catch the local rate-limit failure and degrade gracefully instead of surfacing a raw error. type: 'rate_limited' covers only VernLLM's own local rateLimit queue rejecting the call; a provider's own HTTP 429 is type: 'api' with code: 'provider_rate_limited' instead. rate_limit_queue_full/rate_limit_queue_timeout specifically mean the local queue gave up:

import { isLLMError } from 'vern-llm';

try {
  return await chatLlm.call({ userContent: message });
} catch (err) {
  if (
    isLLMError(err) &&
    (err.code === 'rate_limit_queue_full' || err.code === 'rate_limit_queue_timeout')
  ) {
    return { message: "We're a little busy right now, please try again in a moment." };
  }
  throw err;
}

Telling a local wait apart from a real provider 429

Both a local queue giving up and an actual provider rate limit surface as errors, but they mean different things operationally: one says your own configured ceiling was too tight, the other says the provider itself pushed back.

import { isLLMError } from 'vern-llm';

try {
  await llm.call({ userContent: message });
} catch (err) {
  if (
    isLLMError(err) &&
    (err.code === 'rate_limit_queue_full' || err.code === 'rate_limit_queue_timeout')
  ) {
    console.warn('rateLimit queue gave up, consider raising the configured limits');
  } else if (isLLMError(err) && err.code === 'provider_rate_limited') {
    console.warn('provider itself rate limited this request');
  }
  throw err;
}

A rate_limit_queue_full or rate_limit_queue_timeout error in your logs is a signal that your configured rateLimit numbers are tighter than your real traffic, worth revisiting rather than something to retry away. A provider_rate_limited error, on the other hand, already went through the normal retry and Retry-After handling before you ever see it; seeing one at all after retries were exhausted means the provider is sustained-throttling you past what backoff alone recovers from.

Watching queue pressure before it becomes failures

onEvent's rate_limited event fires on every attempt that had to wait, before any of those waits turn into an actual rate_limit_queue_full or rate_limit_queue_timeout failure. Wiring it into your existing metrics gives you a leading indicator, so you can raise a limit (or investigate a traffic spike) before callers start seeing errors:

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: { requestsPerMinute: 4_500, tokensPerMinute: 720_000, maxConcurrent: 100 },
  onEvent: (event) => {
    if (event.kind === 'rate_limited') {
      metrics.observe('llm.rate_limit.wait_ms', event.waitedMs, {
        provider: event.provider,
        reason: event.reason, // 'concurrency' | 'rpm' | 'tpm'
      });
    }
  },
});

A reason that's consistently 'tpm' points at raising tokensPerMinute or tightening estimateTokens. A reason that's consistently 'concurrency' points at your own connection pool, not the provider, being the actual bottleneck. See Observing waits for the full event shape.

Sizing tokensPerMinute for a mixed workload

If different call sites through the same instance send very different amounts of text (a short classification prompt versus a long document summarization), the default chars / 4 estimate is usually fine, both directions self-correct against real usage once each call completes. But if one call site dominates your traffic and its prompts have a very different token density than chars / 4 assumes (dense code, non-English text, heavily repeated tokens), a custom estimateTokens pays off:

import { encode } from 'gpt-tokenizer';

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: {
    tokensPerMinute: 720_000,
    estimateTokens: (request) => {
      const promptTokens = request.messages.reduce(
        (sum, m) => sum + encode(typeof m.content === 'string' ? m.content : '').length,
        0,
      );
      return promptTokens + (request.max_tokens ?? 0);
    },
  },
});

See Estimating tokens for exactly how the estimate and the post-call reconciliation interact.