Rate Limiting

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: {
    requestsPerMinute: 500,
    tokensPerMinute: 200_000,
    maxConcurrent: 20,
  },
});

rateLimit queues calls locally to stay under configured limits, instead of dispatching every call and letting the provider reject the ones that go over. This is proactive: the existing Retry-After handling in Retries is reactive, it only recovers after a self-inflicted 429 has already cost a round trip. rateLimit avoids tripping the limit in the first place.

Every bucket is independent and optional. Omit rateLimit entirely, or omit any individual field within it, and that dimension is unlimited, exactly matching behavior before this option existed.

The three buckets

A call only proceeds once every configured bucket has room. If any one of them is short, the call queues until all three clear.

Queueing

Calls that can't get capacity immediately queue in strict FIFO order. A large call is never starved by a stream of smaller ones queued behind it, VernLLM never reorders the queue to let a smaller request skip ahead.

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: {
    requestsPerMinute: 60,
    maxQueueMs: 10_000, // default 30000, pass 0 to wait indefinitely
    maxQueueSize: 50, // default 0, unbounded
  },
});

A call that exceeds maxQueueMs, or arrives when the queue is already at maxQueueSize, fails with LLMError('rate_limited') carrying code: 'rate_limit_queue_timeout' or code: 'rate_limit_queue_full' respectively:

import { isLLMError } from 'vern-llm';

try {
  await llm.call({ userContent: '...' });
} catch (err) {
  if (
    isLLMError(err) &&
    (err.code === 'rate_limit_queue_timeout' || err.code === 'rate_limit_queue_full')
  ) {
    // Never reached the provider. Retrying immediately won't help,
    // the wait already happened. See "Interaction with retries" below.
  }
}

An aborted signal on a queued call removes it from the queue immediately and rejects with LLMError('aborted'), freeing that queue slot for the next waiter. See Cancellation & Timeouts.

Estimating tokens

tokensPerMinute needs a token count before the request is sent, when the real count isn't known yet. The default estimate is a chars / 4 heuristic over every message's content, plus the requested maxTokens:

Math.ceil(messagesChars / 4) + (request.max_tokens ?? 0);

This is intentionally rough. Once the call completes, the estimate is reconciled against the provider's real reported usage, so a systematically over- or under-estimating heuristic self-corrects over time rather than compounding.

Provide estimateTokens to use a real tokenizer, or any other heuristic your provider or model mix calls for:

import { encode } from 'gpt-tokenizer';

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: {
    tokensPerMinute: 200_000,
    estimateTokens: (request) => {
      const promptTokens = request.messages.reduce(
        (sum, m) => sum + encode(typeof m.content === 'string' ? m.content : '').length,
        0,
      );
      return promptTokens + (request.max_tokens ?? 0);
    },
  },
});

Per-attempt, not per-call

Capacity is acquired for each attempt, including retries, not once for the whole logical call. Every retry is a real request against the same provider limits, so it needs to clear the same buckets again:

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  maxRetries: 3,
  rateLimit: { requestsPerMinute: 500 },
});

If the first attempt fails and a retry is due, that retry queues for capacity exactly like the original attempt did.

stream: true

For a streaming call, capacity is held for the connection's entire lifetime and released only once the stream completes, either by finishing normally or by a mid-stream failure, not the moment it merely opens. A stream holds a real connection to the provider the whole time it's open, so it continues to count against maxConcurrent for as long as that connection is live.

See Streaming for the rest of the stream: true contract.

Interaction with retries

The three local rate-limit codes (rate_limit_queue_full, rate_limit_queue_timeout, rate_limit_capacity_exceeded) are never retried. For the two queue codes, the wait already happened while the call was queued, so retrying immediately would only requeue it behind the same limit with nothing changed; for rate_limit_capacity_exceeded, the call could never fit the configured capacity regardless of how long it waits:

if (
  error instanceof LLMError &&
  (error.code === 'rate_limit_queue_full' ||
    error.code === 'rate_limit_queue_timeout' ||
    error.code === 'rate_limit_capacity_exceeded')
) {
  return false;
}

See What gets retried for the full list of non-retried error conditions.

Interaction with the circuit breaker

A local rate-limit error never reached the provider, so it says nothing about the provider's health. It does not count toward the circuit breaker's failure threshold, the same treatment tool contract errors already get. See What counts as a failure.

Provider 429s

rateLimit is a local, proactive control. It does not change how an actual provider 429 is handled, that still flows through the existing retry and Retry-After machinery. A provider 429 does now also carry code: 'provider_rate_limited', so you can tell a rate limit the provider itself imposed apart from one VernLLM enforced locally, without inspecting status directly:

if (isLLMError(err) && err.code === 'provider_rate_limited') {
  // A real provider 429. Already retried per the normal retry/Retry-After flow.
}

if (
  isLLMError(err) &&
  (err.code === 'rate_limit_queue_full' || err.code === 'rate_limit_queue_timeout')
) {
  // Never left this process. VernLLM's own queue gave up.
}

Observing waits

onEvent reports a rate_limited event whenever an attempt actually had to wait for capacity:

{
  kind: 'rate_limited',
  requestId: string,
  provider: string,
  model: string,
  waitedMs: number,
  reason: 'concurrency' | 'rpm' | 'tpm',
}

reason identifies which bucket was blocking the call just before it cleared. An attempt that never had to wait (capacity was immediately available) does not emit this event.

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  rateLimit: { requestsPerMinute: 500, maxConcurrent: 20 },
  onEvent: (event) => {
    if (event.kind === 'rate_limited') {
      metrics.observe('llm.rate_limit.wait_ms', event.waitedMs, { reason: event.reason });
    }
  },
});

See Event kinds for the rest of the onEvent union.

Options reference

See Configuration for how rateLimit sits alongside every other constructor option.