Streaming

VernLLM supports streaming responses through the same call() and cachedCall() you already use. Set stream: true, and everything else, retries, the circuit breaker, schema validation, tool calling, and caching keeps working exactly as it does today, just delivered incrementally instead of as a single resolved value.

Enabling streaming

Pass stream: true to call(). Instead of resolving to T directly, you get back a StreamCallResult<T>, an object with two fields: chunks, an async iterable for live rendering, and finalResult, a promise that resolves to the same value call() would have returned had stream been omitted:

const { chunks, finalResult } = await llm.call({
  userContent: 'Write a short poem about the ocean.',
  jsonMode: false,
  stream: true,
});

for await (const chunk of chunks) {
  if (chunk.type === 'text-delta') {
    process.stdout.write(chunk.delta);
  }
}

const result = await finalResult; // the full, validated response

The chunk shape

Every chunk delivered through chunks is a StreamChunk, a discriminated union over three kinds of event:

type StreamChunk =
  | { type: 'text-delta'; delta: string }
  | {
      type: 'tool_call_delta';
      index: number;
      id?: string;
      name?: string;
      argsDelta?: string;
      complete?: boolean;
    }
  | { type: 'usage'; usage: TokenUsage };

Check chunk.type to branch:

for await (const chunk of chunks) {
  switch (chunk.type) {
    case 'text-delta':
      process.stdout.write(chunk.delta);
      break;
    case 'tool_call_delta':
      // accumulate by chunk.index, see Tool calling while streaming below
      break;
    case 'usage':
      console.log('final token usage', chunk.usage);
      break;
  }
}

finalResult mirrors non-streaming call()

finalResult resolves to exactly what call() would have returned for the same request had stream been omitted: a parsed and validated T by default, or a CallWithToolsResult<T> if tools was also set. JSON parsing, schema validation, and every other post-processing step run against the fully accumulated response, not against individual chunks.

This means you can treat chunks purely as a live rendering concern and still get the exact same typed, validated result your application already relies on:

const { chunks, finalResult } = await llm.call({
  userContent: resumeText,
  schema: CandidateSchema,
  stream: true,
});

for await (const chunk of chunks) {
  if (chunk.type === 'text-delta') renderPartialText(chunk.delta);
}

// result is CandidateSchema's inferred type, validated the same way a
// non-streaming call() would validate it
const result = await finalResult;

chunks is single-use

chunks is an async iterable, not a replayable stream. Iterate it once, from one place. Iterating it more than once, or from two places concurrently, does not replay or fork the sequence, both consumers share the same underlying buffered stream and split the chunks between them.

Stopping iteration early (breaking out of a for await) does not cancel the underlying request either. The stream keeps running in the background regardless, since finalResult still needs to settle. If you only care about the final value and never plan to render live output, it's fine to never touch chunks at all and just await finalResult.

Tool calling while streaming

tools and stream combine freely. Tool call arguments arrive incrementally as tool_call_delta chunks, one or more per tool call, keyed by index so you can tell multiple concurrent tool calls apart:

const { chunks, finalResult } = await llm.call({
  userContent: 'What is the weather in New York?',
  tools: [weatherTool],
  stream: true,
});

for await (const chunk of chunks) {
  if (chunk.type === 'tool_call_delta') {
    console.log(`tool ${chunk.index}: ${chunk.name ?? ''}${chunk.argsDelta ?? ''}`);
  }
}

const result = await finalResult; // CallWithToolsResult<T>, same shape as non-streaming

if (result.type === 'tool_calls') {
  for (const call of result.toolCalls) {
    console.log(call.name, call.arguments); // fully accumulated, parsed arguments
  }
}

argsDelta carries a fragment of the JSON-encoded arguments string as it streams in, not parsed JSON. VernLLM assembles and parses the full arguments for you once the stream completes, so result.toolCalls[i].arguments on finalResult is the same fully parsed object you'd get from a non-streaming tool call. Treat argsDelta as display-only unless you have a specific reason to parse it incrementally yourself.

See Tool Calling for everything else about the tool calling contract, none of which changes when stream is also set.

Usage metering and tracking while streaming

reserveUsage and refundUsage work the same way they do for non-streaming calls, with one difference in timing: reserveUsage runs before the stream opens, but because call() must hand back { chunks, finalResult } before the real outcome of the stream is known, the refund/report step is deferred onto finalResult instead of happening inline. onUsage fires once finalResult resolves successfully; onUsageFailure fires if the stream fails after usage was already reserved.

See Usage Metering and Usage Tracking for the full reference, both apply unchanged.

What gets retried

Streaming keeps the same retry policy as non-streaming call(), applied around opening the stream rather than the whole response:

The circuit breaker, when configured, records a failure for the stream-opening step, matching this same before-first-chunk/after-first-chunk split. Success, correspondingly, is only recorded once the stream fully completes, not when the first chunk arrives, so a connection that opens and then hangs isn't masked as a success. An idle-timeout failure (see below) is the one exception on the failure side: it still counts even though it happens after the first chunk, since a provider that reliably streams one chunk and then hangs is exactly the kind of unhealthy behavior the breaker exists to catch. See Circuit Breaker and Retries for the underlying policy this reuses.

Per-chunk idle timeout

timeoutMs only bounds opening the stream and its first chunk. Every gap after that is bounded separately, by chunkIdleTimeoutMs (default 30000, pass 0 or Infinity to disable). Without it, a connection that opens fine, delivers one chunk, then hangs would never fail on its own.

The clock resets on every chunk, including provider keep-alive pings (see below), so it measures the gap since the most recent chunk, not the stream's total duration. If it elapses, finalResult rejects with LLMError('timeout'), same as the stream-open timeout, and the underlying request is aborted rather than left running in the background.

const llm = new VernLLM({
  client,
  model: 'gpt-4o-mini',
  chunkIdleTimeoutMs: 15_000, // fail if no chunk arrives for 15s
});

Override it per call for routes that need a different value than the instance default, reasoning models with documented long silent gaps, for example:

const { chunks, finalResult } = await llm.call({
  userContent: 'Solve this step by step: ...',
  reasoningEffort: 'high',
  stream: true,
  chunkIdleTimeoutMs: 120_000, // this call only, instance default unaffected
});

Provider keep-alive pings

Some providers send periodic keep-alive signals during long pauses in generation: Anthropic's documented ping events during extended thinking, or an SSE comment line used as a heartbeat. fromAnthropic and fromFetch (for the default SSE framing) both recognize these and reset the idle-timeout clock on them. They carry no content, so nothing is surfaced to chunks or finalResult.

Caching a streaming call

cachedCall() supports stream: true in any combination with tools. The behavior differs depending on whether the call is a cache hit, a cache miss, or a miss that coalesces with an in-flight call for the same key:

const { chunks, finalResult } = await llm.cachedCall({
  cacheKey: `poem:${topic}`,
  ttl: 3600,
  call: {
    userContent: `Write a short poem about ${topic}.`,
    jsonMode: false,
    stream: true,
  },
});

for await (const chunk of chunks) {
  if (chunk.type === 'text-delta') process.stdout.write(chunk.delta);
}

const result = await finalResult;

See Caching for everything else about cachedCall, coalescing, and cache adapters, all of which apply the same way to streaming calls.

A note on the call() overload

Typescript only resolves call() to the streaming overload, returning StreamCallResult<T>, when stream: true is visible as a literal at the call site. A similar caveat applies to tools, see the Tool Calling docs: unlike tools, stream currently has no conditional-value overload of its own, so a conditionally-set stream always falls back to the non-streaming Promise<T> overload, not just when params was pre-widened to CallParams<T>.

If you build params as a plain CallParams<T> and conditionally set stream on it, TypeScript picks the non-streaming Promise<T> overload regardless of what stream turns out to be at runtime. The actual runtime result still follows stream, so this mismatch is silent rather than a compile error:

const params: CallParams<string> = {
  userContent: 'Write a short poem about the ocean.',
  jsonMode: false,
  stream: someCondition,
};

// TypeScript infers Promise<string> here, but the real result is a
// StreamCallResult<string> whenever `stream` evaluates to true. Narrow or
// cast accordingly rather than relying on the static return type.
const result = await llm.call(params);

Provider support

Every adapter implements createStream on top of the same client passed to fromOpenAICompatible, fromAnthropic, fromGemini, or fromBedrock, no separate streaming client is needed:

See each adapter's own page for anything streaming-specific to that provider, and Custom Providers - fromFetch for wiring up streaming through the raw HTTP escape hatch.