Usage Metering

Usage metering lets you control whether a request is allowed to run before it reaches the provider. Use reserveUsage to reserve quota or budget before dispatch, and refundUsage to release that reservation when a call fails after reservation succeeded.

reserveUsage and refundUsage receive { coalesced, signal }.

coalesced is true when the caller is sharing an existing in-flight cached request.
signal is the caller-supplied AbortSignal, when one was provided.
A successful reservation is refunded if the request is aborted before the wrapped operation starts.
If reserveUsage fails, no refund is performed because no reservation was created.

usage-metering-setup.ts

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
});

await llm.call({
  userContent: 'Summarize this document',
  reserveUsage: async ({ signal }) => {
    await billing.reserve({ signal });
  },
  refundUsage: async ({ signal }) => {
    await billing.refund({ signal });
  },
});

How it works

Before the request

reserveUsage runs before any provider request is dispatched. If it fails, the call stops immediately.

After a successful reservation

The LLM request proceeds normally. Retries do not trigger reserveUsage again because it wraps the logical call.

After a failed call

refundUsage runs when a reservation succeeded but the call ultimately fails. Errors from refundUsage are caught and logged, then swallowed without changing the original call error or rejection. Monitor refund failures separately if failed refunds require follow-up.

Reservation failures

If reserveUsage throws, the call fails with LLMError('quota_exceeded') and no request is sent to the provider. The original error is preserved on err.cause. A failed reservation is never refunded because no usage was reserved.

With caching

Usage metering also works with cachedCall. Provide usage hooks at the cache level, not inside the nested call options. CachedCallParams's call field doesn't type-permit reserveUsage/ refundUsage at all, so this is a compile error for a normally-typed caller. A caller that bypasses the type system and sets them inside call anyway hits a runtime LLMError('validation') instead of having them silently stripped: reserving usage twice for the same logical request, or silently skipping reservation, are both worth failing loudly over.

On a cache miss, the usage flow runs around the underlying call. A cache hit skips the LLM call entirely, so no new reservation or usage record is created.

cached-call-metering.ts

const result = await llm.cachedCall({
  cacheKey: `cv:${cvId}`,
  ttl: 3600,
  call: { systemPrompt, userContent },
  reserveUsage: ({ coalesced, signal }) => quota.reserve(userId, { coalesced, signal }),
  refundUsage: ({ coalesced, signal }) => quota.refund(userId, { coalesced, signal }),
});

Concurrent cached calls

When multiple callers request the same cache key at the same time, only the first caller runs the underlying call. Other callers share the same in-flight result.

Each caller still gets its own reserveUsage and refundUsage lifecycle, its own signal, and its own cancellation behavior. The coalesced flag lets your application decide how shared requests should be metered.

Although the underlying call executes only once per in-flight window, reserveUsage and refundUsage are still invoked once per caller. If the call fails, every caller is refunded independently. Aborting one coalesced caller does not cancel the shared in-flight call for other callers.

Relationship with usage tracking

Usage metering controls whether a request can start.

Usage tracking reports provider usage after a provider response, through onUsage on success or onUsageFailure if VernLLM's own post-processing then fails.

See Usage Tracking for token reporting details.

A typical billing flow uses both:

billing-flow.ts

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  onUsage: (usage) => {
    billing.record(usage);
  },
});

await llm.call({
  userContent: 'Write a summary',
  reserveUsage: async ({ signal }) => {
    await billing.reserve({ signal });
  },
  refundUsage: async ({ signal }) => {
    await billing.refund({ signal });
  },
});