# llm-sdk

llm-sdk (npm: llm-sdk-js) is an open-source TypeScript router for LLM calls. It runs in your own process — your API keys, direct HTTP calls to providers, zero runtime dependencies, no proxy service in the middle. Given a primary model and a list of fallbacks, it retries and fails over automatically and returns which provider answered, what the call cost, and every attempt on the way. It supports OpenAI, Anthropic, Groq, and local/self-hosted OpenAI-compatible endpoints, plus named routes, response caching, cost tracking, structured output extraction, and tool calling.

The complete documentation, concatenated for language models. See https://www.llm-sdk.dev/llms.txt for a shorter, curated index.

<!-- source: https://www.llm-sdk.dev/guide/getting-started -->

# Getting started

`llm-sdk` runs inside your process using your API keys over `fetch`. There is no proxy to deploy, no
account to create, and no service in the middle that sees your prompts.

## Install

```bash [npm]
npm install llm-sdk-js
```

```bash [pnpm]
pnpm add llm-sdk-js
```

```bash [yarn]
yarn add llm-sdk-js
```

```bash [bun]
bun add llm-sdk-js
```

Node 18 or newer, since the adapters use the built-in `fetch`. The package ships ESM and CJS builds
with TypeScript types, and has no runtime dependencies. `zod` is an optional peer dependency, needed
only for [`extract()`](/guide/extract).

## Set a key

Keys are read from the environment, so nothing sensitive goes in your config:

| Provider              | Variable            | Notes                                      |
| --------------------- | ------------------- | ------------------------------------------ |
| `openai/…`            | `OPENAI_API_KEY`    |                                            |
| `anthropic/…`         | `ANTHROPIC_API_KEY` |                                            |
| `groq/…`              | `GROQ_API_KEY`      |                                            |
| `ollama/…`, `local/…` | —                   | Talks to `http://127.0.0.1:11434/v1`       |

You only need keys for the providers you actually reference. To pass a key explicitly instead — or
to point a provider at a compatible endpoint — see [Configuration](/guide/configuration).

## Your first call

```ts [example.ts]
import { createRouter } from "llm-sdk-js";

const llm = createRouter({
  primary: "anthropic/claude-sonnet-4-5",
  fallbacks: ["openai/gpt-4o", "groq/llama-3.3-70b"],
});

const res = await llm.complete("Summarise this changelog in one line: ...");

console.log(res.text);
console.log(res.provider); // "anthropic" — unless it was down
```

Models are plain `provider/model` strings, so switching a model is a text edit rather than a new
client class. `complete()` takes either a string prompt or an object when you need more control:

```ts
const res = await llm.complete({
  system: "You are a terse release-notes editor.",
  prompt: "Summarise this changelog in one line: ...",
  maxTokens: 120,
  temperature: 0,
});
```

Or pass `messages` directly for multi-turn conversations:

```ts
const res = await llm.complete({
  messages: [
    { role: "user", content: "What broke in v2?" },
    { role: "assistant", content: "The auth middleware changed shape." },
    { role: "user", content: "Show me the migration." },
  ],
});
```

## What comes back

Every call resolves to the same shape, whether it was served by the primary, a fallback, or the
cache:

```ts
res.text; // "Renamed AuthService to IdentityGateway."
res.provider; // "openai"
res.model; // "gpt-4o"
res.usage; // { input: 412, output: 38 }
res.cost; // 0.0014 — USD, estimated from usage
res.unknownModel; // false — true (and cost 0) if the model has no price table entry
res.cached; // false
res.latencyMs; // 184
res.toolCalls; // [] unless the model called a tool
res.attempts; // one record per try, in order
```

`attempts` is the part worth logging. It is the whole story of the call, including the failures:

```ts
[
  { provider: "anthropic", model: "claude-sonnet-4-5", error: "rate_limit", ms: 210 },
  { provider: "openai", model: "gpt-4o", ms: 184 },
];
```

A quiet model switch is the kind of thing that gets a library a bad reputation, so it is never
hidden — `res.provider` and `res.attempts` always tell you what really happened.

## When everything fails

If the chain is exhausted, the call throws `AllProvidersFailed` with the same attempt records
attached:

```ts
import { AllProvidersFailed, BadRequest } from "llm-sdk-js";

try {
  await llm.complete(prompt);
} catch (err) {
  if (err instanceof AllProvidersFailed) {
    console.error("no provider answered", err.attempts);
  } else if (err instanceof BadRequest) {
    // Your mistake, not the provider's: bad model ref, empty prompt, unknown route.
    // Never retried, never failed over.
    throw err;
  }
}
```

Those are the only two error types you catch. Details in [Errors](/guide/errors).

## Run it without a network

There's no shipped mock provider — implement the `Adapter` interface with a small deterministic
double and register it under any provider name via `adapters`, which makes the fallback path
something you can actually test:

```ts [router.test.ts]
import { createRouter, ProviderError, type Adapter } from "llm-sdk-js";

class FakeAdapter implements Adapter {
  readonly name = "fake";
  private calls = 0;

  async complete() {
    if (this.calls++ === 0) {
      throw new ProviderError("rate limited", { kind: "rate_limit", provider: this.name });
    }
    return { text: "served by the fallback", usage: { input: 0, output: 0 }, toolCalls: [] };
  }

  async *stream() {
    yield { type: "done" as const, usage: { input: 0, output: 0 }, toolCalls: [] };
  }
}

const llm = createRouter({
  primary: "fake/fail",
  fallbacks: ["fake/ok"],
  adapters: { fake: new FakeAdapter() },
});

const res = await llm.complete("hi");

res.text; // "served by the fallback"
res.attempts.length; // 2
```

The router doesn't distinguish a real provider from your `FakeAdapter` — it only sees the
`Adapter` interface, so `ProviderError`'s `kind` drives the exact same retry/fallback logic
either way. No key, no network, no clock skew. See [Testing](/guide/testing).

## Name your tradeoffs

Once you have more than one kind of call, put the choices in one place and refer to them by name
instead of repeating model strings:

```ts
const llm = createRouter({
  routes: {
    fast: { primary: "groq/llama-3.3-70b" },
    smart: {
      primary: "anthropic/claude-sonnet-4-5",
      fallbacks: ["openai/gpt-4o"],
    },
    cheap: {
      primary: "openai/gpt-4o-mini",
      cache: { ttl: "24h" },
    },
  },
  default: "smart",
});

await llm.complete(prompt); // uses "smart"
await llm.route("fast").complete(ticket); // uses "fast"
```

Route names are typed from the config, so `llm.route("smrt")` fails at compile time rather than at
3am. Options merge in one direction — call beats route beats global — which is covered in
[Named routes](/guide/routes).

## Streaming

```ts
const stream = llm.stream(prompt);

for await (const chunk of stream) {
  if (!chunk.done) process.stdout.write(chunk.text);
}

const final = await stream.result(); // usage, cost, provider, attempts
```

Streaming is real SSE from the provider, not a fake replay. The router buffers the first ~40
characters before emitting anything so it can still fail over cleanly if that provider dies
early. Once text has reached you, a failure ends the stream — there is no silent hop mid-sentence.
Live streams do not return tool calls; use `complete()` for tools. Details in
[Fallback & retry](/guide/fallback).

## Next

- [Configuration](/guide/configuration) — every option, and where to set it
- [Named routes](/guide/routes) — one config for `fast` / `smart` / `cheap`
- [Fallback & retry](/guide/fallback) — which errors fail over, and how backoff works
- [Caching & cost](/guide/caching-cost) — deterministic keys, TTLs, and pricing
- [Structured output](/guide/extract) — `extract()` with a Zod schema
- [Tools](/guide/tools) — declare tools and complete the round trip
- [Errors](/guide/errors) — what to catch
- [Testing](/guide/testing) — fake adapters, no network

---

<!-- source: https://www.llm-sdk.dev/guide/caching-cost -->

# Caching & cost

Two separate features that show up on every `CompleteResult`: `cached` and `cost`.

## Caching

Caching is **off** unless you set it. There is no surprise memoization.

```ts
const llm = createRouter({
  primary: "openai/gpt-4o-mini",
  cache: { ttl: "1h" },
});
```

### TTL

`ttl` accepts:

| Form | Example |
| ---- | ------- |
| Duration string | `"30m"`, `"1h"`, `"24h"`, `"7d"`, `"500ms"` |
| Milliseconds number | `3_600_000` |

Invalid strings throw at write time.

### When a call is eligible

| Condition | Cached? |
| --------- | ------- |
| No `cache` / `cache: false` | No |
| `temperature` omitted or `0` | Yes (if cache configured) |
| `temperature > 0` | **No**, unless `includeNonDeterministic: true` |
| `stream()` | Same rules — shares the cache with `complete()` |

```ts
cache: { ttl: "1h", includeNonDeterministic: true }
```

### Cache key

The key is a SHA-256 hex digest of a normalized (object keys sorted, so `{role,content}` and
`{content,role}` hash the same) JSON payload:

```ts
sha256(JSON.stringify(normalize({
  models: modelChain, // primary + fallbacks actually configured for this call
  messages,
  temperature: temperature ?? null,
  maxTokens: maxTokens ?? null,
  tools: tools ?? null, // schemas normalized (Zod → JSON Schema)
  raw: raw ?? null,
})))
```

Hashing keeps the key a fixed 64 hex characters no matter how large the prompt is — the `Map`
doesn't store the full prompt twice (once as key, once in the cached value).

Same prompt with a different fallback list is a different key. Same prompt via `complete()` and
`stream()` share a key. Different `tools` or `raw` params also get different keys — Zod tool
schemas are converted before hashing so equivalent shapes collide correctly.

### Hits

```ts
const res = await llm.complete(prompt);
res.cached; // true
res.latencyMs; // 0
res.attempts; // [] — no provider was called
```

`stream()` on a hit emits one immediate chunk (nothing left to generate incrementally).
`stream.result().cached` is still `true`.

### Storage

The built-in store is an in-process `MemoryCache`:

- Per router instance (create the router **once** at startup — see `examples/server.ts`)
- Cap **1000** entries; least-recently-used eviction when full
- TTL expiry checked on read
- Not shared across processes or servers

There is a `Cache` interface in the codebase for a future backend; it is not pluggable from
public config yet.

## Cost

Every successful live call sets `res.cost` in **estimated USD**:

```ts
cost = (usage.input / 1e6) * price.input + (usage.output / 1e6) * price.output
```

Prices live in `src/pricing/prices.json` (also exported as `PRICES` / `PRICING`):

| Model id | Input $/MTok | Output $/MTok |
| -------- | ------------ | ------------- |
| `gpt-4o` | 2.5 | 10 |
| `gpt-4o-mini` | 0.15 | 0.6 |
| `claude-sonnet-4-5` | 3 | 15 |
| `claude-haiku-4-5` | 1 | 5 |
| `llama-3.3-70b` | 0.59 | 0.79 |
| `llama-3.3-70b-versatile` | 0.59 | 0.79 |

Lookup is by the **model id only** (the part after `provider/`), not the full ref.

```ts
import { cost, PRICES } from "llm-sdk-js";

cost({ input: 1000, output: 200 }, "gpt-4o-mini");
```

### Unknown models → `$0`, but not silently

If the model id is missing from the table, `cost` is **`0`** — but `res.unknownModel` is `true`,
so a typo'd or newly-released model reports as explicitly unpriced rather than looking free.
`cost()` also logs a one-time `console.warn` the first time it sees an unrecognized model id
(not repeated on every call for that model). Check `unknownModel` before trusting `cost` in
anything that sums spend, or extend the price table in a fork / local patch until the package
grows coverage.

```ts
import { isKnownModel } from "llm-sdk-js";

isKnownModel("gpt-4o-mini"); // true
isKnownModel("some-new-model"); // false — cost() would return 0 for it
```

Cached hits still return the **original** stored `cost` from when the entry was written (they
do not re-price).

## Putting it together

```ts
const llm = createRouter({
  primary: "openai/gpt-4o-mini",
  cache: { ttl: "5m" },
  temperature: 0,
});

for (const question of faqTraffic) {
  const res = await llm.complete(question);
  total += res.cost;
  if (res.cached) hits++;
}
```

See `examples/cache.ts` for a support-FAQ scenario that prints API calls vs cache hits.

## Next

- [Configuration](/guide/configuration) — `cache` on global / route / call
- [Named routes](/guide/routes) — a `cheap` route with a long TTL
- [Tools](/guide/tools) — why tool calls and cache collide

---

<!-- source: https://www.llm-sdk.dev/guide/configuration -->

# Configuration

Options use the **same keys** at three layers. Later layers win:

1. **Global** — `createRouter({ … })`
2. **Route** — `routes.fast` / `routes.smart` / …
3. **Call** — the second argument to `complete()` / `stream()` / `extract()`

```ts
const llm = createRouter({
  primary: "anthropic/claude-sonnet-4-5",
  fallbacks: ["openai/gpt-4o", "groq/llama-3.3-70b"],

  retry: { attempts: 3, baseDelay: 500, maxDelay: 10_000 },
  timeout: 60_000, // total budget across all attempts (default if omitted)
  cache: { ttl: "1h" }, // omit to disable caching entirely
  system: "Be concise.",
  onFallback: (from, to, err) => console.warn("fallback", { from, to, err }),
});

await llm.complete(prompt, {
  model: "anthropic/claude-haiku-4-5", // overrides primary for this call; fallbacks still apply
  timeout: 5_000,
  cache: false,
});
```

You almost never need every field. Defaults are enough for a first integration; dig in when
latency, cost, or failure modes matter.

## Merge rules

| Field | How it merges |
| ----- | ------------- |
| Most scalars (`timeout`, `temperature`, `maxTokens`, `system`, `tools`, …) | Last layer wins |
| `retry` | Shallow-merged (`{ ...global.retry, ...route.retry, ...call.retry }`) |
| `cache` | Last layer wins; `cache: false` turns caching off for that call |
| `fallbacks` | Last layer wins (the whole list is replaced, not concatenated) |
| `raw` | Shallow-merged per provider (`global.raw.anthropic` + `call.raw.anthropic`) |

`model` and `primary` cannot both appear **in the same object** — `createRouter({…})`, one route,
or one call. `model` always wins inside a layer, so a sibling `primary` would be dead code; that
throws `BadRequest` immediately. A per-call `model` overriding a global or route `primary` is
normal and supported.

```ts
// OK — call overrides global primary
await llm.complete(prompt, { model: "openai/gpt-4o-mini" });

// BadRequest — both in the same object
createRouter({ model: "openai/gpt-4o-mini", primary: "openai/gpt-4o" });
```

## Option reference

Same keys on `RouterConfig`, each route, and each call (except router-only fields marked ★).

| Key | Default | Notes |
| --- | ------- | ----- |
| `primary` | — | First model in the chain (`provider/model`) |
| `model` | — | Alias that replaces `primary` for that layer; fallbacks still apply |
| `fallbacks` | `[]` | Tried in order after the primary fails (see [Fallback & retry](/guide/fallback)) |
| `retry` | `{ attempts: 1 }` | Same-provider retries before failing over. `baseDelay` / `maxDelay` default to `500` / `10_000` ms |
| `timeout` | `60_000` | **Total** budget across every attempt and backoff wait, not per request |
| `cache` | off | `{ ttl }` or `false`. See [Caching & cost](/guide/caching-cost) |
| `temperature` | provider default | Values `> 0` skip the cache unless `includeNonDeterministic: true` |
| `maxTokens` | provider default | Anthropic requires a value; the adapter sends `1024` if you omit it |
| `system` | — | Injected as a system message when the input is a string or `{ prompt }` |
| `messages` | — | Prefer passing these on the call input, not as a sticky global |
| `tools` | — | Tool definitions for this call (see [Tools](/guide/tools)) |
| `raw` | — | Provider-specific body fields, keyed by provider name |
| `allowContentFilterFailover` | `false` | Opt in to fail over on content filter / refusal |
| `routes` / `default` ★ | — | Named tradeoffs — see [Named routes](/guide/routes) |
| `onFallback` ★ | — | Fires on every hand-off to the next model |
| `providers` ★ | — | Per-provider `apiKey` / `baseUrl` overrides |
| `adapters` ★ | — | Bring-your-own `Adapter` for a provider name |

★ Router-only — not valid on a per-call options object.

## Models and keys

Models are plain strings: `"openai/gpt-4o"`, `"anthropic/claude-sonnet-4-5"`. The part before the
slash picks the adapter; the rest is the model id sent upstream.

| Provider | Env var | Default base URL |
| -------- | ------- | ---------------- |
| `anthropic` | `ANTHROPIC_API_KEY` | `https://api.anthropic.com` |
| `openai` | `OPENAI_API_KEY` | `https://api.openai.com/v1` |
| `groq` | `GROQ_API_KEY` | `https://api.groq.com/openai/v1` |
| `ollama` / `local` | `OLLAMA_API_KEY` / `LOCAL_API_KEY` (optional) | `http://127.0.0.1:11434/v1` |

Override connection details without writing an adapter:

```ts
createRouter({
  primary: "ollama/llama3",
  providers: {
    ollama: { baseUrl: "http://gpu-box:11434/v1", apiKey: "ollama" },
  },
});
```

Env vars are the default. An explicit `providers.*.apiKey` wins when you need a key that isn't in
the process environment (tests, multi-tenant hosts, short-lived secrets).

## Custom adapters

Anything not in the table — a gateway, a second OpenAI-compatible host under a new name, or a
deterministic test double — registers the same way:

```ts
import { createRouter, type Adapter } from "llm-sdk-js";

const llm = createRouter({
  primary: "acme/fast-model",
  adapters: {
    acme: myAcmeAdapter, // implements Adapter
  },
});
```

That name bypasses the built-in factory entirely. See [Testing](/guide/testing) for a full
`ProviderError` example.

## Escape hatch: `raw`

When a provider accepts a parameter this library doesn't model, pass it under that provider's
key. It is merged into the request body for that provider only:

```ts
await llm.complete(prompt, {
  raw: { anthropic: { top_k: 40 } },
});
```

## What this page deliberately skips

Retry kinds, content-filter policy, and the attempt trail live in
[Fallback & retry](/guide/fallback). Cache keys, TTL strings, and cost estimates live in
[Caching & cost](/guide/caching-cost). Typed `route("fast")` lives in
[Named routes](/guide/routes).

---

<!-- source: https://www.llm-sdk.dev/guide/errors -->

# Errors

App code only needs to branch on two classes. A third export exists for people **writing**
adapters, not for catching `complete()`.

```ts
import { AllProvidersFailed, BadRequest } from "llm-sdk-js";

try {
  await llm.complete(prompt);
} catch (err) {
  if (err instanceof BadRequest) {
    // Fix the call site — never retried, never failed over
  } else if (err instanceof AllProvidersFailed) {
    // Every useful attempt is in err.attempts
  } else {
    throw err; // unexpected
  }
}
```

## `BadRequest`

**Meaning:** the request is invalid on your side, or `extract()` could not produce valid data
after schema retries.

**Typical causes:**

| Cause | Example |
| ----- | ------- |
| Missing config | `createRouter({})` without `primary` or `routes` |
| Bad model ref | `"gpt-4o"` (no `provider/`) |
| Empty input | `complete()` with no prompt/messages |
| Unknown route | `route("nope")` |
| Conflicting fields | `model` and `primary` in the same object |
| Bad extract schema | no `.parse` / `.safeParse` |
| Extract exhausted | JSON/schema still invalid after 2 tries |

**Behavior:** throw immediately. No retry. No failover. May wrap a cause via `{ cause }`.

## `AllProvidersFailed`

**Meaning:** the model chain finished without a successful answer.

**Fields:**

```ts
err.attempts; // AttemptRecord[] — every try in order
err.message; // default "All providers failed", or content-filter specific text
```

```ts
interface AttemptRecord {
  provider: string;
  model: string;
  error?: string; // ErrorKind or message label; omitted on success
  ms: number;
}
```

**Includes:**

- Exhausted fallbacks after rate limits / timeouts / auth hops
- Default content-filter stop (often a **single** attempt with `error: "content_filter"`)
- Budget exhaustion mid-chain

**Does not mean** "HTTP 500 from one host" by itself — that is usually a failover-able
`ProviderError` that never leaves the router as that class.

## `ProviderError` (for adapters)

```ts
import { ProviderError, type ErrorKind } from "llm-sdk-js";

throw new ProviderError("rate limited", {
  kind: "rate_limit",
  provider: "acme",
  retryable: true, // optional; defaults from kind
  status: 429,
  retryAfterMs: 1_500,
});
```

| Field | Role |
| ----- | ---- |
| `kind` | Drives retry / failover policy ([Fallback & retry](/guide/fallback)) |
| `provider` | Label in logs / attempts |
| `retryable` | Default true for rate_limit, timeout, overloaded, server_error, network |
| `status` | Optional HTTP status |
| `retryAfterMs` | Optional backoff hint |

`complete()` / `stream()` / `extract()` are not documented to reject with `ProviderError` for
normal provider failures — they classify and either continue the chain or throw one of the two
public errors. Mid-stream failures after the buffer has flushed may still surface a raw error
from the iterator; prefer draining via patterns that treat stream abort as failure, and see
[Fallback & retry](/guide/fallback) for streaming limits.

## Mapping HTTP → kind

Built-in adapters use shared classification roughly as:

| Status | Kind |
| ------ | ---- |
| 429 | `rate_limit` |
| 408, 504 | `timeout` |
| 401, 403 | `auth` |
| 529 | `overloaded` |
| 400, 404, 422 | `bad_request` |
| other 5xx | `server_error` |
| network / abort | `network` / `timeout` |

## HTTP APIs

When wrapping the router in a server (`examples/server.ts`):

| SDK error | Sensible HTTP status |
| --------- | -------------------- |
| `BadRequest` | `400` |
| `AllProvidersFailed` | `503` |

## Next

- [Fallback & retry](/guide/fallback) — policy table by kind
- [Testing](/guide/testing) — throw `ProviderError` from a fake adapter
- [Errors API](/api/errors) — field-level reference

---

<!-- source: https://www.llm-sdk.dev/guide/extract -->

# Structured output

`extract()` asks the model for JSON, parses it, and validates it against a schema. The validated
value is typed when the schema is Zod (v3 or v4) or any [Standard Schema](https://standardschema.dev)
library (Valibot, ArkType, …) — with **zero** runtime dependency on those libraries.

```ts
import { z } from "zod";
import { createRouter } from "llm-sdk-js";

const llm = createRouter({
  primary: "openai/gpt-4o-mini",
  temperature: 0,
});

const { data, text, provider, attempts } = await llm.extract({
  prompt: "Extract the invoice details from: …",
  system: "You extract structured data for accounting.",
  schema: z.object({
    total: z.number(),
    dueDate: z.string(),
    lineItems: z.array(
      z.object({
        label: z.string(),
        amount: z.number(),
      }),
    ),
  }),
});

data.total; // number
data.lineItems[0].label; // string
```

`zod` is an **optional peer dependency**. Install it only if you use `extract()` (or Zod tool
schemas).

## What gets sent to the model

The router appends an instruction to your prompt:

1. If `schemaDescription` is set, that string is used.
2. Else, for Zod object schemas, field names and coarse types are read from `.shape`
   (e.g. `{ total: number, dueDate: string }`).
3. Else: `"Respond with JSON only that matches the requested schema. No markdown."`

Markdown fences around the JSON are stripped before `JSON.parse`.

## Schema retries

Validation can fail even when the HTTP call succeeded (invalid JSON, wrong shape, Zod error).
`extract()` will call `complete()` up to **twice**:

1. First attempt with the base prompt + schema instruction.
2. Second attempt feeding `Previous JSON was invalid: … Fix it.` back to the model.

If both fail, it throws `BadRequest` with the last parse/validation error. That uses the
`BadRequest` class (caller-facing "this didn't produce valid data") rather than
`AllProvidersFailed` — even though the underlying model calls may have succeeded.

Only the **last** successful `complete()`'s `usage` / `cost` / `attempts` are returned on
success. Failed parse attempts' provider trails are not merged into the result today.

## Temperature

`extract()` forces `temperature: 0` unless you pass an explicit `temperature` in the call
options. Structured extraction wants determinism.

## Options

`extract(input, callOptions?)` accepts the same second-argument options as `complete()` —
`model`, `fallbacks`, `cache`, `timeout`, `retry`, and so on. Caching works: identical extract
prompts with `temperature: 0` and a configured `cache` can hit.

## Non-Zod schemas

Anything with `.parse()` or `.safeParse()` works at runtime. Type inference and auto
`schemaDescription` need Zod/Standard Schema shape. For a hand-rolled validator, pin both:

```ts
const { data } = await llm.extract<typeof myValidator, { total: number; dueDate: string }>({
  prompt: "Extract the invoice",
  schema: myValidator,
  schemaDescription: "{ total: number, dueDate: string }",
});
```

## What `extract()` is not

- It is not provider-native structured output / JSON mode APIs (those can still be passed via
  `raw` if you want them in addition).
- It does not stream.
- It does not run tools; use `complete()` for tool loops.

## Next

- [Tools](/guide/tools) — when the model should call functions instead of returning JSON once
- [Configuration](/guide/configuration) — caching and model overrides on extract calls
- [Errors](/guide/errors) — `BadRequest` from failed extraction

---

<!-- source: https://www.llm-sdk.dev/guide/fallback -->

# Fallback & retry

A router earns its keep when the primary is rate-limited, timed out, or simply down. This page is
the full policy: what retries, what fails over, what stops the chain, and what you see afterward.

## The model chain

Every call walks a list:

```text
[primary or model, ...fallbacks]
```

Example:

```ts
createRouter({
  primary: "anthropic/claude-sonnet-4-5",
  fallbacks: ["openai/gpt-4o", "groq/llama-3.3-70b"],
  retry: { attempts: 3, baseDelay: 500, maxDelay: 10_000 },
  timeout: 60_000,
  onFallback: (from, to, err) => console.warn("fallback", { from, to, err }),
});
```

For each model in the chain the router may retry the **same** provider a few times (backoff),
then move to the **next** model. When the list is exhausted, it throws `AllProvidersFailed`.

## Error kinds

Adapters map HTTP and network failures into a `ProviderError` with a `kind`. The kind decides
the next step:

| kind | Same-provider retry? | Fail over to next model? | Notes |
| ---- | -------------------- | ------------------------ | ----- |
| `rate_limit` | Yes (honors `Retry-After`) | Yes, after retries | |
| `timeout` | Yes | Yes, after retries | Includes aborted budget |
| `server_error` | Yes | Yes, after retries | 5xx except 529 |
| `network` | Yes | Yes, after retries | DNS, connection reset, … |
| `overloaded` | **No** — skip immediately | Yes | HTTP 529; no backoff wait |
| `auth` | **No** — skip immediately | Yes | Bad/missing key on *this* provider does not imply the next is broken |
| `unknown` | **No** — skip immediately | Yes | Unclassified provider failure |
| `content_filter` | **No** | **Only if** `allowContentFilterFailover: true` | Default: stop the chain |
| `bad_request` | Never | Never | Becomes `BadRequest` — your bug |

`BadRequest` (empty prompt, bad model ref, unknown route, `model`+`primary` in one object) is
never retried and never failed over. It throws before or instead of walking the chain.

## Defaults that surprise people

| Option | Default | Meaning |
| ------ | ------- | ------- |
| `retry.attempts` | **`1`** | **No** same-provider retry unless you raise it — only failover |
| `retry.baseDelay` | `500` | ms, used in exponential backoff |
| `retry.maxDelay` | `10_000` | cap for backoff and `Retry-After` |
| `timeout` | `60_000` | **Total** budget for the whole chain, including sleeps |

```ts
// Production-shaped retry: three tries on the primary before failing over
retry: { attempts: 3, baseDelay: 500, maxDelay: 10_000 }
```

## Backoff

Between same-provider retries:

1. If the response included `Retry-After`, use that delay (seconds or HTTP-date), capped by
   `maxDelay`.
2. Otherwise full jitter: `random(0 … min(maxDelay, baseDelay * 2^attemptIndex))`.
3. The wait is also capped by the **remaining** timeout budget. If the budget is gone, the
   attempt is recorded as `timeout` and the chain moves on or fails.

`overloaded`, `auth`, `unknown`, and opted-in `content_filter` **skip** this wait and jump to
the next model (firing `onFallback`).

## Content filtering

OpenAI may return `finish_reason: "content_filter"`. Anthropic may return `stop_reason: "refusal"`.
Both become `kind: "content_filter"`.

**Default:** the router throws `AllProvidersFailed` with message
`"Content was filtered by the provider"` and does **not** try the next model. Silently shopping
for a looser provider is a bad default.

```ts
await llm.complete(prompt, { allowContentFilterFailover: true });
```

With the flag, content filter behaves like other fail-over-immediately kinds.

## `onFallback`

Fires whenever the router hands off to the next model — including after `auth` and `overloaded`.
Arguments: `(from, to, err)` where `from` / `to` are model refs (`"anthropic/…"`) and `err` is
the failure that caused the hop.

Use it for metrics and logs. Do not assume it means the call ultimately failed — the next
provider may still succeed.

## The attempt trail

Every finished call (success or `AllProvidersFailed`) carries `attempts`:

```ts
[
  { provider: "anthropic", model: "claude-sonnet-4-5", error: "rate_limit", ms: 210 },
  { provider: "openai", model: "gpt-4o", ms: 1840 },
]
```

- Successful attempts omit `error`.
- `res.provider` / `res.model` are **who served the answer**, not who was tried first.
- Log `attempts` in production. When output looks wrong, that array is the postmortem.

## Streaming vs `complete()`

| | `complete()` | `stream()` |
| - | ------------ | ---------- |
| Same-provider retries | Yes (`retry.attempts`) | **No** — one try per model |
| Fail over | Full chain | Only while the first ~40 characters are still buffered |
| After text has been emitted | — | Failure ends the stream; no silent failover |

See the streaming section in the README / Getting started for the buffer rationale.

## Total timeout budget

`timeout` is not "per HTTP request." A `TimeoutBudget` starts when the call begins. Every
attempt's `AbortSignal`, every backoff sleep, and every failover share that deadline. When it
hits zero mid-chain, remaining models may be skipped with `error: "timeout"` records.

## Catching the outcome

```ts
import { AllProvidersFailed, BadRequest } from "llm-sdk-js";

try {
  const res = await llm.complete(prompt);
  console.log(res.provider, res.attempts);
} catch (err) {
  if (err instanceof BadRequest) {
    // Fix the call site
  } else if (err instanceof AllProvidersFailed) {
    console.error(err.attempts); // every try, including content_filter stops
  }
}
```

You do **not** catch `ProviderError` from `complete()` in normal app code — it is classified and
either folded into the trail or rethrown as one of the two public errors. (`ProviderError` is
exported for **authoring** custom adapters; see [Testing](/guide/testing).)

## Next

- [Errors](/guide/errors) — the two classes in more detail
- [Configuration](/guide/configuration) — where to set `retry` / `timeout` / `fallbacks`
- [Testing](/guide/testing) — drive every branch with a fake adapter

---

<!-- source: https://www.llm-sdk.dev/guide/routes -->

# Named routes

Different parts of an app need different tradeoffs. Triage wants cheap and fast. A customer-facing
reply wants quality. A nightly digest wants long cache TTLs. Scatter those choices as string
literals and every call site becomes a policy decision. Put them in one place and call them by
name.

```ts
import { createRouter } from "llm-sdk-js";

const llm = createRouter({
  routes: {
    fast: {
      primary: "groq/llama-3.3-70b",
      fallbacks: ["openai/gpt-4o-mini"],
      timeout: 8_000,
    },
    smart: {
      primary: "anthropic/claude-sonnet-4-5",
      fallbacks: ["openai/gpt-4o"],
      retry: { attempts: 3 },
    },
    cheap: {
      primary: "openai/gpt-4o-mini",
      cache: { ttl: "24h" },
      temperature: 0,
    },
  },
  default: "smart",
  // Shared across every route unless a route overrides them:
  onFallback: (from, to, err) => console.warn("fallback", { from, to, err }),
  timeout: 60_000,
});

await llm.complete(prompt); // uses "smart"
await llm.route("fast").complete(ticket);
await llm.route("cheap").complete(digestBody);
```

## How `route()` works

`createRouter({ routes, default: "smart" })` returns a router already bound to `"smart"`. Calling
`llm.route("fast")` returns a **new** router handle that shares the same cache, adapters, and
global config — it does not recreate anything expensive.

```ts
const fast = llm.route("fast");
const a = await fast.complete("classify A");
const b = await fast.complete("classify B"); // same route, same shared cache
```

An unknown name throws `BadRequest` immediately:

```ts
llm.route("smrt"); // BadRequest: Unknown route "smrt"
```

Route names are typed from the config object. With `as const` inference (the default for
`createRouter`), TypeScript rejects typos at compile time:

```ts
await llm.route("smrt").complete(prompt); // type error — not assignable to "fast" | "smart" | "cheap"
```

## What a route can set

A route is a `RouteConfig` — the same keys as a call (see [Configuration](/guide/configuration)):

| Typical use | Route fields |
| ----------- | ------------ |
| Model chain | `primary`, `fallbacks`, or `model` |
| Latency | `timeout`, `retry` |
| Cost / memoization | `cache`, `temperature`, `maxTokens` |
| Prompt defaults | `system` |
| Tools for that surface | `tools` |
| Provider extras | `raw` |

Router-only fields (`routes`, `default`, `onFallback`, `providers`, `adapters`) stay on
`createRouter()` — they are not per-route.

## Merge order with routes

For every call:

**call options > active route > global config**

```ts
const llm = createRouter({
  timeout: 60_000,
  fallbacks: ["openai/gpt-4o"],
  routes: {
    fast: {
      primary: "groq/llama-3.3-70b",
      timeout: 8_000,
      fallbacks: ["openai/gpt-4o-mini"],
    },
  },
  default: "fast",
});

// Uses route timeout 8s and route fallbacks — not the global ones.
await llm.complete(prompt);

// Call wins: 3s budget, still on the "fast" primary unless you also pass model.
await llm.complete(prompt, { timeout: 3_000 });

// Swap the model for one call; route fallbacks still apply.
await llm.complete(prompt, { model: "openai/gpt-4o-mini" });
```

`fallbacks` are replaced as a whole list, not concatenated. If the route sets
`fallbacks: ["openai/gpt-4o-mini"]`, the global `fallbacks` are ignored for that route.

## Routes vs per-call `model`

| Reach for… | When |
| ---------- | ---- |
| **Named routes** | The same tradeoff appears in many places (product surfaces, queues, cron jobs) |
| **Per-call `model`** | A one-off override on an otherwise shared route or global primary |
| **Both** | Route defines the chain; a rare call pins `model` for that request only |

Do **not** set `model` and `primary` on the same route object — that throws `BadRequest`. A call
passing `model` while the route has `primary` is fine.

## Default route

- With `default: "smart"`, bare `llm.complete()` uses the smart route.
- Without `default`, you must either set a top-level `primary`, or always call `llm.route(…)`.
- `createRouter()` requires `primary` **or** `routes` (at least one). Routes-only without a
  `default` and without calling `route()` will fail at call time with "No primary model
  configured" if nothing resolves a chain.

## Shared state

All `route()` handles share:

- The in-process `MemoryCache`
- Built-in and custom `adapters`
- `providers` overrides
- `onFallback`

So a cache write from `route("cheap")` can be hit by another handle that builds the same cache
key (same model chain + messages + temperature + maxTokens). See
[Caching & cost](/guide/caching-cost).

## Next

- [Fallback & retry](/guide/fallback) — what happens when a route's primary is down
- [Configuration](/guide/configuration) — full option reference
- [Caching & cost](/guide/caching-cost) — when `cheap` actually saves money

---

<!-- source: https://www.llm-sdk.dev/guide/testing -->

# Testing

There is **no** shipped mock provider. Tests use the same extension point as a custom gateway:
implement `Adapter`, register it under a name, route `name/model` to it.

## Minimal fake

```ts
import { createRouter, ProviderError, type Adapter } from "llm-sdk-js";

class FakeAdapter implements Adapter {
  readonly name = "fake";
  private calls = 0;

  async complete() {
    if (this.calls++ === 0) {
      throw new ProviderError("rate limited", {
        kind: "rate_limit",
        provider: this.name,
      });
    }
    return {
      text: "second reply",
      usage: { input: 0, output: 0 },
      toolCalls: [],
    };
  }

  async *stream() {
    yield {
      type: "done" as const,
      usage: { input: 0, output: 0 },
      toolCalls: [],
    };
  }
}

const llm = createRouter({
  primary: "fake/primary",
  fallbacks: ["fake/backup"],
  adapters: { fake: new FakeAdapter() },
  retry: { attempts: 1 },
});

const res = await llm.complete("hi");
// res.text === "second reply"
// res.attempts[0].error === "rate_limit"
```

Both model refs share **one** adapter instance (keyed by provider name `fake`). Call counters and
queues are therefore global to that adapter — construct a **fresh** router (or adapter) per test
when order matters.

## What to assert

| Behavior | How to trigger |
| -------- | -------------- |
| Failover | First `complete()` throws `ProviderError` with retryable / skip-retry kind; second returns OK |
| No failover on bad request | `kind: "bad_request"` → expect `BadRequest`, `onFallback` not called |
| Auth hops and notifies | `kind: "auth"` → next provider; `onFallback` fired |
| Content filter stop | `kind: "content_filter"` without opt-in → `AllProvidersFailed` |
| Content filter hop | same kind with `allowContentFilterFailover: true` |
| Cache hit | `cache: { ttl: "1h" }`, `temperature: 0`, identical prompt twice → `cached: true` |
| Route typing | `route("typo")` type error / runtime `BadRequest` |

## Stream failures

To test pre-buffer vs post-buffer failover, emit a few `delta` events then throw. The router
buffers ~40 characters before the first yield to the caller; failures before that can still fail
over.

The repo's own suite uses `test/helpers/fake-adapter.ts` (not published) with
`fail` / `responses` / `streamFailAfterChunks` queues — copy that pattern or depend on your own
double.

## Live examples

`examples/*.ts` call **real** OpenAI (and optionally Anthropic for failover demos). They are not
unit tests. Copy `.env.example` → `.env` and run:

```bash
node --env-file=.env node_modules/.bin/tsx examples/fallback.ts
```

## CI shape

```bash
npm run typecheck   # src + test + examples
npm test            # vitest, no network
npm run build
```

Do not put real API keys in unit tests. Prefer `adapters` fakes for policy coverage and a small
optional integration job for live smoke if you need it.

## Next

- [Fallback & retry](/guide/fallback) — which `kind` values to throw
- [Errors](/guide/errors) — `ProviderError` fields
- [Configuration](/guide/configuration) — `adapters` map

---

<!-- source: https://www.llm-sdk.dev/guide/tools -->

# Tools

`llm-sdk` serializes tool definitions into each provider's wire format, parses tool calls out of
the response, and — when you push results back onto `messages` — serializes the round trip
correctly for OpenAI-compatible and Anthropic APIs.

It does **not** execute tools or run an agent loop. You own dispatch.

## Declare tools

```ts
const tools = [
  {
    name: "getSubscription",
    description: "Look up a Northwind Analytics customer's plan and seat usage",
    schema: {
      type: "object",
      properties: {
        accountId: { type: "string" },
      },
      required: ["accountId"],
    },
  },
];

const res = await llm.complete({
  messages: [{ role: "user", content: "Is acct_104 over their seat limit?" }],
  tools,
});
```

`schema` may be:

- A plain JSON Schema object (passed through), or
- A Zod schema (v3 or v4) — converted best-effort via duck-typing (`object`, `string`, `number`,
  `boolean`, `array`, `enum`, `literal`, `optional` / `nullable` / `default`). Unrecognized
  wrappers become a permissive `{}` rather than a wrong schema.

## Read `toolCalls`

```ts
if (res.toolCalls.length > 0) {
  const call = res.toolCalls[0];
  call.id; // provider id — required when you answer
  call.name; // "getSubscription"
  call.args; // { accountId: "acct_104" } — already JSON-parsed when possible
}
```

`finishReason` is parsed inside adapters today but is **not** yet exposed on `CompleteResult`.
Use `toolCalls.length` to decide whether to dispatch.

## Full round trip

```ts
const messages = [{ role: "user", content: "Is acct_104 over their seat limit?" }];

const first = await llm.complete({ messages, tools });

if (!first.toolCalls.length) {
  console.log(first.text); // model answered without tools
} else {
  // 1. Echo the assistant turn, including the tool calls it made
  messages.push({
    role: "assistant",
    content: first.text,
    toolCalls: first.toolCalls,
  });

  // 2. Answer each call by id (your code runs the tool)
  for (const call of first.toolCalls) {
    if (call.name !== "getSubscription") {
      throw new Error(`Unknown tool: ${call.name}`);
    }
    const result = getSubscription(call.args as { accountId: string });
    messages.push({
      role: "tool",
      toolCallId: call.id,
      content: JSON.stringify(result),
    });
  }

  // 3. Ask again — model sees the tool results
  const final = await llm.complete({ messages, tools });
  console.log(final.text);
}
```

### Message fields that matter

| Role | Extra fields |
| ---- | ------------ |
| `assistant` | `toolCalls?: ToolCall[]` — echo from `CompleteResult.toolCalls` |
| `tool` | `toolCallId: string` — the `ToolCall.id` you are answering; `content` is the result body |

### Provider differences (handled for you)

- **OpenAI / Groq / Ollama:** `tool_calls` on the assistant message; `tool` role + `tool_call_id`.
- **Anthropic:** `tool_use` / `tool_result` content blocks. Consecutive `tool` messages are merged
  into the single `user` turn Anthropic requires.

## Streaming and tools

`stream()` accumulates tool call deltas from the wire and surfaces the finished calls on the
`done` event's `toolCalls` — same shape as `CompleteResult.toolCalls`. Text deltas still arrive
incrementally; tool calls only resolve once the stream completes, since providers send them as
partial JSON chunks that aren't safe to parse until the block closes.

A **cache hit** replayed through `stream()` surfaces `toolCalls` that were stored from an
earlier call — the cache entry includes them.

## Caching

The cache key includes `tools` and `raw` alongside the model chain, messages, temperature, and
`maxTokens`. Changing the tool set or provider-specific `raw` params will miss the cache even
when the prompt text is identical.

## Next

- [Structured output](/guide/extract) — JSON once, no tool loop
- [Testing](/guide/testing) — fake an adapter that returns `toolCalls`
- `examples/tools.ts` — end-to-end loop against a real provider

---

<!-- source: https://www.llm-sdk.dev/api/create-router -->

# createRouter

```ts
import { createRouter, type Router } from "llm-sdk-js";

const llm = createRouter({
  primary: "anthropic/claude-sonnet-4-5",
  fallbacks: ["openai/gpt-4o"],
});
```

## Signature

```ts
function createRouter<const C extends RouterConfig>(config: C): Router<RouteNames<C>>
```

- Requires `primary` **or** `routes` (or both).
- Validates each config layer for `model` + `primary` conflicts.
- If `default` is set and `routes` exist, the returned router is already bound to that route.
- Route name generics are inferred from `routes` keys.

Throws `BadRequest` on invalid config.

## `Router` methods

### `complete(input, options?)`

```ts
complete(input: string | CompleteInput, options?: CallOptions): Promise<CompleteResult>
```

- `input` string → single user message (plus optional `system` from options/config).
- `CompleteInput`: `{ prompt?, messages?, system?, maxTokens?, temperature?, tools? }`.
- Walks the model chain with retry/failover/cache/cost. See [Fallback & retry](/guide/fallback).

### `stream(input, options?)`

```ts
stream(input: string | CompleteInput, options?: CallOptions): StreamHandle
```

```ts
interface StreamHandle {
  [Symbol.asyncIterator](): AsyncIterator<StreamChunk>; // { text, done? }
  result(): Promise<CompleteResult>; // drains if needed
}
```

- Real SSE from providers; ~40 character failover buffer.
- No same-provider retries; tool calls are accumulated from streamed deltas and surfaced on
  `result().toolCalls` once the stream completes.
- Shares cache with `complete()`.

### `extract(input, options?)`

```ts
extract<S, T = InferSchemaOutput<S>>(
  input: ExtractInput<S>,
  options?: CallOptions,
): Promise<ExtractResult<T>>
```

See [Structured output](/guide/extract).

### `route(name)`

```ts
route(name: R): Router<R>
```

Returns a handle bound to that named route (shared cache/adapters). Unknown name → `BadRequest`.
See [Named routes](/guide/routes).

## Config overview

Full field docs: [Configuration](/guide/configuration).

| Area | Keys |
| ---- | ---- |
| Chain | `primary`, `model`, `fallbacks`, `routes`, `default` |
| Policy | `retry`, `timeout`, `allowContentFilterFailover`, `onFallback` |
| Generation | `temperature`, `maxTokens`, `system`, `messages`, `tools`, `raw` |
| Cache | `cache` |
| Connectivity | `providers`, `adapters` |

## Related

- [Types](/api/types) — `RouterConfig`, `CompleteResult`, …
- [Errors API](/api/errors) — thrown classes

---

<!-- source: https://www.llm-sdk.dev/api/errors -->

# Errors API

```ts
import {
  BadRequest,
  AllProvidersFailed,
  ProviderError,
  type ErrorKind,
} from "llm-sdk-js";
```

Behavioral guide: [Errors](/guide/errors).

## `BadRequest`

```ts
class BadRequest extends Error {
  name: "BadRequest";
  constructor(message: string, options?: { cause?: unknown });
}
```

Fatal for the call. Never retried. Never failed over.

## `AllProvidersFailed`

```ts
class AllProvidersFailed extends Error {
  name: "AllProvidersFailed";
  readonly attempts: AttemptRecord[];
  constructor(attempts: AttemptRecord[], message?: string);
}
```

Default message: `"All providers failed"`. Content-filter default stop uses
`"Content was filtered by the provider"`.

## `ProviderError`

```ts
type ErrorKind =
  | "rate_limit"
  | "timeout"
  | "overloaded"
  | "server_error"
  | "network"
  | "auth"
  | "bad_request"
  | "content_filter"
  | "unknown";

class ProviderError extends Error {
  name: "ProviderError";
  readonly kind: ErrorKind;
  readonly provider: string;
  readonly retryable: boolean;
  readonly status?: number;
  readonly retryAfterMs?: number;

  constructor(
    message: string,
    options: {
      kind: ErrorKind;
      provider: string;
      retryable?: boolean;
      status?: number;
      retryAfterMs?: number;
      cause?: unknown;
    },
  );
}
```

Default `retryable` is `true` for `rate_limit`, `timeout`, `overloaded`, `server_error`, and
`network`; otherwise `false` unless overridden.

Use when implementing a custom `Adapter`. See [Testing](/guide/testing).

## Related

- [Fallback & retry](/guide/fallback) — policy by `kind`
- [createRouter](/api/create-router)

---

<!-- source: https://www.llm-sdk.dev/api/types -->

# Types

Public TypeScript shapes exported from `llm-sdk`. For behavior, prefer the guide pages; this is
the field checklist.

## Model refs

```ts
type ProviderName = "openai" | "anthropic" | "groq" | "ollama" | "local";
type ModelRef = `${ProviderName}/${string}` | (string & {});
```

Custom adapter names are allowed at runtime (`"acme/fast"`); the branded template helps
autocomplete for built-ins.

## Messages

```ts
type Role = "system" | "user" | "assistant" | "tool";

interface Message {
  role: Role;
  content: string;
  name?: string;
  toolCalls?: ToolCall[]; // assistant — echo from CompleteResult
  toolCallId?: string; // tool — id being answered
}

interface ToolCall {
  id: string;
  name: string;
  args: unknown;
}

interface ToolDefinition {
  name: string;
  description?: string;
  schema?: unknown; // JSON Schema or Zod
}
```

## Options

```ts
interface RetryOptions {
  attempts?: number; // default 1
  baseDelay?: number; // default 500
  maxDelay?: number; // default 10_000
}

interface CacheOptions {
  ttl?: string | number;
  includeNonDeterministic?: boolean;
}

interface CallOptions {
  model?: ModelRef;
  primary?: ModelRef;
  fallbacks?: ModelRef[];
  retry?: RetryOptions;
  timeout?: number;
  cache?: CacheOptions | false;
  temperature?: number;
  maxTokens?: number;
  system?: string;
  messages?: Message[];
  raw?: Partial<Record<ProviderName, Record<string, unknown>>>;
  tools?: ToolDefinition[];
  allowContentFilterFailover?: boolean;
}

interface RouteConfig extends CallOptions {}

interface ProviderOverrideConfig {
  apiKey?: string;
  baseUrl?: string;
  timeoutMs?: number; // accepted on the type; not applied by built-in adapters today
}

interface RouterConfig extends CallOptions {
  routes?: Record<string, RouteConfig>;
  default?: string;
  onFallback?: (from: ModelRef, to: ModelRef, err: unknown) => void;
  providers?: Partial<Record<ProviderName, ProviderOverrideConfig>>;
  adapters?: Record<string, Adapter>;
}
```

## Results

```ts
interface Usage {
  input: number;
  output: number;
}

interface AttemptRecord {
  provider: string;
  model: string;
  error?: string;
  ms: number;
}

interface CompleteResult {
  text: string;
  provider: string;
  model: string;
  usage: Usage;
  cost: number;
  unknownModel: boolean; // true if `model` has no price table row — cost is 0, not free
  cached: boolean;
  latencyMs: number;
  attempts: AttemptRecord[];
  toolCalls: ToolCall[];
}

interface ExtractResult<T = unknown> {
  data: T;
  text: string;
  provider: string;
  model: string;
  usage: Usage;
  cost: number;
  unknownModel: boolean;
  cached: boolean;
  latencyMs: number;
  attempts: AttemptRecord[];
}
```

## Streaming

```ts
interface StreamChunk {
  text: string;
  done?: boolean;
}

interface StreamHandle {
  [Symbol.asyncIterator](): AsyncIterator<StreamChunk>;
  result(): Promise<CompleteResult>;
}
```

## Adapter contract

```ts
interface AdapterRequest {
  model: string;
  messages: Message[];
  temperature?: number;
  maxTokens?: number;
  tools?: ToolDefinition[];
  raw?: Record<string, unknown>;
  signal?: AbortSignal;
}

interface AdapterResponse {
  text: string;
  usage: Usage;
  toolCalls: ToolCall[];
  finishReason?: string;
}

type AdapterStreamEvent =
  | { type: "delta"; text: string }
  | { type: "done"; usage: Usage; toolCalls: ToolCall[]; finishReason?: string };

interface Adapter {
  readonly name: string;
  complete(request: AdapterRequest): Promise<AdapterResponse>;
  stream(request: AdapterRequest): AsyncIterable<AdapterStreamEvent>;
}
```

## Extract helpers

```ts
interface ExtractInput<S = unknown> {
  prompt: string;
  system?: string;
  schema: S;
  schemaDescription?: string;
}

type InferSchemaOutput<S> = /* Zod / Standard Schema inference, else unknown */;
```

## Related

- [createRouter](/api/create-router)
- [Configuration](/guide/configuration)
- [Errors API](/api/errors)