mordndocs

Streaming reliability

"Streaming works on localhost but arrives as one blob in production" is the most common deployment failure for SSE chat backends. This page covers the three things that keep token streaming healthy in production: errors that don't vanish, proxies that don't buffer, and a probe that proves it.

For render throttling and tool-loop bounds, see Streaming.

Errors are logged by default

The AI SDK swallows stream errors so a mid-stream failure doesn't crash the server — which is exactly how broken production deploys end up with silent, empty logs. The handler surfaces them by default: it logs stream errors to the server console independently of onError.

code
app/api/chat/[[...chat]]/route.ts
export const { GET, POST, DELETE } = createChatHandler({
  getUserId: getChatUserId,
  model: anthropic('claude-sonnet-4-5'),
  // onError only maps the USER-FACING message; it does NOT silence the log.
  onError: (error) => 'Something went wrong. Please try again.',
  // logErrors defaults to true — the server-side log fires regardless of onError.
});
logErrorsbooleandefault: true

Log stream errors to the server console. Independent of onError (which only maps the user-facing copy). Set false only if you forward errors elsewhere and want to suppress the built-in log.

onError(error: unknown) => string

Map a stream error to the user-facing string the widget shows. Providing it does not silence the server log — a production failure (bad key, rate limit, wrong URL) never disappears into empty logs, the #1 documented streaming pitfall.

Warning

Do not reach for logErrors: false to quiet the console. If a real failure (expired API key, gateway rate limit) stops being logged, a broken deployment becomes invisible. Keep it on unless you are forwarding errors to another sink.

Anti-buffering

A genuinely streamed SSE response sends many small chunks over time. A reverse proxy or CDN that buffers the response collapses that into one late blob — the answer appears all-at-once, and it is invisible until a human notices. The handler sets the response headers that tell intermediaries not to buffer (e.g. disabling proxy buffering and chunk transformation), but a proxy you control can still override them.

If responses arrive all-at-once in production

  • Disable response buffering on any reverse proxy in front of the app (for nginx, proxy_buffering off; and X-Accel-Buffering: no on the route).
  • Disable CDN/edge buffering or "full response" caching for the chat route.
  • Confirm no gzip/transform layer is accumulating the whole body before flushing.
  • Then re-run the probe below against the deployed URL.

Proving it: streamHealthCheck

streamHealthCheck makes buffering testable. Point it at your deployed chat route and it reports whether the response actually arrived incrementally (many chunks over time) or as a single late blob (buffered). Run it in CI against a staging deploy, or as a post-release smoke test, so a buffering misconfiguration fails the pipeline instead of reaching users. It validates the transport, not model correctness — it only cares that bytes stream back over time.

code
scripts/stream-health.ts
import { streamHealthCheck } from '@mordn/chat-widget/server';
 
const result = await streamHealthCheck({
  url: 'https://staging.example.com/api/chat',
  headers: { Cookie: process.env.SESSION_COOKIE! }, // must satisfy your getUserId
});
 
if (!result.ok) {
  console.error(result.note);
  process.exit(1); // fail CI on a buffered/broken stream
}
console.log(`Streamed ${result.chunks} chunks; first byte at ${result.firstChunkMs}ms.`);

Options

urlstringrequired

Absolute URL of the mounted chat route's POST endpoint (e.g. https://staging.example.com/api/chat).

headersRecord<string, string>

Headers used to authenticate the probe. Your getUserId must accept these (a session cookie or a test bearer token). Without valid auth the endpoint returns 401 and the probe reports ok: false.

messagestringdefault: "ping"

Probe message text.

conversationIdstring

Conversation id for the probe turn. Defaults to an ephemeral health-* id.

timeoutMsnumberdefault: 20000

Abort the probe after this many ms.

fetchImpltypeof fetch

Custom fetch implementation (defaults to global fetch).

Result

code
interface StreamHealthResult {
  ok: boolean;              // true when it streamed incrementally with a 2xx
  status: number;           // HTTP status (0 if the request never completed)
  chunks: number;           // non-empty network reads observed
  firstChunkMs: number | null; // ms to the first non-empty chunk
  totalMs: number;          // ms to stream end
  likelyBuffered: boolean;  // heuristic: arrived as ~one late chunk
  note: string;             // human-readable summary / remediation pointer
}

streamHealthCheck never throws — failures come back as { ok: false, note }. The likelyBuffered heuristic flags a response that arrived as a single chunk, or whose first chunk landed right as the stream ended — both strong signs an intermediary buffered the body.

Note

A 401 from the probe usually means the auth headers don't satisfy your getUserId, not that streaming is broken. Fix the probe's credentials first, then interpret likelyBuffered.

See Streaming for throttling, stopWhen, and the onChatFinish telemetry hook.