Streaming model responses: partial output and cancellation

How the stream arrives

Providers stream over server-sent events. Under the WHATWG parsing rules, a blank line ends an event and the stream must be UTF-8. If the connection ends mid-event, “any pending data must be discarded”. A cut connection can therefore drop the last event without any error.

Know when it finished

A closed connection does not mean a finished answer. Wait for the provider’s completion event. OpenAI’s Responses API emits response.completed. Anthropic’s Messages API sends message_delta with a stop_reason, then message_stop. Anthropic also says errors such as overloaded_error can arrive inside the stream, after a 200 response has started.

Partial JSON

Tool arguments stream as fragments. Anthropic’s docs say the deltas are partial JSON strings, and the final input is always an object. Accumulate the fragments and parse after content_block_stop. Do not act on a tool call before then.

Cancellation

When the user cancels, abort the request. In browsers, AbortController.abort() rejects an in-flight fetch with an AbortError. Treat that rejection as an expected outcome of cancelling.

What to store

Keep the text received so far and mark it with a status: complete, cancelled, or errored. Never present cancelled text as a full answer. Anthropic notes that tool use and thinking blocks cannot be partially recovered, so discard unfinished ones. OpenAI warns that partial output is harder to moderate, so check text before showing it where that matters.