When Your AI Agent Quietly Gets a Half-Finished Answer

Every model API has a ceiling on how many tokens it will generate in one response. When a generation reaches that ceiling, the model stops mid-sentence, mid-object, or mid-tool-call — and the API returns HTTP 200 with a well-formed body. Nothing throws. In a multi-step agent pipeline, the next step receives that half-finished answer and treats it exactly like a complete one.

Three terms carry most of the weight here, so it is worth fixing them before the field names start multiplying. Max output tokens is the request-side ceiling on generated tokens — max_tokens on most APIs, max_output_tokens on OpenAI’s Responses API, n_predict on llama.cpp’s server. Truncation is what happens when a generation stops because it reached that ceiling rather than because the model was finished. And the stop reason — also called the finish reason, the done reason, or the stop type, depending on whose API you are holding — is the metadata field that tells you which of those two things happened. It is the only signal you get.

Anthropic states the premise better than any third party could. Its stop-reason guide opens with the instruction to read the field, and then draws the distinction the rest of this post depends on: “Unlike errors, which indicate failures in processing your request, stop_reason tells you why Claude completed its response generation.” A truncated response is not a failed request. It is a successful request whose result happens to be incomplete, and the two are indistinguishable unless you look at the metadata.

This post is a pattern reference, not a measurement. It collects the field name and the verbatim truncation value for eighteen API surfaces across fourteen vendor organisations, documents the cases where the signal is rewritten or lost between the engine and your client, and covers what detection actually requires. The per-model numbers — what each model’s maximum output is on each serving surface — are a companion dataset and are deliberately not built here; a later post owns that census. This one owns the failure mode.

The mental model most engineers bring to an API call is that a problem produces an exception. Truncation defeats that model at the protocol level: the request was valid, the server processed it correctly, and the response body is exactly what the schema describes. The only thing wrong with it is that the content stops early, and the protocol has no way to express “this is fine” versus “this is a fragment” other than a small metadata field that most client code never reads.

The consequence in a single-call application is mild — a user sees a sentence end abruptly and asks again. The consequence in a chain is different in kind. A research step that was cut at 40% of its output hands a plausible-looking summary to a planning step, which hands a plan built on incomplete evidence to an execution step. Every downstream stage behaves correctly on the input it was given. Nothing in the trace is red. This is the same structural problem as the orchestration mistakes covered in our agentic workflow anti-patterns guide, with one aggravating feature: the defect enters the chain from outside your code, carrying a success status.

Two design decisions make this worse than it needs to be. The first is that the client libraries mostly do not raise. The OpenAI Python SDK does define an exception for the case — LengthFinishReasonError, whose message reads “Could not parse response content as the length limit was reached” — but it is raised from the structured-output parse path, not from the ordinary create call. An open pull request against that SDK, openai-python #3589 (opened August 10, 2026), states the asymmetry in the vendor’s own repository: “The equivalent non-streaming client.chat.completions.create() never raises for the same response — it just returns a completion with finish_reason=“length”.” Structured output gets an exception; plain text gets a value you have to go and look at.

The second is that the field you have to look at is not the same field from one API to the next, and neither is the value it carries. That is section 03, and it is the part worth bookmarking.

The strongest documented instance of this failure is vllm-project/vllm issue #53269, filed and closed on August 21, 2026, and titled “[Bug]: streaming reports finish_reason=”tool_calls“ for a tool call truncated by max_tokens, hiding ”length“”. It matters because the signal was not merely absent. It was replaced with a value that means the opposite.

The report describes the mechanism precisely: “In the streaming chat handler, once any tool-call delta has been emitted the terminal reason is rewritten to “tool_calls” unconditionally. That discards the engine’s actual reason, so a generation cut short by max_tokens part way through the arguments is reported to the caller as a completed tool call.” The engine knew. vLLM’s own FinishReason enum documents length as “max_tokens was consumed, or max_model_len was reached”. The frontend overwrote it on the way out.

Two details make this more than a single bug report. The first is that the same request answered differently depending only on whether it was streamed. In the reporter’s words: “The non-streaming path does not have this problem — it only performs the translation when output.finish_reason == “stop”. So the same request answers differently depending only on stream, and the streaming answer is the wrong one: OpenAI reports “length” on truncation.” A test suite that exercises the non-streaming path and a production deployment that streams will disagree about whether the bug exists.

The second is the reproduction. The report carries a swept reproduction — greedy decoding, temperature 0, a fixed seed, and max_tokens swept across the truncation point on vLLM 0.27.0 with --enable-auto-tool-choice --tool-call-parser openai. Atmax_tokens values of 64, 80, 96 and 112 the non-streaming path reported length while the streaming path reported tool_calls, and the streamed arguments did not parse as JSON. From 128 upward both paths reported tool_calls and the arguments parsed. That is one person’s reproduction on one version in one bug report, not a measurement of vLLM in general — but it is a reproduction with the version string, the flags and the sweep written down, which is more than most reports of this class carry.

It is also not an isolated incident. Two earlier pull requests attack the same bug class: #46303, “Keep length finish_reason for max_tokens-truncated streaming tool calls” (opened June 21, 2026), and #47963, “Report finish_reason=’length’ for tool calls truncated by max_tokens in streaming” (opened July 8, 2026). Three dated items on one seam in ten weeks makes it a recurring class rather than a one-off, and the seam is structural: any layer that translates a terminal reason has an opportunity to lose the original.

A serving framework shipped a fix for the downstream half of the same problem a week later. NVIDIA’s Dynamo, in PR #13986 (opened August 28, 2026), describes dispatching a tool call from a first streamed fragment holding only a prefix such as {"path":"/a/very, then deduplicating the later fragments that would have completed the arguments. The PR’s own summary of the consequence is the cleanest sentence on the subject: “A harness that trusts the event — which is the entire point of the event — was handed truncated JSON to execute.” The same PR notes why bugs like this hide: “The failure is currently masked on the vLLM path, where long string arguments happen to arrive in one frame.” An unrelated framing coincidence upstream can keep a truncation bug invisible until the coincidence goes away.

The table below is the reason this post exists. It lists the field that carries the truncation signal, the truncation value quoted verbatim from the vendor’s own documentation or source tree, and what happens to that signal when the response is streamed — for eighteen API surfaces across fourteen vendor organisations. Documentation states are current as of retrieval on August 30, 2026; source-tree reads are from each project’s default branch on the same date.

Three structural facts fall out of it. Six different field names carry the signal (stop_reason, finish_reason, finishReason, stopReason, done_reason, stop_type), plus incomplete_details.reason nested inside a seventh shape. Seven distinct values mean truncation, and two of them do not contain the word “length” at all. And the same two words can mean opposite things across two APIs: on Anthropic’s Messages API stop_reason is the truncation field, while inside vLLM stop_reason is a different field entirely — “The stop string or token id that caused the completion to stop” — and finish_reason is the one you want.

Empty cells in that table are honest. Where a vendor’s retrieved documentation does not state the streaming placement, the cell says so rather than carrying an inference. One surface is missing altogether: Together AI’s finish_reason enum was not enumerated on the retrieved reference page, so it has no row here at all. A reference whose value is that every cell is quotable cannot fill one in from memory.

Three details in the table are worth pulling out because they break assumptions rather than merely varying. Anthropic splits truncation into two reasons — your ceiling (max_tokens) and the model’s context window (model_context_window_exceeded), the second of which is “currently typed only in the SDKs’ beta namespace” and requires the model-context-window-exceeded-2025-08-26 beta header on models earlier than Sonnet 4.5. Cohere merges the same two conditions into one MAX_TOKENS value covering both “the model’s context length” and “the value specified via the max_tokens parameter”. So a normalisation layer between them cannot round-trip the distinction in either direction.

And Gemini’s FinishReason enum is by far the largest surveyed, at 21 values. Exactly one of them — MAX_TOKENS — means you hit your ceiling. Four more (MALFORMED_FUNCTION_CALL, UNEXPECTED_TOOL_CALL, TOO_MANY_TOOL_CALLS, MALFORMED_RESPONSE) mean the output is unusable for other reasons. The common client shape — a switch on STOP with everything else falling into a default branch — swallows twenty distinct conditions into one.

Streaming is where consumers lose the signal without any bug being involved, because the field is not on the frames most code reads. Anthropic documents the placement exactly: “When using streaming, stop_reason is: null in the initial message_start event; Provided in the message_delta event; Not provided in any other events.” A consumer that subscribes to content_block_delta to append text to a buffer — which is the obvious way to write a streaming client — never touches the event that carries the reason.

The pattern repeats with local variations. OpenAI’s Chat Completions types finish_reason as nullable on the streaming chunk schema, where the non-streaming object has no null: every intermediate chunk carries null and exactly one chunk carries the value. The Responses API emits a dedicated terminal event, type: “response.incomplete”, which means a stream consumer listening only for response.completed waits for an event that is never coming. OpenRouter deliberately diverges from OpenAI’s spec by putting a content-free choice with the finish reason in the final chunk before [DONE] — correct behaviour that a strict OpenAI parser may discard as malformed. And llama.cpp’s native server does not send the field during a stream at all: “In streaming mode (stream), only content, tokens and stop will be returned until end of completion.”

Ollama’s OpenAI-compatible shim deserves a precise statement, because it is easy to get backwards. The max-tokens path is correct: Ollama’s internal DoneReasonLength stringifies to “length”, and the shim passes that through. It is the connection-closed path that degrades — DoneReasonConnectionClosed falls through the switch to the default branch and serialises to the empty string, and the shim’s FinishChunk then applies cmp.Or(r.DoneReason, “stop”), which fills the empty value with “stop”. So a generation cut off by a dropped connection reaches an OpenAI-shaped client as a clean, natural completion. A generation cut off by the token ceiling does not.

The practical rule for any streaming consumer is that reading the content stream is not enough. You need the terminal event, whatever it is called on your surface, and you need to record what it said — which is an observability requirement as much as a correctness one. Our guide to trace-quality mistakes covers the span-level side of that: the stop reason belongs on the span, next to the token counts, on every model call.

Fireworks writes the consequence into its API reference more plainly than anyone: “Also note that the message content may be partially cut off if finish_reason=“length”, which indicates the generation exceeded max_tokens or the conversation exceeded the max context length. In this case the return value might not be a valid JSON.” That sentence is doing something unusual for API documentation: it is telling you that a successful response may contain a payload your parser cannot read.

It helps to order the outcomes by what the consumer can still detect. The ladder below runs from “annoying but visible” to “undetectable from the response”, and the last rung is the one worth thinking about before it happens to you.

Rung 5 needs its label kept on. It is reasoning about the shape of the failure, not a documented case and not a measured one: a cut inside an array of objects can leave a document that is syntactically complete and semantically short — ten results where twenty were requested, and nothing anywhere saying so. No vendor documents a rate for it and there is no number to give. The two practitioner reports that come closest describe the adjacent shape, truncated tool-call arguments being silently replaced with an empty object.

This is also the boundary with a subject that already has its own post, and the boundary is sharp. Schema conformance — grammar- constrained decoding, strict tool-use modes, the portability limits of one shared schema across vendors — is the subject of our guide to structured output reliability in production. None of that machinery detects a cut, and structurally it cannot: conformance needs a complete document to check, and truncation is precisely the case where no complete document exists. The two problems look similar in a stack trace and have nothing in common in their causes. If you are working the parse-path angle specifically, our OpenAI structured outputs guide covers where the SDK does raise.

Fireworks documents one more wrinkle worth knowing, because it turns a truncation into a latency incident first: “when using JSON mode, it’s crucial to also instruct the model to produce JSON via a system or user message. Without this, the model may generate an unending stream of whitespace until the generation reaches the token limit, resulting in a long-running and seemingly ’stuck’ request.” The request that eventually returns a truncated response can spend its whole budget emitting nothing at all.

The intuitive place to look for a dangerous default is the vendor API. That is the wrong place. Across the surveyed surfaces, the APIs and first-party SDKs either require you to choose a ceiling or default to the model’s own maximum. The fixed low numbers are in the frameworks that sit between you and the API — the layer you are least likely to be reading when you debug a short answer.

LlamaIndex core ships DEFAULT_NUM_OUTPUTS = 256 # tokens in its constants module. Phrase that carefully: the framework-level default constant is 256, which is roughly 190 English words. Individual LLM integrations may override it, and this post did not check every one, so it is not a claim that every LlamaIndex call is capped at 256 — it is a claim about the constant the framework falls back to.

smolagents is the more surprising row, because the ceiling depends on which backend you selected. Its vLLM backend uses max_tokens=kwargs.get(“max_tokens”, 2048), while its Transformers backend resolves through a chain of optional keyword arguments and falls through to 1024. Two different silent ceilings inside one agent framework, differing by a factor of two, selected by a configuration choice that has nothing to do with output length.

Two entries in the safe group are worth reading as design lessons. Anthropic’s Messages API has no default because max_tokens is a required parameter — the design that forces an explicit choice is the design with no surprise in it. LangChain’s ChatAnthropic resolves the same required parameter dynamically from the model’s own max_output_tokens rather than pinning a constant, which is the right way to satisfy a required field on the caller’s behalf. Any advice you may have read that this integration pins a hardcoded 1024 is out of date against the current source.

One folk claim deserves killing while we are here. The widely repeated line that Hugging Face transformers defaults to 20 output tokens does not hold against the current source: the GenerationConfig constructor pops both max_length and max_new_tokens as None. A particular checkpoint’s generation_config.json can still set a low value, which is a per-model fact, not a library default — and worth checking on the checkpoint you actually load.

Raising a ceiling is not free, which is the one place this subject touches cost. Anthropic’s extended thinking shares the budget: “Requires a minimum budget of 1,024 tokens and counts towards your max_tokens limit”, so turning thinking on without raising the ceiling reduces what is left for the answer. And OpenAI’s max_output_tokens on the Responses API is “An upper bound for the number of tokens that can be generated for a response, including visible output tokens and reasoning tokens” — on a reasoning model, thinking can consume the whole ceiling and leave a structurally valid, visibly empty response. The pricing side of running with generous ceilings is covered in our work on long-context pricing thresholds.

Detection has one primary mechanism and three supporting ones, and it is worth being clear about which is which. The primary mechanism is reading the stop reason on every model call, on the surface you are actually calling, including the streaming terminal event. Everything else corroborates.

Anthropic’s own recommended handling for the tool-call case is a two-field check rather than a single comparison: test whether stop_reason equals “max_tokens”, then inspect the last content block and see whether its type is tool_use. That shape generalises. The reason alone tells you the generation was cut; the shape of what you are holding tells you what to do about it.

The sentinel idea is worth stating carefully because it is the pattern most likely to be mistaken for a feature. No vendor documents a canonical sentinel convention. Anthropic’s sample handling does the inverse — it appends a literal notice such as [Response truncated due to max_tokens limit] to the text it returns to a user — which is the same idea run the other way: a marker a consumer can look for. If you adopt one, your pipeline defines it, your prompts request it, and your validation enforces it.

Two things follow for how you instrument this. First, the check belongs in the pipeline, not in the model. Truncation is one of the clearest cases in the taxonomy of what an agent can and cannot verify about its own output: a model that was cut off cannot report that it was cut off, because the report would have to come after the token that never arrived. Only external metadata can tell you. Second, whatever you read should be recorded. LangChain surfaces the raw Anthropic value in response_metadata[“stop_reason”] and never raises on it, which makes a one-line assertion in the chain step the cheapest fix available. Observability vendors are treating this as a first-class case too — LiteLLM merged test coverage for empty, whitespace and truncated tool-call arguments in its OpenTelemetry output in August 2026.

Once you can see truncation, the response depends on what got cut, and the vendor guidance is more specific than it first appears. Anthropic’s stop-reason table prescribes “Raise max_tokens or continue the response” for the max_tokens case — two options for prose. For the tool-call case the same page prescribes something different: “If Claude’s response is cut off because it hit the max_tokens limit, and the truncated response contains an incomplete tool use block, you’ll need to retry the request with a higher max_tokens value to get the full tool use.” A full retry, not a continuation.

That distinction is the most useful operational line in this post. Continuation works on prose because prose concatenates: the second half attaches to the first and the result reads correctly. It does not work on a structured emission, because a tool call cut mid-arguments has no valid prefix to continue from — you are holding a fragment of a serialisation, and stitching a second fragment onto it produces something no schema described. The vendor’s own guidance is that you re-run it at a higher ceiling.

For the second Anthropic value, model_context_window_exceeded, the same table’s prescription is simply to “Treat the response as truncated” — raising your request ceiling does not help when the model’s own window is what filled. That is a prompt-size and context-management problem rather than a parameter problem, and it belongs with the runtime context work in our context engineering playbook.

One more habit is worth building in early: decide what a truncated step should do to the run. Discarding the partial output and retrying costs tokens; passing it downstream costs correctness; failing the run loudly costs a restart. Whichever you choose, choose it explicitly and log the reason, because the alternative — the default in most pipelines today — is to pass the fragment along and find out three steps later.