The Hidden Cost of Truncation

Introduction

Truncation is any cut made to fit a limit. In a system built on a large language model it happens in four places: documents are cut to fit training sequences, inputs are cut to fit the context window, the model’s working memory drops what it has already read, and answers stop at a cap on tokens, the word pieces a model reads and writes. For a business, the useful question is not whether a model’s window is big enough but what never reached it, and whether anything said so. Our reading of the research is that what a cut costs depends more on what it deletes than on how much, that business records put what matters where common cuts fall, and that the cost stays hidden because a model given incomplete evidence usually answers anyway. A disclosure up front, because it bears on how you read this: Ardenus sells a data layer that retrieves and computes over an operator’s records instead of pasting them into a model’s window, so a finding that truncation is harmful is convenient for us. Weigh it knowing that. Several of the studies below carry vendor ties of their own, and we name them as we go.

• Not all of your data reaches the model. In one peer-reviewed study, 85 of 100 open-source applications had code that builds prompts larger than the model’s limit, and 22 of the same 100 lost data at ingestion, before the window came into play.
• What a cut costs depends on what it deletes. The same cut is nearly free when the answer sits in the part that is kept and fatal when it does not, so a benchmark that finds truncation harmless has mostly found where its answers sit.
• Business records are not shuffled. They are appended over time and queried in aggregate, so cuts that keep either end or cap the rows tend to fall on what some question needs. That is our inference: no study measures truncation on service or billing records.
• The model cannot see what it was not given. In controlled tests, models handed evidence-deleted input mostly answered rather than abstained, and found deleted lines far less well than added ones. That most people do not know their data was cut is our inference, since no study we found measures it.

What truncation is

“Truncation” is one word for more than a dozen operations, and they do not all hurt. The map below is our mapping, not any single source’s: it sorts the operations by where in the pipeline the cut happens.

Two further senses of the word cut nothing from your data. Truncation sampling trims the model’s own list of candidate next words: a peer-reviewed survey files top-k, nucleus and similar methods as “truncation” on the next-token distribution, separate from temperature. Representation truncation shortens embedding vectors or the rank of weight matrices, and can even help: removing about 99 percent of one weight matrix’s rank, chosen per task, raised GPT-J’s accuracy on one fact benchmark from 13.1 to 24.0 percent. Neither says anything about whether your records reached the model, and the rest of this essay sets them aside.

Training-side truncation matters least to an operator. Splitting only the documents longer than a training sequence, instead of cutting at every boundary, raised four task averages by 0.07 to 3.89 accuracy points in the contrast that isolates truncation, one training run per setting with no significance test, by authors at AWS. In reinforcement learning, leaving attempts that hit the length cap out of the training signal, instead of penalizing them, lifted one math score from 30 to 36 in a ByteDance Seed study that added one change at a time. So “does truncation hurt?” has no single answer. The useful question is what a given cut removed, and who was told.

Not all of your data reaches the model

In a manual study of 100 open-source applications that combine a language model with retrieval, Shao and colleagues found that 85 “construct prompts that exceed LLM context limit,” the most common of the 13 defect patterns they assessed across all 100 apps (ICSE 2025). When that happens, they write, “the agent will truncate the input, leading to inaccurate responses.” In their examples, one app sends the whole chat history and passes a 4,096-token limit after three or four rounds, and another pastes all the text extracted from a PDF, so a book-length file gets cut. The 85 counts applications whose code contains the pattern, not a rate of lost requests, and the authors mark the same defect as causing crashes too; they do not report how often an overflow goes unannounced.

Data is also lost before the window comes into play. In the same study, 22 of the 100 apps mishandled documents as they were loaded, for example ignoring small files, “wrongly regarding them as empty files,” or splitting CSV rows mid-passage with fixed-size chunks, and 15 prepared text for search improperly, for example cutting words in the middle. A short paper on three retrieval systems names the failure “Not in Context”: documents with the answer “were retrieved from the database but did not make it into the context for generating an answer,” lost in a consolidation step that the paper says exists to cope with the model’s token and rate limits (CAIN 2024; no frequency reported). Some builders know they are choosing what to drop. One engineer of 26 interviewed in a study of product-copilot builders by Microsoft, GitHub and Carnegie Mellon authors described having to “selectively truncate because it won’t all fit into the prompt” (SANER 2025).

Some of the loss is silent, though the evidence for that comes from two preprints, and no peer-reviewed study we cite reports how silent and loud failures split. Of 255 bugs annotated in the issue trackers of four agent projects, 11 were triggered by context overflow: 6 crashed, 4 were silent and 1 hung. In one silent case, automatic compaction dropped the agent’s earlier file edits, and “no crash or warning tells the user that the task state was lost.” In another preprint, the code that ran a set of alignment benchmarks “silently right-truncated the test question” on long prompts padded with worked examples, cutting the question itself out of the model’s view until a code review found it.

Token limits were 2.0 percent of 2,364 coded questions from OpenAI’s developer forum (TOSEM 2025). That small share is consistent with loss going unnoticed, but no study tests that, and we offer it only as our reading. And overflow can fail loudly: without any context management, coding agents in a preprint study of agent harnesses, the software that runs them, simply ended with an error when their history outgrew the window, losing 78.7 percent of one benchmark’s tasks to overflow at a 32,000-token window, averaged over four models: a loud failure, not a silent one.

Where the cut falls

In the usable size of a context window, we cited a preprint suggesting that, for the strongest models, what fills the window can matter more than how much of it there is. The studies here point the same way for cuts. On the standard web and document ranking benchmarks, a cut to the opening is nearly free. Across more than 20 ranking models on MS MARCO, TREC Deep Learning and Robust04, no model built for long documents beat its own first-512-token version by more than 5 percent on average (IJCNLP-AACL 2025). Those models themselves read at most 1,431 tokens, but the one ranker given whole documents scored about the same as its 512-token version. The authors trace this to the benchmarks: “in the vast majority of cases, the first relevant passage occurred within the initial 512 tokens.” On MS MARCO FarRelevant, a set they built so that the relevant passage always starts at or after token 512, the same first-512 rankers, untrained on that set, scored 0.015 to 0.021 in mean reciprocal rank, a score from 0 to 1 that rewards putting the right document first: below the 0.052 of a random order. A BERT ranker that scores the document in chunks and keeps the best reached 0.297. The chart’s right panel plots it.

Which part a budget keeps decides the cost. Cutting EU legislation to its first 512 tokens moved one classifier’s micro-F1, an accuracy score that balances missed labels against wrong ones, only from 0.711 to 0.709, because the label-bearing header and recitals open each act (ACL 2019). Moving that header to the end of each document cost a 512-token BERT 3.88 micro-F1 points, on a 100-point scale, on documents longer than 512 tokens (ACL 2022). On European Court of Human Rights cases, 512 random tokens scored 79.0 micro-F1 against 73.5 for the first 512, though where the evidence sat was not measured (Findings of EMNLP 2022).

So a verdict that truncation is harmless is mostly a verdict about where a benchmark’s answers sit. Outputting a news article’s first three sentences as its summary nearly matches trained summarizers on CNN/DailyMail. On the 100 articles whose summary content sits latest, it scored 20.18 in average ROUGE, a word-overlap score, against 42.21 for the best sentences any extractor could pick, and trained extractors reading each article’s first 100 sentences did little better, at 20.62 to 22.12 (EMNLP-IJCNLP 2019).

Deletion can also cost less than distraction. In multi-document summarization, dropping the middle documents kept 91.32 to 100.37 percent of the all-documents score on Multi-News, while filling the middle with random documents from other examples kept only 78.13 to 89.81 percent, though the reference summaries may under-weight middle documents (ACL 2024; Salesforce co-authors, with Salesforce’s Xgen-7B among the models).

Records are not shuffled

Unlike those benchmarks, business records are appended over time and queried in aggregate. No study we found measures truncation on service, CRM or billing records, so every transfer to them in this section is our inference: the mechanisms transfer, the magnitudes do not. In a record kept oldest first, a cut that keeps the start drops the newest entries, and one that keeps the end drops the history. We treat a patient record as the nearest peer-reviewed proxy for a long customer history.

The chart at the top of this page holds one model fixed and changes only what a token budget keeps. Myers and colleagues gave GPT-5.4-mini 200 real hospital stays, averaging 74,000 tokens of notes, and asked it to list every imaging procedure with its date and site (JAMIA 2026). Given the most recent 2,560 tokens, about 3.5 percent of an average stay, it scored 6.61 F1, and most of the loss was in recall: it found only 3.63 percent of the stay’s procedures. Given 20 retrieved chunks of 128 tokens, about the same amount of text chosen from across the stay by similarity to a fixed query for the task, it scored 36.68, more than five times as much. Given the most recent 120,000 tokens, the whole stay whenever it fits, it scored 48.32. Across the three models tested, and for retrieval across the paper’s two retrievers, the most recent notes scored 6.61 to 7.69, retrieval 36.68 to 43.94 and the 120,000-token window 44.85 to 49.40. This is one of three tasks, and the one where recency fails hardest, because it asks for a list that spans the whole stay. On antibiotic timelines the gap was smaller, a Jaccard overlap score of 0.3142 to 0.3748 for the most recent 2,560 tokens against 0.4426 to 0.5441 for 20 chunks from either retriever, and for naming the stay’s key diagnoses the most recent notes did slightly better than retrieval on two of three targets.

A large recent window can come close to reading everything when the questions are about recent events. On 303 clinician-written instructions over real patient records, GPT-4 given the most recent 2,000 or so tokens of each record was judged correct 51.8 percent of the time; given the most recent 32,000 or so, 60.1 percent; reading the whole record in chunks, 65.0 percent (AAAI 2024). But those questions skew recent: 55.3 percent draw only on the final quarter of the patient’s timeline (npj Digital Medicine 2025), though that is a position in time, not in tokens, and neither study checked whether each answer’s evidence survived the cut.

Aggregates are where we expect a cut to do quiet damage. No language-model study we found deletes rows from a fixed table by a stated policy and measures the signed error of the resulting counts or sums, so the mechanism comes from databases. A running total or average over the rows read so far is a statistical estimate of the final answer, with meaningful error bounds, only if the rows arrive in random order; read in sorted or grouped order, the estimates are, in Hellerstein, Haas and Wang’s words, “wildly skewed” (SIGMOD 1997). Even a uniform random sample can miss small groups (SIGMOD 2017). Records sorted by date are the opposite of random order, so applying this to the first rows of an export pasted into a prompt is our inference, not a measured result.

When everything fits, the evidence on computing over the data is mixed. On the same 20 rows, gpt-3.5-turbo writing pandas code that was then executed scored 63.0 percent overall and 75.9 percent on numeric answers, while pasting the rows into the prompt scored 32.7 and 34.2 percent, though the pasted version did better on true-or-false questions (LREC-COLING 2024). In the paper’s worked counting example, the pasted rows produced 9 and the code produced the correct 12. But on 100 questions over tables that fit the models’ windows, reading the whole table beat retrieval and Text-to-SQL for all three models tested (Findings of EMNLP 2025; Zhejiang University with two Ant Group co-authors), a comparison with a fully visible table rather than with a cut one. Counting from random samples pasted into a prompt erred by 23.39 to 78.42 percent across two models, against 10.15 to 26.95 percent for labelling a sample one row at a time and 4.98 to 12.32 percent for an engine working from a stratified sample, in a NeurIPS 2024 evaluation where Google authors tested their own engine. But the comparison also changed the model, in-prompt counting was not worse on every query than the row-at-a-time baseline, and the errors carry no sign, so whether counts ran high or low is unknown.

What matters is whether a cut removes rows from what the model sees or from what the computation uses. A table agent shown 30 random rows of very large tables scored below one shown only the schema, the list of columns, in all six model-dataset pairs (NeurIPS 2024; Google authors evaluating Gemini among three models), but its code could still reach every row: the authors state this for their own method, and the baselines share its solver, so the cut cost what the model saw, not the data. When the cut falls on the rows the model must reason over, it is not cheap: on database-style questions over 64,000 tokens of rows that were all relevant, keeping the top 2,000 retrieved tokens scored 14.85 against 39.06 for the full context (ICLR 2025). Running a query over the full table and passing only the result helped where tables were cut hardest: on 128 questions that lost more than half their table to a 2,000-token cap, a baseline reading the capped table scored 33.3 percent and a method passing a SQL result scored 56.2 percent, in an observational comparison (NAACL 2024). Schema pruning applies the same cut to the list of tables a model is shown when it writes SQL from a question. Retrieving only the single most similar table cut execution accuracy on the BIRD benchmark to 38.01 percent, the share of generated queries that return the right answer, against 62.32 percent when every table of the target database was passed (PVLDB 2025; Google authors evaluating Gemini 1.5).

Conversations, agents and working memory

Long conversations and agent runs are where cuts happen on a schedule. On long generated conversations, gpt-3.5-turbo’s question-answering F1 rose from 23.9 with a window of about 4,000 tokens to 35.9 at about 16,000, though F1 on unanswerable questions fell from 34.8 to 14.8, and whether the needed turns had been deleted was not measured (ACL 2024). Cutting a gpt-4.1 agent’s history first-in, first-out to its last five turns lowered its AppWorld score from 56.0 to 45.8, and from 39.7 to 15.9 on hard tasks, though it scored higher on easy ones and whether the dropped turns held what the agent needed was not measured (ICML 2026; Microsoft authors proposing a competing compressor). Keeping everything is not safe either, the point of the diminishing returns of context: reading a whole chat history of about 115,000 tokens, which fit the window, scored 30.3 to 55.1 percent lower than reading only the sessions that held the evidence (ICLR 2025).

What is removed matters more than that something is. In a study of a coding agent’s history, an LLM step that removed only useless, redundant or expired content kept resolve rates within 1.0 point below to 2.0 points above the original while cutting input tokens by 39.9 to 59.7 percent. A blind baseline that erased the content of every long step once it was two steps old resolved 58 percent of issues against 65 percent, on a 100-instance subset with no significance test (FSE 2026; ByteDance co-authors testing their own method on Trae Agent, whose technical report two of them co-wrote).

The model’s working memory is cut too. When that cache evicts tokens, they are gone: in StreamingLLM’s own test, accuracy was 0.00 at every tested distance beyond the cache (ICLR 2024; Meta AI and NVIDIA co-authors). Across multi-turn sessions keeping 1/32 of that memory, two eviction methods averaged 15.5 and 20.9 against 48.7 for the full cache (ICLR 2025; mostly Microsoft authors, whose own methods are among those tested). The damage is uneven. At 10 percent retention, grade-school arithmetic fell 35.3 percent in relative terms while a general-knowledge test lost under 1 percent (ICML 2026). With a defense placed first in a compressed system prompt, several eviction policies kept a high share of the directive while evicting more of the defense, and leakage of the instructions rose, peaking at intermediate compression, shown only in plots (ACL 2026). And under one eviction method with only 64 cache entries kept, a model’s summary hallucination more than tripled (ACL 2026; Amazon authors proposing their own fix).

Cut-off answers

Output caps cut the other end. Small open reasoning models’ math scores depend heavily on the maximum-token cap, most sharply at 4,000 to 8,000 tokens: one 7-billion-parameter model scored roughly 0.27 on AIME24 at 4,096 tokens against about 0.54 at 32,768, read from a figure (COLM 2025). Force-stopping DeepSeek-R1’s thinking on grade-school math cost 3.2 percent of its accuracy at a 768-token cap and 40.2 percent at a 64-token cap, both relative (TMLR 2026). Forcing an answer after 20 percent of the reasoning kept 60.8 percent of originally correct answers on MATH-500 and 35.1 percent on GPQA (ICLR 2026).

How the cut is made matters. On AIME, simply cutting the output off at the budget did worse than forcing a final answer, which did worse than reserving tokens for the solution, read from a figure in a paper by Salesforce authors evaluating their own models (ICLR 2026). Even with tokens reserved, a coding model not yet trained for budgets solved under 10 percent of problems on the first try below a 4,000-token budget, against 58.1 percent unconstrained. Structured output is fragile under a cap, though the evidence is two preprints. Under a JSON schema with a 2,048-token cap, 49 of one model’s 100 responses hit the cap and 45 of those 49 were wrong, in a single-author preprint. With a hard cap of 1.1 times the target length, two small models produced valid JSON only 1 to 36 percent of the time under standard and grammar-constrained decoding, in a preprint by authors proposing their own decoder.

Silent cuts, confident answers

When the evidence is gone and nobody tells the model, the usual result for most models tested is an answer, not an abstention. With the supporting paragraph for one step of a multi-step question replaced, three 2022-era instruction-tuned models flagged the question as unanswerable with F1 of only 6.8 to 17.1 under a plain prompt (EMNLP 2023). Told explicitly to answer “unknown” if the context lacked the answer, 18 chat models given contexts with the answer deleted said so on 6.3 to 62.1 percent of items under strict matching (ICLR 2025; Salesforce AI Research). With the patient context deleted from clinical questions, every model shown abstained on well under half, read from a figure (NeurIPS 2025; Meta FAIR, evaluating Meta’s Llama models among others). In a workshop paper, a 7-billion-parameter model answered confidently and wrongly on 232 of the 366 items, about 63 percent, whose evidence a cache compressor had removed. These studies deleted or withheld the evidence rather than cutting the input to a length; that a silently cut input usually comes back as an answer is our inference from them. Whether retrieved context is sufficient in the first place is the subject of grounding and the reliability of AI.

Models are also poor at noticing what is missing. Asked to list what had been removed from a document, models that found added lines almost perfectly found deleted ones far less well: Claude-3.7-Sonnet scored 99.52 against 73.5 on poetry and 97.1 against 35.7 on GitHub pull requests (NeurIPS 2025). Marking each gap with a “<missing line>” placeholder raised detection by 16.4 to 19.6 F1 points, in a task where the model sees the original beside the cut copy, rather than with a note at the end of a cut input. In a preprint on absence questions, models often declared an item absent from partial evidence, 37.2 to 76.0 percent of the time on synthetic pairs under a plain prompt, though the item was absent from the complete source too, so those answers were unsupported rather than false. With explicit wording that a source was complete, their paired sensitivity, the share of matched pairs where they got both the complete source and its partial twin right, was 3.8 to 45.4 under a plain prompt and 94.9 to 100.0 once a stated rule and step-by-step reasoning were added; where coverage was only implied, it stayed at 6.4 to 15.1 even then. Nothing there was cut by a token budget. Even a benchmark that tells the model its input was trimmed, ZeroSCROLLS, never measures the notice’s effect; in 30 of 200 sampled cases GPT-4 replied that the trimmed context did not contain the answer, against 5 of 200 for Claude, statements by the models rather than a check of whether the evidence was cut (Findings of EMNLP 2023).

A generic permission to abstain trades one error for another. It sharply raised unanswerability detection in the EMNLP study but cut F1 on answerable questions by 8.3 percent in relative terms, and in the workshop paper, at 80 percent retention, it made the model refuse 49 percent of the 69 items whose evidence had survived, against 17 percent without it; as a count of unjustified refusals, 49 percent is an upper bound because survival there is a heuristic label.

The people reading the answer are not well placed to catch it either, though the evidence here is adjacent. Shown a model’s default explanations, participants told right answers from wrong ones far worse than the model’s own confidence did, an AUC, where 0.5 is a coin flip and 1 is perfect, of about 0.59 to 0.60 against 0.75 to 0.78 (Nature Machine Intelligence 2025). In a preprint study of public chat logs, users’ prompts presupposed internet access the deployment lacked, such as reading a link or writing scraper code for a site the model could not see, and only 25.3 percent of those conversations contained a first-person statement of what the model could not do, by a pattern match the authors call an underestimate. Some answers read as if the link had been visited. That content never reached the model by design, not because of a token limit. Some users do notice: a theme the authors call context understanding or memory limits made up 6.65 percent of the risks extracted from Reddit discussions of chatbots, with context-window limitations alone at 1.89 percent (CHI 2026). No study we found measures whether users know that part of their input can be dropped. That most people do not is our inference, and it is why we call the cost hidden.

Where the argument bends

The evidence against a simple story deserves its full weight. Many cuts are nearly free: on front-loaded benchmarks, as above, and on recent-skewed clinical questions. In MedAlign’s arXiv appendix, the most recent 32,000 tokens matched whole-record reading on the 259 items free of the content-filter errors that its headline figures count as wrong, 69.9 against 68.3 percent, and beat it on the longest quarter of records, 72.6 against 62.9 percent on 62 items with no significance test. Retrieval is not automatically better than truncation: on meeting summaries, a 1,024-token cut nearly matched BM25 keyword retrieval, 32.6 against 32.9 in ROUGE-1, for a fine-tuned BART model (Findings of EMNLP 2021). Nor does retrieval reliably beat reading a whole document: peer-reviewed comparisons on financial filings have no consistent winner, with gaps of 0.7 to 2.2 points either way on one claim-checking set (EMNLP 2024), and 0.3 to 12.0 points on another benchmark, where the whole document led for 5 of 12 models (ACL 2024).

Keeping the first rows was not consistently worse in the only controlled comparison of row-retention policies we found. On the 136 of 779 queries whose table overflowed the prompt, with four 2025 and 2026 models, keeping the head rows was strictly lowest in 5 of 12 model-task cells and strictly highest in 3, by our count; the one gap clearly beyond run-to-run noise went against head retention, 12 against 20 of 30 queries, derived from its scores (ICML 2026). Its questions ask for predictions, not exact counts or sums, and it has no uncut arm. Much reasoning is redundant: among responses that contained a correct solution, more than 80 percent had it in the first solution round (ICML 2025), though nothing was cut there, so it suggests rather than shows that an early cap can be cheap. Removing only stale agent context was close to free, as above, and overflow can fail loudly, with an error rather than a silent cut. Finally, most of the input-side evidence here comes from models released between 2019 and 2024. Among the studies here we found no comparison of head, tail and middle cuts on 2025 or 2026 frontier models, so the magnitudes for today’s models are open.

What this means for the operator

A bigger window alone is not the fix. In the chart, the window that held the whole stay scored highest, and larger windows raised overall scores in two other studies above, but neither of those measured whether the needed evidence then fit, and a bigger window fixes neither silent loss nor arithmetic done inside a prompt. Our synthesis of the evidence is to compute over the full data and pass the model the result, at least when the data is too large to fit. That fits the case made in the regimes of prediction, where models built for records do the predicting, and in the structure is the asset, where a language model looks up what changes rather than holding it. The support here is thin, an observational comparison and a vendor’s evaluation of its own method, and when the rows fit, the evidence is mixed. When the records do not fit, fetch what a question needs rather than letting a cut decide what survives: in the chart, retrieval scored more than five times what the most recent notes did. When they fit, whole-document and retrieval comparisons have no consistent winner.

Make every cut explicit to the model and to the person reading the answer, with a count such as “showing 50 of 1,200 visits”; the support for that is thinner, mainly one preprint plus narrower peer-reviewed evidence: in-line gap markers, and a generic note that a model “may not have complete information about your context,” which improved appropriate reliance in one of two tasks when nothing had actually been cut (CHI 2025). Reserve output budget for the answer instead of letting a cap fall mid-thought. Walk your own pipeline through the four rows of the table above, and log when and where each cut happens, because a cut nobody recorded cannot be checked. And test on your own records, for example with questions whose answers sit only in the oldest visits, since no study measures any of this on service histories, customer records or billing.

The disclosure from the top belongs here in full. Ardenus sells a data layer that retrieves and computes over an operator’s records instead of pasting them into a model’s window, so an essay concluding that truncation carries a hidden cost is convenient for us, and a reader should weigh it that way. To be fair to the other side, many cuts are nearly free, overflow can fail loudly, token limits are a small share of what developers ask about, the silent cases come from preprints, and that most people do not know their data was cut remains our inference. The supporting evidence carries vendor ties of its own, which we have named wherever authors evaluate their own models or methods. No result, saving, or metric is attributed to Ardenus here, no client or first-party data appears anywhere in this essay, and the chart transcribes published tables. You can read more of our research on the Ardenus articles hub, or see the platform itself on the technology page.