An AI request has a limited context budget. Instructions, retrieved passages, conversation history, tool definitions and outputs all compete for that budget. The usable limit and accounting rules depend on the model and interface.
A large context window does not remove the need to decide what belongs in it. The application still needs to supply the right information, preserve authoritative state and keep unrelated or restricted material out.
Separate context from durable state
Context is information available to a particular model invocation. Durable state is the application’s record of what has happened: the current task, verified values, pending approvals and completed external actions.
A model-generated conversation summary can help prepare the next request. It should not become the sole authority for whether a payment was approved or a record was updated. Those facts belong in controlled application storage and the connected system of record.
When a workflow resumes after interruption, rebuild its context from the current state. Do not rely on the model remembering that an earlier action failed or that an approval was withdrawn. This is one responsibility of an AI harness.
Budget for the work that remains
Measure context use on realistic tasks. Long documents, verbose tool schemas and unbounded tool responses can consume capacity before the model reaches the step that matters.
A practical request builder distinguishes information that must stay from information that can be retrieved again. Keep the task, relevant instructions and verified constraints available. Retrieve the source passages needed for the next decision rather than replaying every document from every earlier step.
Leave room for the model’s output and any additional tokens the selected interface requires. Exact limits are model-specific; avoid a single hard-coded token assumption across providers.
Summaries are derived information
Compression can omit a qualification, merge two similar records or lose the source of a number. If a summary is used later, preserve references to the records from which it was produced and test whether important constraints survive.
A rolling conversation window is not inherently wrong. It is appropriate for some interfaces and insufficient for others. The question is what the next step needs and which authoritative records must remain accessible.
For a document-review workflow, keep extracted fields with their page references and validation status. A narrative summary is useful for reading, while the structured record supports checking and recovery.
Treat incoming content as untrusted
Documents and tool responses may contain malicious instructions or misleading claims. Adding them to context does not give them authority to change the system’s permissions.
Keep credentials out of prompts. Restrict tool access through application-side checks. Limit response sizes and validate tool outputs against the expected schema before using them. Avoid copying unrestricted tool results into logs or cross-user caches.
Where context includes personal information, review the complete data path, including model processing, diagnostics and evaluation storage. A private endpoint or local database does not describe every place that data may travel.
Evaluate context failures directly
Test cases should include long histories, conflicting document versions, interruptions and a relevant fact appearing well before the final question. Check whether the system uses the correct source and preserves the constraints needed for the task.
Record the input configuration, selected source identifiers and observable actions. These records can explain what the application supplied and did. They do not expose a complete causal account of the model’s internal reasoning. Research on unfaithful chain-of-thought explanations is a useful reminder of that limit.
Compare candidate context strategies on the same tasks. A shorter request may be faster and cheaper, but it is only an improvement if the necessary evidence remains available and the output still meets acceptance criteria.
Give context an owner
Context construction needs versioning and tests just like business logic. Changing retrieval, summaries or tool descriptions can change behaviour even when the model stays the same.
Define who reviews those changes and how failed tasks become regression tests. The engineering discussion of long-running agent harnesses provides one concrete approach to persistent progress and handoff. The right design depends on your workflow, rather than a universal memory recipe.
Written by Raj Dalal, Advisor at Ryzolv.
Back to Insights