A prototype can show that a model performs a task on selected inputs. A production system has to manage the inputs that do not fit, dependencies that fail and actions whose outcome is uncertain.
The transition is an engineering exercise. Define what success means in the business process, record authoritative state and make failure visible to someone who can act on it.
Distinguish the failure you are handling
An incorrect answer, a malformed tool request and an external service timeout are different failures. Treating all of them as “ask the model again” can hide the cause and repeat an unsafe action.
Classify at least these cases:
- The input is missing information or falls outside the task’s scope.
- The model’s proposal fails schema or business-rule validation.
- A source or tool returns stale, unauthorised or unexpected data.
- An external action was requested but its outcome is not yet known.
- The final result fails the task’s acceptance checks.
Choose a response for each case. Some can be retried. Others need more information, reconciliation or a person’s decision.
Make state survive the conversation
Store the current task, verified inputs, proposed changes, approvals and external results in the application. A conversation transcript is not a reliable substitute for this state.
Each transition should have an owner and a defined condition. For example, “approved” should identify who approved which version of a proposed change. If the proposal changes afterwards, the old approval must not silently authorise the new action.
Keep enough information to resume after a restart without asking the model to infer what probably happened. That includes the identifiers returned by connected systems and the status of unresolved requests.
Reconcile before retrying a write
Suppose a system requests creation of a job record and the connection drops. The record may have been created even though the application did not receive confirmation.
Retrying immediately can create duplicates. Use a stable idempotency key where the target supports one, or query the external system using an agreed reference before deciding whether to repeat the request. If neither is possible, stop and expose the uncertainty for review.
Do not call a retry “self-healing” unless the recovery behaviour is defined and tested. Some actions are irreversible, and some require a compensating action that must itself be authorised.
Set finite operating limits
Bound attempts, elapsed time, resource use and tool access. A task that keeps generating slightly different plans can consume resources without making progress.
Define an exit state for an unfinished task. Show the user what was completed, what remains uncertain and what information or decision is needed next. Avoid a success message based only on the model’s final sentence.
New inputs can expose gaps, but no novelty detector will recognise every unfamiliar situation. Include representative unusual cases in evaluation and review actual failures after launch.
Test the surrounding system
A useful regression suite tests more than answer quality. Simulate tool failures, revoked credentials, malformed results, interrupted runs and repeated events. Verify that the application stops, resumes or escalates as designed.
Protect sensitive data in fixtures and logs. Record enough observable information to investigate a run while applying access and retention limits. Model-generated explanations should not be treated as a complete record of internal reasoning.
Give changes a release path
Model, prompt, retrieval and tool changes can all alter behaviour. Version them, compare them against the same acceptance tasks and keep an operational way to stop or roll back a release where feasible.
Retraining is not the default repair. A quality problem may originate in document parsing, a changed API, stale retrieval or a workflow rule. Diagnose the failing stage before changing the model.
The NIST Generative AI Profile provides a broader risk-management reference. Ryzolv’s AI Software Engineering practice addresses the application and operating work around these failure paths.
Written by Raj Dalal, Advisor at Ryzolv.
Back to Insights