A pilot answers a limited question: can this approach perform useful work under the tested conditions? Production asks a wider one: can the organisation rely on the complete workflow, including its failure paths and human responsibilities?
Keep the useful evidence from the pilot. Then identify what it has not demonstrated before expanding access or letting it change real records.
Turn the use case into acceptance criteria
Write down the inputs, expected outputs and boundaries of the task. Include cases where the system should refuse, request clarification or ask for review.
For a document process, a correct-looking summary is not enough. The team may need source references, required fields, consistent calculations and an explicit approval step. Define how each will be checked and who can accept the result.
Use a versioned test set that reflects the actual work. Keep training or prompt-development examples separate from held-out evaluation cases where possible. Add failures found during the pilot rather than reporting only the demonstrations that worked.
Integrate through explicit contracts
Identify the systems of record and the interfaces the application can use. Document request formats, permissions, expected responses, timeouts and error handling.
Start with narrow access. A read-only integration can establish retrieval and data quality before a write path is enabled. Where a write is necessary, test it in an appropriate environment and define how uncertain outcomes are reconciled.
A successful model response does not mean an external change succeeded. Verify the connected system’s result and return the correct task status to the user.
Choose a rollout method for the risk
Shadow mode can compare proposed outputs with an existing process without allowing those proposals to drive the live action. It can be useful, but it still processes data and requires appropriate access, privacy and operational controls.
A limited-user pilot, read-only release or staged workload may be more suitable in another situation. Choose the approach based on the consequences of failure and the evidence still needed.
Set a clear limit for the rollout. Name the person who can stop it, the conditions that trigger a pause and the process for handling affected records. Do not assume that disabling new requests will reverse changes already committed elsewhere.
Prepare the people who use it
Users need to understand what the system does, which outputs need checking and how to report a problem. Reviewers need access to the evidence required for a decision, rather than a generic approval button.
Define the responsibility split between the business owner, engineering, security and support. If an output is wrong, someone must be able to identify the task, assess its impact and decide what to do next.
Avoid creating a parallel process that only the implementation team understands. The new system needs to fit the organisation’s actual operating arrangements.
Observe the work after release
Monitor task completion, validation failures, tool errors, review effort and unresolved exceptions. Interpret them by task type and configuration version rather than relying on one average success rate.
Logs should capture observable events and relevant references. They need access control, redaction where appropriate and defined retention. More logging is not automatically safer if it duplicates sensitive information into an uncontrolled store.
When performance changes, investigate the retrieval, prompt, model, tool and source-data layers separately. A new model is not a universal remedy for an integration failure.
Make expansion conditional on evidence
Agree the evidence needed before adding users, data sources or actions. Review both the intended benefits and the remaining limitations. Use rollback or compensating procedures appropriate to the connected systems.
The NIST AI RMF Core provides a structure for risk decisions across the lifecycle. It is a voluntary framework, not a deployment certificate.
Our How We Work page describes how requirements, scope, build, verification and operational support fit together. The production decision should follow the evidence from those stages, rather than a fixed calendar promise.
Written by Raj Dalal, Advisor at Ryzolv.
Back to Insights