Finding the Real Cost of an AI Workflow

The cost of an AI workflow is not the price of its first model request. A completed task may include several requests, retrieved documents, tool calls, retries and human review. It may also fail after consuming all of them.

Measure the complete path before optimising a single component. A cheaper request can produce a more expensive workflow if it causes more retries or leaves more work for a reviewer.

Start with a task-level measure

Define what counts as a successful task. For document extraction, that might require valid fields, correct source references and acceptance by a reviewer. A response saying “done” does not establish success in the connected business process.

Track the total cost of the tasks that meet that definition, alongside failures and abandoned runs. Include the model, embedding and retrieval services, hosting, storage and any paid external tools. Record human review separately so it is visible rather than hidden inside a token comparison.

Do not combine dissimilar tasks into one average. A short classification and a multi-document review have different workloads. Segment measurements by task, input size and workflow version.

Inspect repeated context

Long instructions, duplicated passages and verbose tool definitions may be transmitted repeatedly. Count the actual input and output tokens reported by the service where available, and record requests that fail before a complete response.

Prompt caching can change the economics, but availability, matching rules, retention and prices are provider-specific. A cache hit should not be assumed merely because two prompts look similar to a person.

Shortening a prompt can remove an important rule. Compare the revised version against the same evaluation tasks before calling the reduction an improvement. More examples and longer generated reasoning can increase cost and latency as well as change quality.

Bound retries and tool loops

Classify errors before retrying. A transient timeout may justify another attempt. A permissions failure, invalid business request or persistent schema mismatch usually needs a different path.

Set task-level budgets for attempts, elapsed time and resource use. Return a meaningful failure or ask for review when the budget is exhausted. Otherwise a small error can trigger an expensive cycle of near-identical requests.

Retries also create operational risk. A network timeout does not prove that an external write failed. Reconcile the external record before sending the action again, and use the target system’s idempotency facilities where available.

Evaluate model routing

A smaller model may be sufficient for a bounded task. A larger model may reduce failures on another. Neither choice should be justified by price or model size alone.

Compare cost per accepted result, latency distributions and failure modes on representative inputs. Keep sensitive-data requirements and deployment constraints in the comparison. A low-cost fallback is unsuitable if it is not approved to process the data.

Batch interfaces can offer different prices or throughput where a provider supports them. Grouping requests in application code does not automatically qualify for a batch discount. Check the actual interface, completion deadline and retry semantics.

Keep caches safe

Response and semantic caches need access and freshness boundaries. A result containing one customer’s private information cannot be safely served to another customer simply because their questions are similar.

Define what invalidates an entry: source changes, revoked access, a new policy or a changed model configuration. Include cache misses and invalidation work in the operating model.

Compare hosting on the same workload

For self-hosted inference, account for hardware, utilisation, operations, updates and capacity needed during peaks. For hosted services, include service limits and the actual request distribution. There is no universal token volume at which private hosting becomes cheaper.

The vLLM performance dashboard illustrates why throughput and latency need a specified model, hardware and workload. An isolated benchmark is not a forecast for your application.

Put these measurements beside the quality evidence in a release review. AI Software Engineering includes the instrumentation and workflow design that make cost understandable at the task level.