Industry
High Tech
High Tech Solutions to Drive Business Transformation through Technology
Highlights
- Much of the GenAI bill is determined inside the application. Long prompts, excessive retrieval, unnecessary use of premium models, and repeated agent steps can increase cost without appearing in traditional infrastructure reports.
- AI FinOps connects token-level telemetry to business outcomes through Cost per Resolved Intent. A shared gateway provides the control point needed to apply routing, caching, usage limits, and reporting consistently.
- A measured, phased programme can reduce run costs while improving latency and governance, provided the savings claims are based on the client’s baseline and every optimisation is protected by quality evaluation.
On this page
Hidden design costs
The GenAI bill is created by software behaviour at request time, not simply by infrastructure capacity purchased in advance.
Traditional FinOps practices work well when teams can see and measure the resource being consumed, such as a virtual machine, storage volume, or cluster. Managed AI inference is different because application design determines how many tokens are processed. We have seen costs increase because old instructions remain in a system prompt, retrieval brings back more material than the answer needs, or routine requests are sent to a premium model by default.
Conversational and agentic applications used in customer care, network operations, field service, and content workflows, especially across telecom and media workloads, make consumption harder to predict. Resending the full history causes input volume to accumulate and the conversation grows. An agent can plan, call a tool, and inspect the result. It may then revise its plan and repeat this cycle many times before resolving one business task. Cost analysis must therefore examine the full request lifecycle, including retries, retrieval steps, tool interactions, validation activities, and escalation pathways.
We separate token consumption into four layers during the diagnostic: intent, context, inference, and iteration. At the intent layer, we check whether the request requires a language model. Some repeated or deterministic requests are better handled through a cache or conventional service. Context includes system instructions, examples, tool definitions, retrieved content, and conversation history. Inference covers the model and output settings used for the request. Iteration measures the additional cost created when an agent retries a step, reflects on a result, or repeats a tool call.
During architecture and cost reviews of enterprise GenAI workloads, context frequently accounts for a significant share of token consumption. Request traces help isolate the underlying causes, including repeated instructions, unused retrieved passages, and accumulated conversation history. These observations provide a practical starting point for prioritising optimisation opportunities.
Measure AI value
Cost per token is a vendor metric while Cost per Resolved Intent (CPRI) is an enterprise management metric.
We have seen that a reduction in token price does not necessarily indicate that an AI estate is becoming more efficient. Consumption can increase faster than unit costs decline, while traditional spending metrics provide limited visibility into delivered business value. Cost per call focuses on individual transactions and total monthly spend aggregates all activity together. Neither measure explains whether user requests are being completed successfully. Organisations therefore need a metric that connects technology consumption directly to outcomes.
CPRI is the fully loaded cost of completing one request successfully. It includes inference, retrieval, tool execution, guardrails, retries, and the share of failed interactions that must be attempted again or escalated to a person. CPRI requires an explicit resolution signal such as task completion, verified answer quality, successful deflection, or avoidance of a human handoff. Establishing that signal is essential because an apparently cheaper model can increase total cost if it generates more retries or poor answers.
CPRI shows whether the overall economics are improving, but it doesn't explain what caused the change. That requires a closer look at the request itself. A rising context-to-question ratio may point to prompt inflation, while routing data can reveal whether routine work is unnecessarily reaching premium models. Agent iteration deserves separate attention too, since a small number of looping tasks can quietly distort the total bill. Each request must be tagged with tenant, team, and use case. It should also carry environment, model, prompt version, and policy decision. In practice, organisations often request showback and chargeback reporting before the required tagging standards have been established. Defining a consistent tagging contract early ensures that financial and operational reporting remains reliable as AI adoption expands.
Telecoms are accelerating production-grade AI adoption, where even small routing errors compound fast. The routing, caching, and prompt inefficiencies that go unnoticed in a pilot get multiplied across tens of millions of monthly calls in production.
Control AI spend
A shared control layer transforms cost visibility into enforceable decisions across every AI request.
When applications connect directly to model endpoints, each team tends to implement cost and policy controls differently. A shared AI gateway provides one control point for the request path. The gateway identifies the tenant and applies the relevant policy before selecting a model. It can also check available caches, apply token and retry limits, and record the information needed for cost and quality analysis.
The optimisation sequence should begin with high-confidence, low-risk changes. Prompt and context hygiene remove obsolete instructions, redundant examples, and unnecessary conversation history. Stable context ordering can improve prefix-cache reuse without changing the answer. Semantic caching can serve repeated intents in suitable domains, provided the similarity thresholds, freshness rules, and explicit exclusions protect stateful or regulated answers.
During production reviews, teams discovered that retrieval settings chosen during pilots remain unchanged in production long after usage patterns have evolved. Extracting only the supporting spans reduces context while often improving grounding. Smart model routing can produce significant savings by directing routine requests to smaller validated models while escalating uncertain cases when necessary. Routing decisions should remain within approved quality, latency, and risk thresholds. Otherwise, apparent inference savings can reappear as retries, human support effort, or service degradation.
Agent loops need their own controls. Loops need hard iteration limits, token budgets per task, memorisation of repeated tool calls and early termination when progress stalls. When a task reaches its budget, the service needs an orderly fallback. Depending on the use case, this may mean switching to a smaller validated model, using an approved cached response, queuing the request, or returning a limited but safe result.
A simple hard spending cap can turn an economic problem into an availability incident.
In telecommunications and media programmes, these controls become crucial in high-volume operations such as customer care, network operations, fraud management, content discovery, and personalisation.
SBuilding cost discipline
Credible cost optimisation begins with a measured baseline.
Teams should implement the lowest-risk improvements first and use governance to prevent the same waste from returning.
In CMI engagements where we have established this type of programme, we begin with a defined diagnostic rather than an early savings promise. The work starts by instrumenting the request path and observing representative production traffic. We then calculate CPRI, examine the main efficiency ratios, and show where spend originates by use case. The baseline records not only the opportunity but also the confidence behind it and the constraints that cannot be compromised. Our experience is that the gains last longer when AI cost becomes part of regular engineering work rather than a one-time clean-up.
Teams do not need to wait for the entire programme to be established before acting on clear findings. Low-risk prompt, context, and caching improvements can begin once the evidence is available. In parallel, the organisation can put shared telemetry and financial reporting in place. Changes to prompts or agent behaviour should be tested against representative traffic, and the expected run-cost effect should be reviewed alongside functional and quality results. This makes cost visible during engineering decisions instead of after deployment.
Intelligent optimisation comes next. An evaluation harness enables retrieval tuning and tiered routing to run in shadow mode before influencing production. Automatic rollback should respond to a quality threshold breach.
The operating model determines whether improvements last. Each GenAI service should carry a CPRI objective alongside latency and availability goals. Application teams own their spending profile, platform teams own shared controls, finance the unit-economics model, and architecture teams the standards and review gates. Governance reviews should focus on ratio trends, budget exceptions, major cost drivers, and corrective actions. With this discipline in place, GenAI spending becomes a measurable engineering and business outcome rather than an unpredictable technology expense.