When organisations first experiment with Artificial Intelligence, the focus is usually on capability. Can the model answer questions? Can it summarise documents? Can it write code? Can it automate a business process? As AI moves from proof of concept into production, however, the conversation quickly changes.
The question is no longer "Can we build this?" It's "How much will it cost to run?"
Many organisations are now discovering that Large Language Models (LLMs) introduce a new operational cost that behaves very differently from traditional software licensing. Every prompt, every response and every interaction consumes tokens. Scale that across thousands of employees, millions of customer interactions or enterprise-wide AI agents, and token consumption can rapidly become one of the largest components of an AI operating budget.
The instinctive response is often to look for a cheaper model. In reality, this is usually the wrong place to start. The biggest determinant of AI cost isn't the price of an individual token. It's the architecture of the system using those tokens. The cheapest AI token isn't the one bought at a discount. It's the one you never have to use.
Token costs are only the visible part of the iceberg. Below the surface lie engineering effort, infrastructure, orchestration platforms, governance, monitoring, observability, security, integration, model evaluation and ongoing operational support.
Many organisations successfully build AI proof of concepts over a few weeks using a handful of prompts and APIs. Those demonstrations can be genuinely impressive. Production is a very different proposition.
Enterprise AI requires:
security
operational management
resilience
Suddenly the AI application that appeared inexpensive during a hackathon becomes a business-critical platform supporting thousands of users. If every interaction invokes an expensive frontier model, operational costs grow rapidly. This is why architecture matters.
When organisations discuss reducing AI costs, conversations often begin with questions such as: "Should we switch to a cheaper model?" "Can we negotiate better token pricing?" Those questions have some value. A far better question is: "Do we need an LLM at all?"
One of the most common architectural mistakes is assuming every step in an AI workflow requires AI reasoning. It doesn't. Not Every Step Needs an LLM.
Consider an AI-powered payment repair solution. A failed payment arrives.
The system must:
determine why it failed
identify the appropriate repair
update the payment
validate the result
resubmit the payment
Only one of these activities genuinely requires intelligence: The diagnosis. Everything after that is deterministic. Unfortunately, many AI systems repeatedly invoke the LLM throughout the workflow.
This results in:
Six separate opportunities for latency.
Six token bills.
Instead, good architecture separates reasoning from execution. The LLM determines what should happen. Traditional software performs how it happens.
The resulting workflow is:
Faster
More reliable
Easier to govern
The LLM behaves like a senior consultant making the decision. The workflow engine behaves like the operations team carrying it out.
There is a tendency within AI projects to assume that because AI can perform a task, it should. That isn't always true.
Traditional software remains exceptionally good at:
business rules
calculations
validations
workflow execution
API orchestration
database updates
For example:
if payment_amount > daily_limit:
reject_payment()
There is absolutely no reason to ask an LLM whether £7,500 exceeds a £5,000 limit.
The software already knows.
It performs the calculation instantly
Without latency
Without hallucinations
Without token costs
The role of AI is to solve problems that deterministic software cannot. Everything else should remain deterministic.
Before an LLM can answer a question, it must retrieve information. Many organisations underestimate how much this affects cost. Suppose somebody asks: "What are our sanctions screening procedures?"
If the system retrieves twenty Confluence pages and sends all of them to the model, the LLM must:
read everything
identify relevant sections
discard irrelevant content
generate an answer
Thousands of unnecessary tokens have already been consumed before reasoning even begins. A better architecture retrieves only the information the model actually needs. This is where choosing the correct technology becomes important.
Different technologies solve different problems. Trying to use one technology for every workload almost always increases cost.
Consider a few examples.
Operational Data Layer (ODL)
An Operations analyst asks: "Where is payment ABC123?" A historical data warehouse may need to search billions of records. An Operational Data Layer designed for real-time operational queries returns the answer almost instantly.
The LLM receives only the small amount of information required to explain the payment status, resulting in:
• Less data.
• Fewer tokens.
• Lower latency.
Vector Databases
Suppose someone asks: "How do I configure Confirmation of Payee?" Searching thousands of documents using SQL is inefficient. A vector database retrieves only the semantically relevant passages. Instead of sending 80 pages to the model, perhaps only three paragraphs are required.
Again:
• lower token consumption
• faster response
• improved accuracy
Graph Databases
Relationship problems are different. Imagine detecting mule accounts. The question isn't: "Find me documents about fraud." The question is: "Which customers share the same phone number, address, device, beneficiary account and employer?" A graph database traverses those relationships directly. The LLM receives only the relevant network rather than millions of disconnected records. The AI reasons over a carefully selected graph instead of searching the entire enterprise.
Poor data quality has a direct financial cost:
Incomplete information leads to repeated prompts.
Conflicting information leads to clarification questions.
Duplicate information increases context length.
Every one of these problems consumes additional tokens. Investing in data quality is therefore one of the most effective ways of reducing AI costs. The better the data, the less work the model performs.
Graph AI is often discussed in terms of fraud detection or recommendation systems. Its cost benefits receive much less attention. Knowledge graphs allow AI systems to retrieve information based on relationships rather than simply keywords.
Instead of asking: "Find everything about customer John Smith."
The graph can identify:
linked accounts
connected businesses
shared devices
payment networks
common beneficiaries
The search space becomes dramatically smaller. The prompt becomes dramatically shorter. The reasoning becomes dramatically simpler.
Graph AI therefore improves:
precision
latency
token consumption
explainability
It is a perfect example of architecture reducing AI cost.
Many organisations automatically select the largest available model. This is rarely the most cost-effective approach. Different tasks require different levels of intelligence.
A frontier model may be appropriate for:
complex reasoning
strategic analysis
architecture design
legal interpretation
A much smaller model may be perfectly capable of:
summarisation
document classification
translation
Good architectures use model routing. Simple requests are directed to inexpensive models. Complex requests are escalated only when necessary. Just as an organisation wouldn't ask the CEO to approve every expense claim, an AI architecture shouldn't ask its most expensive model to perform every task.
Another overlooked cost is duplication. Many organisations build the same AI capabilities repeatedly.
Different teams create:
prompt libraries
RAG pipelines
evaluation frameworks
AI agents
orchestration workflows
This duplication increases engineering effort and operational complexity. Reusable architecture avoids this.
Examples include:
AI reference architectures
reusable AI agents
shared orchestration services
MCP servers
prompt templates
evaluation frameworks
Instead of every project starting from scratch, proven components become organisational assets. The result is lower implementation cost and lower operational cost.
Several architectural patterns consistently reduce both token consumption and operating costs.
Model Routing
Choose the smallest appropriate model.
Reserve expensive reasoning for genuinely complex tasks.
Semantic Caching
Many users ask similar questions.
Rather than generating the same answer repeatedly, cache responses and reuse them where appropriate.
Context Filtering
Only send information relevant to the current task.
Don't ask the model to read entire document collections.
Human-in-the-Loop
Not every decision requires additional AI reasoning.
Sometimes a human review is both cheaper and safer than repeated autonomous retries.
Workflow Orchestration
Use AI for decision-making.
Use software for execution.
This remains one of the most effective techniques for reducing operational costs.
There is a growing tendency to think of AI as replacing software. The opposite is usually true.
The most successful enterprise AI systems combine multiple technologies:
Relational databases
Operational Data Layers
Knowledge graphs
Vector databases
Workflow engines
Rules engines
Caching layers
Traditional software
Large Language Models
Each performs the task it is best suited to. The LLM becomes one component within a much larger enterprise architecture rather than the centre of it.
Perhaps the most important question organisations should ask is “Who benefits from higher token consumption?.” Many AI vendors generate revenue every time a model is called. Their commercial incentive is naturally aligned with increasing AI usage.
An independent consultancy approaches the problem differently. The objective is not to maximise token consumption. It is to maximise business value:
Sometimes that means using a frontier model
Sometimes it means using an open-source model
Sometimes it means not using AI at all.
Good architecture is technology-agnostic. It selects the right technology for each task, balancing capability, cost, performance, governance and long-term flexibility.
Artificial Intelligence will transform how organisations operate, yet the organisations that achieve the greatest return on investment won't necessarily be those with the biggest models or the largest AI budgets.
They will be those that design intelligent architectures:
Architectures that retrieve the right information
Architectures that separate reasoning from execution
Architectures that combine AI with deterministic software
Architectures that treat data as a strategic asset
Architectures that reserve expensive reasoning for the problems that genuinely require intelligence
Ultimately, the cheapest AI token isn't the one purchased at the lowest price. It's the one your architecture never needs to spend.