The cheapest AI token is the one you never use

How good architecture keeps AI costs under control without compromising capability

When organisations first experiment with Artificial Intelligence, the focus is usually on capability. Can the model answer questions? Can it summarise documents? Can it write code? Can it automate a business process? As AI moves from proof of concept into production, however, the conversation quickly changes.

The question is no longer "Can we build this?" It's "How much will it cost to run?"

Many organisations are now discovering that Large Language Models (LLMs) introduce a new operational cost that behaves very differently from traditional software licensing. Every prompt, every response and every interaction consumes tokens. Scale that across thousands of employees, millions of customer interactions or enterprise-wide AI agents, and token consumption can rapidly become one of the largest components of an AI operating budget.

The instinctive response is often to look for a cheaper model. In reality, this is usually the wrong place to start. The biggest determinant of AI cost isn't the price of an individual token. It's the architecture of the system using those tokens. The cheapest AI token isn't the one bought at a discount. It's the one you never have to use.

AI costs more than you think

Token costs are only the visible part of the iceberg. Below the surface lie engineering effort, infrastructure, orchestration platforms, governance, monitoring, observability, security, integration, model evaluation and ongoing operational support.
Many organisations successfully build AI proof of concepts over a few weeks using a handful of prompts and APIs. Those demonstrations can be genuinely impressive. Production is a very different proposition.

Enterprise AI requires:

• security

• operational management

• resilience

• integration with enterprise systems 

Suddenly the AI application that appeared inexpensive during a hackathon becomes a business-critical platform supporting thousands of users. If every interaction invokes an expensive frontier model, operational costs grow rapidly. This is why architecture matters.

Architecture is more important than model selection

When organisations discuss reducing AI costs, conversations often begin with questions such as:  "Should we switch to a cheaper model?"  "Can we negotiate better token pricing?" Those questions have some value. A far better question is: "Do we need an LLM at all?"

One of the most common architectural mistakes is assuming every step in an AI workflow requires AI reasoning. It doesn't. Not Every Step Needs an LLM.

Consider an AI-powered payment repair solution. A failed payment arrives.

The system must:

• determine why it failed

• identify the appropriate repair

• update the payment

• validate the result

• resubmit the payment

• notify Operations 

Only one of these activities genuinely requires intelligence: The diagnosis. Everything after that is deterministic. Unfortunately, many AI systems repeatedly invoke the LLM throughout the workflow.

This results in:

• Six separate opportunities for latency.

• Six token bills.

Instead, good architecture separates reasoning from execution. The LLM determines what should happen. Traditional software performs how it happens.



The resulting workflow is:

• Faster

• More reliable

• Easier to govern

• Dramatically cheaper

The LLM behaves like a senior consultant making the decision. The workflow engine behaves like the operations team carrying it out.

Deterministic software is still brilliant

There is a tendency within AI projects to assume that because AI can perform a task, it should. That isn't always true.

Traditional software remains exceptionally good at:

• business rules

• calculations

• validations

• workflow execution

• API orchestration

• database updates

For example:
if payment_amount > daily_limit:
    reject_payment()

There is absolutely no reason to ask an LLM whether £7,500 exceeds a £5,000 limit.

The software already knows.

• It performs the calculation instantly

• Without latency

• Without hallucinations

• Without token costs

The role of AI is to solve problems that deterministic software cannot. Everything else should remain deterministic.

Good retrieval costs less

Before an LLM can answer a question, it must retrieve information. Many organisations underestimate how much this affects cost. Suppose somebody asks: "What are our sanctions screening procedures?"

If the system retrieves twenty Confluence pages and sends all of them to the model, the LLM must:

• read everything

• identify relevant sections

• discard irrelevant content

• generate an answer

Thousands of unnecessary tokens have already been consumed before reasoning even begins. A better architecture retrieves only the information the model actually needs. This is where choosing the correct technology becomes important.

The right data platform matters

Different technologies solve different problems. Trying to use one technology for every workload almost always increases cost.
Consider a few examples.

1. Operational Data Layer (ODL)

   An Operations analyst asks: "Where is payment ABC123?" A historical data warehouse may need to search billions of records. An Operational Data Layer designed for real-time operational queries returns the answer almost instantly.

   The LLM receives only the small amount of information required to explain the payment status, resulting in:

   •    Less data.

   •    Fewer tokens.

   •    Lower latency.

2. Vector Databases

   Suppose someone asks: "How do I configure Confirmation of Payee?" Searching thousands of documents using SQL is inefficient. A vector database retrieves only the semantically relevant passages. Instead of sending 80 pages to the model, perhaps only three paragraphs are required.

   Again:

   •    lower token consumption 

   •    faster response 

   •    improved accuracy 

3. Graph Databases

   Relationship problems are different. Imagine detecting mule accounts. The question isn't: "Find me documents about fraud." The question is: "Which customers share the same phone number, address, device, beneficiary account and employer?" A graph database traverses those relationships directly. The LLM receives only the relevant network rather than millions of disconnected records. The AI reasons over a carefully selected graph instead of searching the entire enterprise.

    

Better data means cheaper AI

Poor data quality has a direct financial cost:

• Incomplete information leads to repeated prompts.

• Conflicting information leads to clarification questions.

• Duplicate information increases context length.

Every one of these problems consumes additional tokens. Investing in data quality is therefore one of the most effective ways of reducing AI costs. The better the data, the less work the model performs.

Graph AI can reduce token consumption

Graph AI is often discussed in terms of fraud detection or recommendation systems. Its cost benefits receive much less attention.  Knowledge graphs allow AI systems to retrieve information based on relationships rather than simply keywords.

Instead of asking:  "Find everything about customer John Smith."

The graph can identify:

• linked accounts

• connected businesses

• shared devices

• payment networks

• common beneficiaries

The search space becomes dramatically smaller. The prompt becomes dramatically shorter. The reasoning becomes dramatically simpler.

Graph AI therefore improves:

• precision

• latency

• token consumption

• explainability 

It is a perfect example of architecture reducing AI cost.

Bigger models are not always better

Many organisations automatically select the largest available model. This is rarely the most cost-effective approach. Different tasks require different levels of intelligence.

A frontier model may be appropriate for:

• complex reasoning

• strategic analysis

• architecture design

• legal interpretation 

A much smaller model may be perfectly capable of:

• summarisation

• document classification

• translation

• extracting structured data 

Good architectures use model routing. Simple requests are directed to inexpensive models. Complex requests are escalated only when necessary. Just as an organisation wouldn't ask the CEO to approve every expense claim, an AI architecture shouldn't ask its most expensive model to perform every task.

Reuse is the hidden cost optimiser

Another overlooked cost is duplication. Many organisations build the same AI capabilities repeatedly.

Different teams create:

• prompt libraries

• RAG pipelines

• evaluation frameworks

• AI agents

• orchestration workflows

This duplication increases engineering effort and operational complexity. Reusable architecture avoids this.

Examples include:

• AI reference architectures

• reusable AI agents

• shared orchestration services

• MCP servers

• prompt templates

• evaluation frameworks

Instead of every project starting from scratch, proven components become organisational assets. The result is lower implementation cost and lower operational cost.

Patterns that reduce AI costs

Several architectural patterns consistently reduce both token consumption and operating costs.

Model Routing

• Choose the smallest appropriate model.

• Reserve expensive reasoning for genuinely complex tasks.

Semantic Caching

• Many users ask similar questions.

• Rather than generating the same answer repeatedly, cache responses and reuse them where appropriate.

Context Filtering

• Only send information relevant to the current task.

• Don't ask the model to read entire document collections.

Human-in-the-Loop

• Not every decision requires additional AI reasoning.

• Sometimes a human review is both cheaper and safer than repeated autonomous retries.

Workflow Orchestration

• Use AI for decision-making.

• Use software for execution.

• This remains one of the most effective techniques for reducing operational costs.

Architecture before AI

There is a growing tendency to think of AI as replacing software. The opposite is usually true.

The most successful enterprise AI systems combine multiple technologies:

• Relational databases

• Operational Data Layers

• Knowledge graphs

• Vector databases

• Workflow engines

• Rules engines

• Caching layers

• Traditional software

• Large Language Models

Each performs the task it is best suited to. The LLM becomes one component within a much larger enterprise architecture rather than the centre of it.

The independent advantage

Perhaps the most important question organisations should ask is “Who benefits from higher token consumption?.”  Many AI vendors generate revenue every time a model is called.  Their commercial incentive is naturally aligned with increasing AI usage.

An independent consultancy approaches the problem differently. The objective is not to maximise token consumption. It is to maximise business value:

• Sometimes that means using a frontier model

• Sometimes it means using an open-source model

• Sometimes it means not using AI at all.

Good architecture is technology-agnostic. It selects the right technology for each task, balancing capability, cost, performance, governance and long-term flexibility.

Final thoughts

Artificial Intelligence will transform how organisations operate, yet the organisations that achieve the greatest return on investment won't necessarily be those with the biggest models or the largest AI budgets.

They will be those that design intelligent architectures:

• Architectures that retrieve the right information

• Architectures that separate reasoning from execution

• Architectures that combine AI with deterministic software

• Architectures that treat data as a strategic asset

• Architectures that reserve expensive reasoning for the problems that genuinely require intelligence

Ultimately, the cheapest AI token isn't the one purchased at the lowest price. It's the one your architecture never needs to spend.