AI cost optimization: how to reduce costs across products and teams

Melinda Gutermuth
Technical Content Writer
at Merge

AI cost optimization is the practice of reducing and managing AI workload costs while maintaining quality, performance, and business outcomes. As organizations continue to scale AI across their customer-facing products and employee tools, it becomes increasingly difficult to keep those costs in check.

Spending is growing faster than most organizations can track it. Unlike traditional software subscriptions, AI costs can fluctuate with request volume, token consumption, model selection, and the number of steps an agent takes to complete a task. Those costs are spread across providers, applications, and employee accounts, making it hard to see where they're accumulating and what value they deliver.

According to the State of FinOps 2026, 98% of respondents now manage AI spending, up from 31% in 2024. Managing AI spending has quickly become a core responsibility for FinOps teams.

Below, we'll examine what drives AI costs, six strategies to optimize them, and how to build a cost management program that works across production applications and employee AI use.

What drives AI costs?

AI spending depends on how workloads are designed, how often they're used, and the infrastructure that supports them. Understanding these cost drivers is the first step toward pinpointing where optimization can have the greatest impact.

‍

Cost driverWhat increases spendingOptimization approach
Model inference and token usageHigh token volume, request volume, expensive models, and long context windowsModel routing, prompt optimization, and context compression
AI agents and multi-step executionMultiple model calls, retrieval steps, tool execution, and retriesExecution limits, workflow optimization, and workload budgets
Employee AI usageAdoption across teams, fragmented accounts, and unmanaged model selectionCentralized attribution, approved models, and spending caps
Infrastructure and operational overheadGPU utilization and maintaining multiple provider integrationsInfrastructure right-sizing and centralized governance

‍

Model inference and token usage

Most hosted language models charge based on the number of input and output tokens processed. Input tokens include instructions, conversation history, and retrieved context, whereas output tokens are the model's generated response.

Model inference costs depend on the model, number of tokens processed per request, and total request volume. Pricing varies significantly, with smaller models typically costing less than frontier models designed for complex reasoning. Longer prompts and larger context windows raise the cost per request, while more requests increase total spending.

AI agents and multi-step execution

Agents introduce another cost variable because a single user request can trigger multiple model calls and tool interactions. A task that takes a user 1,000 tokens to complete with a single prompt could consume 8,000–15,000 tokens or more when an agent iterates, checks its work, and calls external tools.

For example, an AI agent preparing a customer account summary needs to retrieve CRM records, search support tickets, summarize each source, and generate a final report. Each step increases the number of model requests, tokens, and infrastructure costs. If the agent encounters an error or retries a tool call, costs increase further.

Employee AI usage

Employees across product, engineering, sales, and support use AI tools for day-to-day work, whether it's coding assistants, chat interfaces, or document-drafting applications. Usage varies widely depending on the task type and the model. When employees rely on separate provider accounts, subscriptions, and application-specific billing systems, IT can struggle to track departmental spending or identify which models account for it.

Infrastructure and operational overhead

Organizations that run self-hosted or private models also need to account for costs associated with GPU capacity, compute utilization, storage, and the engineering effort required to manage multiple providers. Maintaining enough capacity to meet peak demand can leave expensive infrastructure underutilized during quieter periods.

Even organizations that rely solely on hosted APIs still incur operational overhead from integrating providers, maintaining routing logic, keeping model configurations up to date, and reconciling billing data. While infrastructure optimization focuses on capacity planning and resource efficiency, API-level strategies address a different layer that includes model routing and cost allocation. Both are relevant to overall AI spending, but they require different controls.

Why traditional cost monitoring isn't enough for AI

Cloud infrastructure costs are generally predictable. A provisioned VM costs the same regardless of whether it handles a high or low load, making it possible to forecast expenses based on reserved capacity. Traditional cost management often involves reviewing past spending and infrastructure allocations. AI costs are different because they scale directly with request volume, model selection, and token usage, all of which can change from day to day.

As a result, AI spending is harder to forecast than spending on fixed subscriptions or provisioned infrastructure. Provider dashboards and billing reports help teams understand spending after requests have been processed, but a more effective approach pairs this visibility with proactive controls. Routing policies can direct requests to cost-appropriate models by task type, while spending caps can pause or redirect requests before they exceed a budget.

This is complementary to, but not a replacement for, broader FinOps practices for AI. Cloud cost governance, reserved capacity planning, and software license management address different layers of spending. Request-level AI controls sit alongside those practices and handle what cloud FinOps tooling wasn't designed to manage.

6 AI cost optimization strategies

The most effective optimization strategy depends on what's driving the spending. Some organizations benefit more from changing the model selection, while others might need better cost attribution, tighter budgets, or more efficient agent workflows. The following strategies address different aspects of the problem.

1. Track and attribute AI spending

Establish a baseline by collecting request-level usage data across AI providers and applications. Useful metrics include the model and provider used, input and output tokens, request volume, and estimated cost.

Attribute spending by provider, model, application, customer, team, or employee at the level of granularity you need in order to report costs and assign accountability. A production application might account for 80% of total AI spending, or one customer segment may be driving a disproportionate share of token volume. Use these insights to set your starting point and prioritize the workloads driving most of your overall spending.

2. Enforce budgets and spending limits

Budget alerts and spending limits serve different purposes. An alert notifies administrators when usage nears or surpasses a threshold, but doesn't necessarily stop further spending. An enforced limit blocks additional requests after the allocated budget is used up.

Apply budgets at different levels, depending on how AI is consumed:

  • Customers. Assign each customer a monthly inference budget that's aligned with its subscription plan.
  • Applications and workloads. Assign internal engineering teams separate budgets for development, testing, and production workloads to prevent experimental features or resource-intensive jobs from consuming an excessive share of the budget.
  • Teams. Allocate department-level budgets based on expected usage and business needs.
  • Employees. Set individual spending limits to keep usage of approved AI tools under control.

However, hard limits create an operational trade-off: blocking requests can disrupt critical workflows or degrade the customer experience. In most implementations, the right approach combines soft alerts that signal they're approaching the limit, followed by hard limits at a higher threshold to protect against unexpected spikes.

3. Route requests to cost-effective models

Not every AI task requires the most capable or most expensive model. A product might use a lightweight model to classify incoming support tickets while reserving a more capable model for complex troubleshooting or multi-document reasoning.

Model routing makes dynamic selections based on a workload's requirements. Before changing routing policies, evaluate candidate models against representative samples of production tasks. A model that's 70% cheaper and delivers equivalent results represents a genuine cost reduction. A model that's 70% cheaper but requires multiple retries or additional processing may not.

Routing decisions should account for cost, latency, and task-specific quality requirements. These three factors don't always align, and the right trade-off depends on the workload.

For more on routing approaches, see our guides on LLM routing and multi-model routing.

4. Reduce unnecessary token consumption

Token volume is the main cost lever for inference. Reducing unnecessary input and output tokens can lower the inference costs without changing the underlying model.

Several techniques can help:

  • Prompt optimization. Remove redundant instructions and unnecessary examples while retaining the information needed for accurate responses. Even modest reductions, like trimming 200 tokens from a 1000-token prompt, compound across millions of requests.
  • Relevant context selection. Retrieve only documents or passages that are directly relevant to the current task.
  • Output limits. Cap response length, or use structured outputs when lengthy explanations aren't needed.
  • Context compression. Condense long conversations, retrieved documents, or tool results before sending them to a model.
  • Prompt caching. When supported by the provider, reuse repeated prompt prefixes to avoid paying full processing costs for the same input.

Prompt caching differs from semantic response caching, which stores and reuses previously generated responses for new queries that are semantically similar. Prompt caching still requires model execution, but it can reduce the cost of processing repeated input.

The main risk of these techniques is removing information the model needs. Excessive compression or context filtering can lower response quality and trigger more retries, offsetting the initial savings. Token optimization is most useful for high-volume applications, long-context workloads, and agent workflows that repeatedly process similar information.

For additional inference-level techniques, see our LLM cost optimization guide.

5. Limit unnecessary agent execution

Agent workflows generate costs that are difficult to predict on a per-request basis. Each additional planning step, retrieval call, or retry adds tokens and can introduce latency and tool execution costs.

Set application-level guardrails, such as maximum step counts, retry limits, and tool-call budgets per task, to control execution costs. Establish these limits based on typical task-completion profiles, leaving enough headroom for legitimate complex tasks while preventing runaway loops. For example, a research agent might be allowed to make a defined number of retrieval attempts before returning its findings or escalating an incomplete result.

Application-level execution limits directly address agent behavior. Workload budgets provide an additional safeguard by limiting aggregate spending, even when individual executions stay within their allowed limits. Together, these controls help reduce the risk of unexpectedly expensive agent runs. Execution limits are especially important for autonomous agents, multi-step workflows, and applications that invoke external tools.

6. Measure cost per successful outcome

Lowering the price of an individual AI request isn't necessarily the same as making an AI workload more cost-effective. If a cheaper model requires more attempts to produce a usable result, the total cost of a completed task may be higher than with the more expensive model.

To evaluate the real economics of AI, measure spending against successful outcomes:

Cost per successful outcome = Total workload cost ÷ Number of successful task completions

For example, imagine a support agent costs $500 to operate and successfully resolves 1000 tickets. Its cost per successful resolution is $0.50. Now consider a cheaper model that reduces operating costs to $400 but resolves only 600 tickets. Its cost per resolution is higher at $0.67, despite the lower total spending.

Track cost alongside task success rates, output quality, and latency to provide a complete picture of whether a routing or optimization change actually improved efficiency or simply shifted cost from one place to another.

How to build an AI cost optimization program

Individual optimizations deliver the greatest value when they're part of a repeatable cost-management process. These four steps establish a foundation that can scale as usage and model options evolve:

1. Establish visibility

Inventory all active AI workloads across production applications and employee tools, including the models, providers, and accounts used.

Collect baseline spending and usage data, segmented by customer, application, team, and model wherever possible. Identify which workloads incur the highest costs and where attribution is missing.

2. Assign ownership

Define who is responsible for AI budgets, optimization decisions, policy enforcement, and reporting.

Platform engineering typically owns production AI cost governance, including routing policies, workload budgets, and customer attribution. IT or AI enablement teams generally manage employee AI governance, including approved models, team-level caps, and usage reporting.

Finance and FinOps stakeholders can help establish allocation methods, reporting requirements, and value metrics across both functions.

3. Apply controls

Prioritize improvements based on the largest cost drivers identified in the baseline.

Determine which workloads, customers, or teams should receive routing policies, spending caps, and prompt optimizations first. Workloads that account for a disproportionate share of spending are usually the best place to start. Introduce changes incrementally, with clear thresholds and fallback procedures for critical workloads.

4. Measure and refine

Review costs alongside task success rates, output quality, latency, and business outcomes to see whether optimization changes deliver value.

A routing policy that performs well in benchmarks might yield inconsistent results under real-world production workloads. Use these findings to refine your controls, and reassess them as usage patterns shift or new models become available. The goal isn't simply to reduce AI spending once, but to keep costs aligned with performance and business needs as your AI systems evolve.

How Merge helps control AI costs across products and teams

Managing AI costs requires different approaches for customer-facing applications and for internal employee tools.

Production AI needs customer-level attribution, workload budgets, and routing policies that balance quality, latency, and cost. Employee AI needs model governance, team-level spending caps, and visibility into usage across the people and tools your organization has approved.

Merge Gateway and Merge for Workforce address these needs separately, with controls tailored to each context.

Merge Gateway for production AI costs

Merge Gateway lets you ship production AI faster, with routing, cost controls, and observability built in.

Instead of building and maintaining separate integrations for each AI provider, engineering teams can use Merge Gateway to route model requests and manage spending from a centralized layer.

Merge Gateway supports several cost-management capabilities:

  • Intelligent model routing: Route each request by cost, latency, or quality. Build Your Own Router blends published benchmarks with your own evaluation scores, weighted the way you choose, so model selection reflects task-specific quality requirements.
  • Real-time budgets and spending caps: Cap spend by project, team, or customer tier in real time. Soft limits send alerts, and hard limits block further model requests once a budget is exhausted.These controls also apply when customers use their own provider API keys.
  • Customer-level cost attribution: Gateway's Customer object gives each of your end customers its own routing policies, budget, and usage. Link requests to individual customers to apply tailored controls and attribute costs.
  • Token optimization: Use context compression to shrink long requests with no application change, and prompt caching to reduce the cost of repeated inputs.

Customer-level controls are especially useful for SaaS products that need to track inference costs per account or manage usage across subscription tiers. For example, a customer-facing AI application could route straightforward classification requests to a less expensive model, reserve higher-capacity models for complex tasks, and enforce separate spending limits for each customer.

These capabilities have also delivered measurable results for Merge customers. Windmill, an AI-powered performance management platform, reports saving more than $10,000 per month on LLM spending by using Merge Gateway's LLM routing and spend cap rules, without sacrificing performance.

Start free with Merge Gateway.

Merge for Workforce for employee AI costs

Merge for Workforce gives employees AI access to approved connectors, company context, skills, and models while giving IT visibility and control.

Employees continue to use the AI tools they already know. Merge for Workforce makes those tools more capable and better governed by adding company context, approved connectors, and skills. IT configures approved models, connectors, and spending policies once, then distributes those settings to employee tools. This gives teams the AI capabilities they need without requiring administrators to configure each employee's setup individually.

Merge for Workforce also provides centralized cost governance:

  • Approved models by group: IT can define which models are available to different teams and establish routing policies appropriate to their work.
  • Enforced spending caps: Hard budgets apply at the group level, blocking additional model requests when a group's budget is exhausted.
  • Detailed cost attribution: Usage reporting associates model requests and spending with individual employees, teams, and models. This granularity also supports allocation reporting by linking employee AI spending to the teams and use cases they serve.
  • Centralized administration: IT can manage model access and spending policies across supported employee AI tools rather than maintaining disconnected provider configurations.

This approach allows employees to keep the AI tools that work best for them while giving IT consistent visibility and control over AI costs across the organization.

Book a Merge for Workforce demo.

FAQs

How do you calculate the cost of an AI request?

The cost of an AI request depends on the input and output tokens processed and the model's per-token rates. To calculate it, multiply the input and output token counts by their respective provider rates, then add the results. For example, a request with 2,000 input tokens (everything sent to the model, such as prompts and context) and 500 output tokens (the model's generated response) costs $0.007 if the provider charges $1 per million input tokens and $10 per million output tokens. Requests that use tools may consume additional tokens for tool calls and results. Some providers also vary their rates based on context window length or whether tokens are cached.

How can businesses forecast AI costs when usage is unpredictable?

Start with historical request volumes, token usage, model distribution, and costs per workload. Use this baseline to model low, mid, and high scenarios for increased user adoption, longer conversations, or more complex agent executions. Spending caps limit downside risk in the high scenario, while routing policies can further manage exposure if actual usage exceeds expectations. Attribution by workload and customer also helps identify which usage changes are driving variance, allowing you to refine forecasts as you gain a better understanding of your workload patterns.

Does routing requests to cheaper models reduce output quality?

Not necessarily, but it depends on the task. A less expensive model may match a frontier model on straightforward classification or extraction tasks, but struggle with complex reasoning or nuanced analysis. Evaluate models against representative samples of your actual workload before changing routing policies, rather than relying on generic benchmarks.

How is AI cost optimization different from FinOps for AI?

FinOps for AI applies standard financial operations practices, including visibility, allocation, forecasting, and optimization, to AI workloads. AI cost optimization is a component of that broader practice, focused specifically on reducing and controlling the cost of AI inference and execution. FinOps for AI also includes organizational practices such as defining ownership, building allocation frameworks, and linking spending to business value. Request-level optimization supports that framework by providing controls teams can use to act on spending data.

Can AI budgets still be enforced when customers bring their own provider keys?

Yes, with the right gateway architecture. A bring-your-own-key (BYOK) setup changes how provider usage is billed, but it doesn't inherently stop a gateway from tracking requests or enforcing spending limits. If API calls go through a centralized gateway instead of directly to a provider, it can enforce budgets regardless of which credentials are used. This matters for platforms where customers provide their own API keys for cost pass-through.

Melinda Gutermuth
Technical Content Writer
@Merge

Read more

How Merge's sales team automated its monthly overage review with Claude and Agent Handler

Merge vs Composio: when to choose one over the other

Insights

What is a company brain? How it gives enterprise AI company context

AI

Subscribe to the Merge Blog

Get stories from Merge straight to your inbox

Subscribe

But Merge isn’t just a Unified 
API product. Merge is an integration platform to also manage customer integrations.  gradient text
But Merge isn’t just a Unified 
API product. Merge is an integration platform to also manage customer integrations.  gradient text
But Merge isn’t just a Unified 
API product. Merge is an integration platform to also manage customer integrations.  gradient text
But Merge isn’t just a Unified 
API product. Merge is an integration platform to also manage customer integrations.  gradient text