Cutting AI Token Costs With Smarter Prompts and Compressed Replies

You lower AI token costs by sending fewer input tokens, asking for fewer output tokens, and not paying full price for tokens you repeat on every call. In practice that means three moves: trim and reorder the prompt so the repeated part can be cached, constrain the format and length of the reply, and send simple tasks to a smaller model. Output tokens are usually priced higher than input tokens, so a shorter reply often saves more per token removed than a shorter prompt does.

The catch is that cheaper per call is not the same as cheaper per result. A prompt compressed until the model misunderstands it, or a reply cut until it breaks your parser, creates retries that cost more than the savings. This article explains how the bill is built, gives you a calculator to estimate your own savings, and then walks through each lever with its trade-offs.

The key factors

• Output price per token. Providers typically charge more for output than input, so reply length has an outsized effect on cost.
• Repeated context. System prompts, instructions, examples, and conversation history are re-sent on each call, so their cost multiplies with volume.
• Hidden tokens. With reasoning-style models, internal thinking is generally billed as output even when you never see it.
• Model choice. Using a large model for a task a small one handles is often the largest single waste.
• Success rate. The real unit is cost per correct result, not cost per call.

How the bill is built

A language model does not read words; it reads tokens, which are chunks of text such as whole short words, parts of longer words, punctuation, or spaces. Most API providers bill for tokens in two separate buckets: input (everything you send) and output (everything the model generates). Each bucket has its own price, usually quoted per million tokens. Always read the current price list from your provider, because prices, discount programs, and model names change often.

Why input grows faster than it looks

In a multi-turn conversation, the model has no memory between calls. Your application re-sends the system prompt and the earlier turns each time. A conversation that reaches turn 10 pays for turns 1 to 9 again as input, so cost per turn rises as the chat gets longer. Retrieval systems add another layer: every retrieved passage you attach is billed as input whether or not the model uses it.

Why output is the expensive side

Generating text is more computationally demanding than reading it, and pricing generally reflects that, with output tokens costing a multiple of input tokens. The exact ratio differs by provider and model, so treat any single ratio as an assumption to check, not a rule.

Hidden and uneven token counts

• Reasoning tokens. Models that think before answering typically bill that thinking as output. A one-line answer can still carry a large invisible cost.
• Language and format. The same meaning can take different token counts depending on language, code versus prose, or heavy formatting. Text in some languages and scripts generally needs more tokens per word than English, so measure with your own data.
• Tokenizers differ. Counts from one model family do not transfer exactly to another. Use the provider's own token counter when precision matters.

An illustrative example

The numbers below are an illustrative example, not real pricing. Assume a support assistant where each call carries 1,500 tokens of system prompt, 2,000 tokens of retrieved context, 100 tokens of user message, and a 400-token reply. Also assume output costs 5 times as much per token as input (an assumption chosen for easy arithmetic).

Before: 3,600 input tokens at 1 unit plus 400 output tokens at 5 units gives 5,600 cost units. Output is only 10 percent of the tokens but about 36 percent of the cost. After: 2,200 input units plus 1,250 output units gives 3,450, roughly 38 percent lower. The result depends entirely on the assumed price ratio and on quality holding steady, so it does not generalize to your workload without measurement.

Try it: estimate your savings

This calculator compares your current monthly token spend with a projected spend after you reduce input and output tokens. Enter your own volumes and your provider's current prices per million tokens. The values already in the fields are editable examples, not real prices or real usage.

If the output share comes out high, start with the reply-side levers. If input dominates, start with the prompt-side levers, especially caching and retrieval limits.

Prompt-side savings

1. Remove duplication before removing meaning

Long system prompts often accumulate rules added after past incidents, and many overlap. Merge duplicates, delete rules the model already follows by default, and replace long background paragraphs with the two sentences that change behavior. The safest first cut is text whose removal changes no output on your test set.

Illustrative example of a tighter instruction:

Before: "You are a helpful and friendly assistant. Please make sure that you always carefully read the customer's question and then provide a complete and thorough answer that is polite and professional in tone."

After: "Answer the customer's question in a polite, professional tone."

The second version says the same thing in a fraction of the tokens. Test it, because wording that looks redundant to you sometimes steers the model.

2. Put stable content first so it can be cached

Many providers offer reduced pricing for a repeated prefix of the prompt, often called prompt caching. The mechanism generally depends on the beginning of the prompt being identical across calls. That leads to a practical rule: put the system prompt, tool definitions, and fixed examples at the start, and put anything that changes (user message, timestamps, retrieved passages) at the end. A timestamp or user name near the top can break the match and silently cost you the discount. Discount size, minimum prefix length, and cache lifetime vary by provider, so check the current documentation.

3. Retrieve less, rank better

In retrieval-based systems, attaching ten passages when two would do is a common source of waste. Reduce the count, shorten passage size, and improve ranking so the relevant text arrives first. Measure answer quality while you do it, since too little context causes wrong answers and retries.

4. Trim or summarize conversation history

Options include keeping only the last few turns, replacing older turns with a short running summary, or storing facts the user stated in a compact notes field. Each has a cost: summaries can drop details, and truncation can lose instructions given early on. Pin anything that must never be forgotten, such as a user's stated constraints.

5. Use fewer, better examples

Few-shot examples teach format and tone but are billed on every call. Try going from five examples to two, or to one well-chosen example, and compare results. For well-defined formats, a clear schema description often replaces several examples.

6. Route by difficulty

Classification, extraction, simple rewriting, and routing decisions are often handled well by smaller, cheaper models. Reserve the larger model for tasks with real reasoning demands. A simple pattern is to try the small model first and escalate only when a check fails, such as invalid output or low confidence. Build an evaluation set first so that "good enough" is measured, not assumed.

7. Batch what is not urgent

Some providers discount asynchronous batch processing for work that can wait, such as nightly summarization or bulk labeling. Availability, discount level, and turnaround time vary, so verify with your provider before planning around it.

Compressing replies without breaking them

Reply compression has two goals that are easy to confuse: removing words the reader does not need, and removing steps the model needs in order to be correct. Do the first freely and the second carefully.

Constrain format, not just length

• Say what the reply should contain. "Return only the category name" or "Return a JSON object with keys status and reason" removes filler better than "be brief".
• Ban the usual padding. Instructions like "do not restate the question" and "no introduction or closing remarks" target the words that carry no information.
• Prefer labels over prose. For classification, a fixed set of labels is shorter and easier to parse than a sentence.
• Ask for changes, not rewrites. When editing a long document or file, request only the changed sections or a diff instead of the full text.
• Answer first, expand on request. A short default reply with an option to ask for detail costs less than a thorough reply most users never read.

Use limits as a safety net, not a style tool

A maximum-output-tokens setting stops runaway replies, which protects your bill. But it cuts text mid-sentence rather than making the model concise. If the reply must be short and complete, instruct the model to be short, then set the cap above the length you expect. Stop sequences are another useful tool: they end generation once a defined marker appears, which helps when you only need the first section of a structured reply.

Be careful with reasoning

For simple lookups, forcing a direct answer is fine. For multi-step math, logic, or planning, asking the model to skip its working can reduce accuracy, and a wrong answer that triggers a retry costs more than the tokens you saved. If you use a model with adjustable reasoning effort or a thinking budget, match it to task difficulty: low for extraction and formatting, higher for problems where mistakes are costly. Test both settings on your own tasks rather than assuming.

Do not compress what a human must read

Telegraphic replies suit machines. If customers read the output, over-compressing can make answers curt or unclear and push people to contact support, which has its own cost. Compress for the consumer of the text.

Which lever to pull first

The table gives a general ordering by effort, typical impact, and main risk. Impact depends on your traffic pattern, so treat the ratings as starting hypotheses to test.

Common mistakes

• Measuring cost per call instead of per success. A cheaper call that fails 20 percent more often can cost more overall once retries and human review are counted.
• Optimizing without a test set. If you cannot compare outputs before and after, you will not notice quality loss until users do.
• Breaking the cache with dynamic text. Placing a date, session ID, or user name at the top of the prompt can defeat prefix reuse.
• Using the output cap as the only brevity control. The reply gets cut off instead of shortened, and downstream parsing may fail.
• Compressing the prompt into shorthand. Abbreviations and dropped context save tokens but can increase ambiguity, which raises error rates.
• Forgetting hidden tokens. Reasoning output and tool-call overhead appear in the bill even when the visible reply is short.
• Assuming token counts transfer between models. Re-measure when you switch providers or model versions.
• Optimizing a small line item. If a feature represents a tiny part of spend, engineering time may cost more than it saves.

Edge cases and what does not apply

• Flat-rate subscriptions. If you use a chat app on a fixed monthly plan, you are not billed per token, but usage limits still tend to be consumed faster by long conversations and long replies. The same habits, such as starting new chats for new topics and asking for concise answers, can stretch those limits.
• Low volume. At a few hundred requests a month, the savings may be too small to justify added complexity. Reliability usually matters more.
• Quality-critical tasks. In areas such as legal, medical, or financial text, a shorter or cheaper answer that is wrong carries costs beyond tokens. Optimize only after you have a way to verify accuracy, and keep qualified human review where decisions affect people.
• Caching limits. Caching helps only if requests share an identical prefix within the provider's retention window. Sporadic, highly varied traffic may see little benefit.
• Changing prices. Pricing, discount programs, and model lineups change. Numbers in this article are illustrative, so check your provider's current pricing page before building a business case.

How to think about it

Use this method to turn the levers above into a decision you can repeat:

1. Measure the baseline. Log input tokens, output tokens, model, and outcome for a representative sample of requests. Work out what share of cost is input versus output.
2. Define "good enough" first. Build a small set of real or realistic test cases with a pass or fail check, so quality is measured before you change anything.
3. Change one lever at a time. Start with the lowest-effort lever that targets your largest cost share, then re-run the test set.
4. Judge by cost per successful task. Include retries and any human correction in the total.
5. Keep what holds, revert what does not. Re-check after provider price changes, model upgrades, or prompt edits, since each can shift the balance.

The pattern behind all of this: pay full price only for tokens that change the answer. Repeated text can be cached or cut, reply padding can be constrained, and easy tasks can go to cheaper models, but reasoning that the task needs is not waste.

Read next