Tokens are what you pay for — every character you send in, every character the model sends back. In a demo, that's pocket change. At production scale, serving thousands of users a day, sloppy token usage is the difference between a feature that pays for itself and one that quietly bleeds money every month.

This guide walks through the architectural patterns, prompt techniques, and system designs that actually move the needle on token efficiency — without gutting output quality.

1. Why This Actually Matters

Three things are at stake here, and they compound:

• Cost. Providers charge per million tokens, and output tokens are typically priced several times higher than input tokens. Careless generation costs more than careless retrieval.

• Latency. Time-to-first-token and total generation time both scale with token volume. Fewer tokens, faster responses — it's that direct.

• Accuracy. Even with a 128k or 1M token context window, stuffing it full doesn't help. Models get measurably worse at retrieving information buried in the middle of a long context — the so-called "lost in the middle" effect. More context isn't free, even when it fits.

2. Stop Stuffing the Context Window: Smarter RAG

Retrieval-Augmented Generation is how most teams give an LLM knowledge it wasn't trained on. The problem is the naive version of it: grab the top-k vector search results, dump them straight into the prompt, and hope for the best. That's usually where the token waste starts.

A better pipeline compresses before it generates:

Two changes do most of the work:

Chunk by meaning, not by character count. Splitting text every 500 characters guarantees you'll cut sentences in half, and the model burns tokens trying to make sense of a fragment. Chunk along sentence or paragraph boundaries instead.

Re-rank before you generate. Use a cheap embedding model to pull back 20 candidate chunks, then run those through a cross-encoder (Cohere Rerank and BGE-Reranker are both solid choices) to narrow it down to the 3 that actually matter. Only those go to the expensive model.

3. Say Less: Prompt Compression

The way you phrase a prompt has a real cost attached to it.

Cut the pleasantries. The model doesn't care about "please" or "could you kindly."

• Verbose: "Could you please analyze this JSON and tell me what the user's name is? Thank you!" — 17 tokens

• Direct: "Extract user name from JSON." — 6 tokens

Be deliberate with few-shot examples. They improve output quality, but you pay for them on every single call. If you're leaning on a large few-shot prompt to get consistent behavior, that's usually a sign you should fine-tune instead. A fine-tuned GPT-5.4 nano, or an open-weight option like Llama 4 Scout, can match a much larger model's performance on a narrow task with zero examples in the prompt — which adds up fast at volume. (Worth keeping an eye on: OpenAI's Luna model, part of the GPT-5.6 preview, looks like it'll be an even cheaper option once it's generally available — just not something to build on yet.)

4. Controlling the Expensive Half: Output Tokens

Output tokens usually run 3-5x the cost of input tokens, so how much the model says back matters more than how much you send in.

Force structured output. Don't let the model narrate when it should just answer.

Tip: Use JSON mode or function calling with a strict schema, every time you're extracting data rather than having a conversation.

// Instead of:
// "Here is the information you requested. The user's name is John and he is 30 years old." (20 tokens)

// Force this:
{"name": "John", "age": 30} // 9 tokens

Use stop sequences. Cut generation off the moment the task is done. Generating a Python function? Set stop=["\n\ndef", "```"] so the model doesn't keep going and generate test cases nobody asked for.

5. Semantic Caching: Don't Pay for the Same Answer Twice

Traditional caching falls apart with LLMs because people rarely phrase things identically — "How do I reset my password?" and "forgot my password, help" are the same question to a human, but not to a string-match cache.

Semantic caching fixes this by comparing embeddings instead of exact strings:

Wire this up with a vector database like Redis or Pinecone, and repetitive queries — FAQs, onboarding questions, common coding asks — stop touching the LLM at all.

6. Model Routing: Match the Model to the Job

Not every request needs your biggest model. A quick classification step — using something fast and cheap like GPT-5.4 nano or Claude Haiku 4.5 — can sort incoming queries by difficulty before anything hits the expensive model.

• Simple task (formatting, pulling a date out of text)? Route to the cheap model.

• Genuinely hard task (writing a complex SQL query, multi-step reasoning)? That's when you reach for GPT-5.5 or Claude Sonnet 5.

Most production traffic skews simpler than teams expect — this one change alone often cuts costs more than any prompt optimisation does.

Conclusion

None of this is about starving your model of context — it's about being precise instead of lazy with it. Re-rank before you generate. Enforce schemas instead of letting the model ramble. Cache what's already been answered. Route simple work to cheap models. Do all four, and you'll cut infrastructure spend meaningfully while making the product faster for the people actually using it.

As LLM features move from prototype to production, this kind of token discipline is quickly becoming table stakes for backend and full-stack engineers — not a nice-to-have.