Most production LLM apps in 2026 are running on a model that is one tier too large. The default for "let me try this with an LLM" is whatever the developer used in the chat UI yesterday, which is almost always a frontier-tier model, which is almost always overkill for the task that ended up in production. Cutting one tier off the default for the queries that do not need it is typically a 10–50x cost reduction with almost no accuracy loss.

This article is the heuristic for which tasks genuinely need a frontier model, which do not, and the pattern for routing between them.

The tier ladder, working-engineer version

The big providers each have three tiers worth distinguishing:

  • Small / fast. GPT-4o-mini class, Claude Haiku class, Gemini Flash class. Single-pass tasks, narrow extraction, short reasoning chains, classification, formatting. These models are dramatically faster (often 3–5x lower latency) and dramatically cheaper (often 10–20x lower per-token rate) than mid tier.
  • Mid. GPT-4o class, Claude Sonnet class, Gemini Pro class. The general-purpose workhorse. Handles most production workloads — multi-step reasoning, code generation, longer-context understanding, nuanced writing. Where most teams land for the default model after they have done the work to know better.
  • Frontier / reasoning. Claude Opus class, the o-series reasoning models, Gemini Deep Think. Reserved for hard reasoning, novel synthesis, complex code, and tasks where the cost of being wrong is high enough to justify the price.

The ratio between tiers is roughly 10x per step for input rate, similar on output, and reasoning-tier models add hidden reasoning tokens on top.

Tasks that almost always work on small models

These are the classic "the small model is fine" tasks. Run the eval to confirm on your specific data, but the prior is strong:

  • Classification into a fixed taxonomy. Spam vs not. Intent labels for routing. Sentiment with a small label set. Small models hit 90%+ accuracy on most well-defined classification tasks.
  • Routing decisions. "Does this query need the SQL agent or the docs agent?" Small models are excellent at this and ship a 1-token decision.
  • Structured extraction with a tight schema. Pull dates, names, prices, IDs out of short text. With a Pydantic / JSON schema, small models match larger ones for accuracy on narrow extraction.
  • Reformatting. Markdown to JSON. JSON to YAML. CSV to a typed object. Anything where the semantics are preserved and the work is syntactic.
  • Short-context QA where the answer is directly present in the provided text. Pure retrieval-and-quote tasks live on the small tier.
  • Summarization of short-to-medium documents into a known structure (TL;DR + bullet points). Small models are good at this; the failure mode is usually a too-aggressive compression, fixable with a temperature/format tweak.
  • Embedding-adjacent tasks: query rewriting, hypothetical document generation for RAG (HyDE), query decomposition.

For all of these, the move is: try the small model first, measure accuracy on your real data, ship if it is within your tolerance. Most of the time it is.

Tasks where a smaller model usually does worse

Here the smaller model often regresses enough to matter:

  • Multi-step reasoning across 4+ steps where each step depends on the last. Small models drop steps or shortcut to the answer.
  • Long-context synthesis where the answer requires integrating information from 20+ pages of input. Small models tend to over-anchor on the first or last chunk.
  • Code generation for non-trivial code (anything more than a few-line snippet). Small models miss edge cases and hallucinate API surfaces more often.
  • Open-ended creative writing where voice, taste, and originality matter. The gap is real and human reviewers consistently rate the larger model higher.
  • Adversarial prompts including jailbreak attempts, edge-case classification, and ambiguous inputs. Larger models tend to handle the messy middle better.

You can sometimes claw back accuracy on a small model with better prompting (more examples, more structure) or with a verifier on top (small model proposes, larger model checks, or vice versa). But the default expectation here should be: the bigger model is meaningfully better.

The cascade pattern

The shape that captures most of the cost saving in practice:

small_model_response = small_model(query)
if confident(small_model_response):
    return small_model_response
return frontier_model(query)

The confident() check is the interesting part. Three implementations that work:

  1. Structured self-check. Have the small model return JSON with both an answer and a confidence or well_supported field. If the model says "I'm not confident," escalate.
  2. External verification. Does the JSON validate? Does the code compile? Does the SQL parse? Did the answer match the constraint? If any verifier fails, escalate.
  3. Routing model. A tiny classifier (often the same small model with a different prompt) decides up front: "is this query likely to need the bigger model?" Cheap, fast, and surprisingly accurate.

In production this pattern typically catches 70–90% of traffic on the small tier and leaves only the hard queries for the expensive call. The cost saving is the cascade ratio times the tier price ratio — easily an order of magnitude reduction in total LLM spend without changing what the user experiences.

The trap: prompt overfitting

The one trap to watch for: a prompt that was tuned for the frontier model often regresses on the small model in non-obvious ways. The instructions assume capabilities the small model does not have, the few-shot examples were chosen with frontier-model behavior in mind, the output schema is too elaborate. Switching models without retuning the prompt makes the small model look worse than it really is.

The fix is to do an explicit small-model pass on the prompt: shorter, more concrete, more examples of the specific failure modes you saw, tighter output schema. Tuned well, the gap between tiers on narrow tasks is much smaller than the default comparison suggests.