Build a Cost-Aware Model Router in Python: A Hands-On Tutorial

Stop Paying Frontier Prices for Simple Tasks

Most production AI systems send every request to the same model, and most of those requests do not need it. A grammar fix, a classification, a short extraction — these are cheap tasks dressed in expensive clothes. The fix is a model router: a small layer that inspects each request and sends it to the least expensive model that can handle it. In this tutorial we will build one in Python, with a working classifier, a fallback path, and a way to measure whether it is actually saving money.

The idea is not new, but it has moved from research curiosity to standard practice. Cursor's router reports meaningful savings against running a single frontier model for everything, and the pattern generalizes to any provider. The key insight is that routing is an evaluation problem, not a model problem. You are not trying to be clever. You are trying to spend the minimum that clears your quality bar.

Step 1: Define Your Model Tiers

Start by listing the models you can actually call, ordered by cost. A typical three-tier setup looks like this:

• Tier 1 — cheap and fast — a small model for classification, extraction, formatting, and short rewrites.
• Tier 2 — mid-range — a balanced model for summaries, drafting, and moderate reasoning.
• Tier 3 — frontier — the expensive model for hard reasoning, long-context analysis, and anything high-stakes.

Keep the list short. Two or three tiers is enough. More tiers mean more decisions to validate, and the marginal saving rarely justifies the added complexity.

Step 2: Write the Classifier

The classifier is the heart of the router. It should be cheap — ideally a tier-1 call itself, or even a set of heuristics — and it should output a tier, not a score. Here is a minimal version that combines simple rules with a cheap model call:

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])

MODELS = {
    'cheap': 'gpt-5.6-luna',
    'mid': 'gpt-5.6-terra',
    'frontier': 'gpt-5.6-sol',
}

def classify(prompt: str) -> str:
    words = len(prompt.split())
    if words < 40 and any(k in prompt.lower() for k in ('classify', 'extract', 'format', 'rewrite')):
        return 'cheap'
    verdict = client.chat.completions.create(
        model=MODELS['cheap'],
        messages=[{'role': 'user', 'content':
            'Reply with exactly one word: cheap, mid, or frontier. '
            'Rate the reasoning difficulty of this request:\n' + prompt}],
        max_tokens=4,
    ).choices[0].message.content.strip().lower()
    return verdict if verdict in MODELS else 'mid'

def route(prompt: str) -> str:
    return MODELS[classify(prompt)]

Two design choices matter here. First, the rules short-circuit the obvious cases so the cheap model is not even called. Second, the classifier returns a known tier or falls back to mid, which prevents a malformed response from sending a trivial task to the frontier model.

Step 3: Call the Chosen Model

With the tier decided, the execution step is boring on purpose. You call the selected model and return the result. The only subtlety is that you should log which tier handled each request, because without that log you cannot compute savings or catch misrouting:

def answer(prompt: str) -> str:
    model = route(prompt)
    resp = client.chat.completions.create(
        model=model,
        messages=[{'role': 'user', 'content': prompt}],
    )
    print('routed_to=' + model)
    return resp.choices[0].message.content

That print statement is the cheapest observability you will ever add. In production it becomes a structured log line with the token counts, so you can compute cost per request by tier.

Step 4: Add a Quality Fallback

A router that silently sends a hard task to a weak model is worse than no router at all. Add an escalation path: if the cheap model's answer fails a quick check — a schema validation, a confidence flag, or a length heuristic — retry the request on the next tier up.

This keeps the common case cheap and the failure case correct. The important discipline is that the fallback must be triggered by an objective signal, not by the model's own opinion of its work. Models are famously bad at knowing when they are wrong.

Step 5: Measure Savings Honestly

The final step is the one most tutorials skip. Run a representative sample of real requests through both your router and the frontier model, then compare three numbers: cost, latency, and quality. Quality can be measured with a small rubric, a human spot-check, or a task-specific validator. If the router retains ninety-eight percent of the quality at a fraction of the cost, ship it. If quality drops, your classifier is too aggressive.

Track cost per accepted task rather than cost per call. A cheap model that fails and forces a retry can end up costing more than the frontier call you were trying to avoid. The metric that matters is the one a finance team would recognize.

One practical refinement is to cache the classifier result for repeated prompts. Many applications see the same question shape dozens of times a day, and re-classifying identical inputs wastes the very tokens you are trying to save. A small in-memory dictionary keyed on a normalized prompt string is usually enough.

Finally, guard the frontier tier. A router is only as good as its worst classifier output, and a single ambiguous request can silently consume a large share of your budget. Add a hard cap on frontier calls per hour and alert when it is hit. That way a misroute is a small incident rather than a monthly surprise.

Routing is not about using weaker models. It is about refusing to use stronger models where they change nothing.

Where to Take It Next

Once the basic router works, two upgrades pay off. The first is a learned classifier: collect the requests you routed and the outcomes you observed, then train or prompt a classifier on your own traffic instead of generic heuristics. The second is per-tenant budgets, so one team cannot quietly burn the frontier quota for everyone.

Neither upgrade is required to get value. A rule-based classifier and a three-tier map will capture most of the available savings in an afternoon. Start there, measure the result, and only add sophistication when the numbers demand it.

The code in this tutorial is deliberately minimal, but it captures the whole pattern: classify cheaply, route explicitly, fall back on objective failure, and measure cost per accepted task. Everything else is tuning.