# How Edward chooses models and routes work

# How Edward chooses models and routes work

Edward's answers to the engineering questions about model selection, in his own terms: production practice, not slides.

## Evaluating two models for the same task

Edward does not pick a model; he picks a task contract. Same prompt, same tools, same retrieval, same temperature band, same token budget, run on a frozen golden set that looks like production rather than a blog demo. He scores three things separately:

- **Correctness** against ground truth or a structured schema: valid JSON, correct tool arguments, a citation that actually exists.
- **Behavior**: did it call the right tools in the right order, refuse what it should refuse, stay in policy.
- **Cost and latency** at the p50 and p95 that will actually ship.

He never lets an LLM-as-judge be the only score. Judges are useful for "was this useful, was it on tone"; they are not a substitute for a deterministic check on the fields that can hurt a patient, a claim, or a ledger. If the delta is small, he keeps both models behind a router and lets traffic decide. If one model is cheaper and passes the contract, it wins. Brand name is not a metric.

## Local model versus Bedrock, Azure OpenAI, Vertex, or a lab API

Local when the data cannot leave the box, when predictable unit economics at volume matter, or when the task is narrow enough that a 7B to 32B model with the right tools beats a frontier model that has to be babysat over the wire. Cloud or lab APIs when the current reasoning ceiling is needed (planning, messy tool use, hard synthesis) and the data classification allows it.

In regulated work Edward defaults to the customer's already-approved cloud: Bedrock for AWS shops that want IAM, VPC endpoints, and Guardrails; Azure OpenAI for Microsoft shops with existing Purview and private endpoints; Vertex where GCP is the control plane. Lab APIs (Anthropic, OpenAI, xAI direct) for his own products and for prototypes, then the same model is pulled behind the customer's boundary when the work ships.

He also mixes them: a high-reasoning model as orchestrator, a cheaper cloud or local model for extraction, classification, embedding, and first-pass drafts, and Grok and Codex in the loop while building. The question is never "which vendor"; it is "which hop is allowed to see this payload."

## Setting reasoning effort per task in a live router

The router sees the task type, not a vibe. Edward classifies the request first with a cheap model, a small prompt, and no tools if possible. Extract this field, classify this ticket, rewrite this sentence: those get low reasoning, low max tokens, and a tight latency SLO. Planning, multi-tool workflows, ambiguous policy, or anything that will write to a system of record gets a high-reasoning model and a higher budget, and that budget is capped: "think forever" is how a $0.02 ticket becomes a $4 ticket.

Signals include user-selected mode when it exists, required tools, data class, estimated steps, and past failure on this task type. If the cheap path fails the contract (schema miss, low judge score, user retry), escalate once; do not loop. Chosen effort, tokens, and outcome are logged so the router can be evaluated like everything else. Otherwise everyone quietly turns every call up to max and finds out on the bill.