Most teams shortlist models the way they hire: skim the rankings, argue about the top three, and stop there. On OrcaRouter‘s compare llm models page the twelve rows make a better drill set, because the table forces every shortlisting shortcut out into the open. The candidate pool on read 2026-10-12 spans input prices from $0.60 to $30.00, output prices from $1.98 to $180.00, context windows from 205K to 1M, and quality scores that barely move at all — eleven of the twelve rows sit between 8.0 and 10.0. A procedure that narrows twelve rows to two has to use the columns that actually vary.Here is one that worked, run against the same table you can open yourself.
The first cut is not about quality. It is about what the model can physically accept. Eight of the twelve rows carry a 1M context window; the rest break down as 500K for Grok 4.7, 262K for kimi-k2.6, and 205K for MiniMax M2.7 highspeed, with GPT-5.5 Pro’s context unpublished entirely — not listed, not a known number. If the workload feeds long documents, the short list has to respect the envelope: rows below the document size are out of the running regardless of how they score. That single constraint already removes models whose context cannot cover the input.Context is the honest first cut because it is a hard constraint, not a preference. A quality gap can be worked around — chunking, retrieval, a stronger model on the hard slices. A context gap cannot be worked around when the whole document must pass through at once. Shortlist against the columns you cannot negotiate first, and the pool shrinks on objective grounds rather than on taste.
Read the quality column the right way
The second cut uses the quality column, but with a warning about what it is. Our 0–10 composite quality score is our own judgment across quality dimensions such as instruction adherence, reasoning quality, output coherence and consistency — not a third-party benchmark and not a rank. That matters because the column is nearly flat. Eleven of the twelve rows land between 8.0 and 10.0, which means quality, as measured here, does almost none of the narrowing. The cheapest row scores 8.0; the most expensive rows score 10.0; most of the middle sits at 9.0.A flat quality column is not a failure of the page; it is the finding. When every candidate lands in a two-point band, the decision between candidates has to come from the columns that still vary — price, latency, context, traffic — and the composite score serves only as a floor, not as a tiebreaker. If a candidate drops below the floor, cut it; if it clears the floor, stop arguing about the tenth of a point, because the measurement was never that precise to begin with.
The two finalists, decided by a surprise
After the context cut and the quality floor, the procedure leans on price, and this is where the surprise sits. Two candidates carry our composite quality score of 10.0 — the top of the scale — yet their prices differ by a factor of five on input. Claude Fable 5.1 lists at $10.00 per million input tokens and $50.00 per million output tokens; Gemini 3.1 Pro Preview lists at $2.00 and $12.00. Same composite score of 10.0 on the quality dimensions we score ourselves, and the input price alone is five times apart. On a workload heavy on reading, the same nominal quality costs a fivefold difference per token.That pair — same top score, a five-times price gap — is the entire reason a shortlist cannot stop at the leaderboard. A ranking names a winner and stops; the shortlist has to ask what the winner costs relative to the row that ties it. The procedure lands on both of them as finalists because they tie on the floor and split on price, and the split belongs in front of the human who owns the budget. Shortlisting is supposed to surface exactly this decision, not to make it for you.
Why the procedure beats picking a favorite
A structured shortlist earns its keep in three ways that gut instinct does not. It is reproducible: a colleague can rerun the same cuts and reach the same pair, because every step names its column. It is honest about trade-offs: the context cut and the quality floor admit that some constraints are absolute while others are judgments. And it defuses the loudest voice in the room — the person who always argues for the most expensive row on the grounds that it is the safest. The procedure gives that row a turn and then sends it to the final pair, where price gets to answer back.There is also a latency note to fold in before the pair leaves the page. The candidate at $2.00 input, Gemini 3.1 Pro Preview, sits at the 10.00 s ceiling for time to first token in our production telemetry — the measurement ceiling, not a real measured spread. The premium finalist, Claude Fable 5.1, measures 3.98 s. Two rows, tied on the top quality score, five times apart on input price, and separated on latency by a full ceiling. The final decision is now a genuinely useful question: is the extra speed and the extra polish worth the fivefold input bill on your traffic mix?Note what did not help us get here. The independent coding ranks, for instance, are real numbers — Claude Fable 5.1 leads them at 81.6 out of 138 models — but they did not do the narrowing, because both finalists score above the floor and the ranks distinguish one model from another only when the workload is specifically coding. Our procedure kept the ranks where they belong: as context for the final evaluation, not as a first cut. A shortlist that starts from a rank and works backward gets you the best model at any price; the procedure here is deliberately the reverse, and the reverse is what a budget holder actually needs.
The takeaway
Shortlisting twelve models to two is a procedure, and it works in the order a hard constraint demands: cut on context first, apply a quality floor, then let price do the narrowing. The table on the comparison page is built for exactly this drill — context windows that range from 205K to 1M, and a composite quality column where eleven of twelve rows sit within two points, so the shortlist has to lean on price and latency instead. The result is a pair nobody would have guessed from the rankings alone: Claude Fable 5.1 and Gemini 3.1 Pro Preview, tied at a composite 10.0, five times apart on input price.No shortlist survives contact with a real workload unchanged, so the last step is the one that matters most. Re-read the numbers the week you plan — every latency and volume figure here is a rolling window dated 2026-10-12 — and run both finalists against your own traffic before the decision is final. The procedure narrows; your traces decide.
Leave a Reply