Summary
Picture two models sitting side by side in your evaluation spreadsheet. One tops every public leaderboard. The other barely registers outside its own niche. Run both against your actual customer service documents. The leaderboard winner stumbles on your edge cases while the quiet one holds up case after case. The benchmark told you almost nothing about which one you should trust.
The right large language model (LLM) is the one that performs the approved workflow dependably within its operating constraints. The model with the highest general benchmark score can still perform poorly on your own documents, edge cases, or required output format. The cheapest model can create greater total cost through corrections, retries, and failed workflow actions. The model with the largest context window can accept every document while still missing the detail the final answer needed.
OpenAI’s evaluation framework supports custom tests built on private data that represents real workflow patterns. Anthropic recommends the same discipline: task-specific evaluations that mirror actual work and deliberately include edge cases. Both point at one practical principle. General benchmarks can build a shortlist. Representative workflow testing has to make the production decision.
Start selection with the workflow contract, not the model
“Choose the best model for customer service” is too broad to test against anything. Customer service alone covers case classification, policy retrieval, response drafting, account updates, refunds, and escalations, each needing different capabilities and different controls. A workflow contract fixes that ambiguity before you compare any model. Name the business purpose, the triggering event, expected input types and length, and approved data classifications. Name the required output, acceptance criteria, allowed tools, response time, volume, review requirements, exception path, and the business consequence of getting it wrong too.
The contract does more than organize the testing. It keeps the evaluation from drifting every time one model produces an impressive result outside the original scope. A model can write an excellent customer response while ignoring the required policy. Another can produce flatter prose while identifying the correct policy and escalation path every time. Only the contract says which result deserves the higher score.
Use seven dimensions to decide whether a model fits the work
A complete model review has to cover task performance, context limits, data handling, integration support, output consistency, speed, and cost. These dimensions interact rather than sitting independently. A more capable model may cut corrections while burning more tokens. A faster model may handle routine classification well and struggle with the unusual case. A larger context window may reduce document prep while raising both cost and response time. Judge the combined operating result, not any one dimension in isolation.
Task performance carries the most weight because general intelligence is an incomplete proxy for it. A strong coding model can produce weak brand writing. A model built for long-form analysis can struggle with a strict classification label. Test the exact abilities the workflow needs: extracting the required fields, following business rules, and using the supplied sources. Test producing valid structured output and refusing conclusions the evidence does not support too. Anthropic recommends defining criteria in observable terms. “Produces a high-quality answer” leaves too much room for inconsistent scoring. “Identifies every required policy condition and routes prohibited cases to review” does not. Give abstention its own test. A model that answers every request looks productive. It can create more risk than one that correctly stops and asks for a missing field.
Context limits describe how much information a model can use, not only accept. The published context window is an eligibility check, nothing more. The widely cited “Lost in the Middle” research found that model performance can drop when relevant information sits buried inside a long context. The RULER benchmark found substantial declines as both length and task complexity increased together. Test to reflect the real information pattern the workflow will face: mixed document types, conflicting facts, similar names, and both current and outdated versions. A single clean test document says nothing about a production case built from 30 mixed records.
Data handling can eliminate a model before performance testing even starts. This goes well beyond a provider’s general privacy page. OpenAI excludes business and API data from training by default and documents standard retention alongside eligible zero-retention arrangements. Anthropic excludes commercial data from training without explicit permission, with retention arrangements that vary by feature. Google’s Gemini terms treat paid and unpaid tiers differently. Individual features like web grounding or prompt caching can introduce their own logging and retention conditions on top of the base terms. A model name alone cannot answer the data question. Match the specific product, account type, deployment, region, and enabled feature set to your company requirements. A failure here should remove a candidate regardless of how well it performs everywhere else.
Integration support determines whether a model that performs well in a chat interface can run the production workflow. The requirements often include structured outputs, function calling, file handling, streaming, batch processing, and identity controls. Providers support different combinations of these across their own model lineup. A model producing accurate prose can still generate unreliable tool arguments. Let the workflow’s actual priorities, not a general reputation for quality, decide which capability wins when the two trade off against each other.
Output consistency matters differently depending on the task. Some variation is fine, even desirable, in creative or ideation work. An invoice extraction workflow needs the same field identified the same way every time. A policy assistant needs the same rule applied consistently across similar cases. Compare decisions and required fields in your test, not prose style. Two polished responses can use entirely different wording and still land on the same valid answer. Two responses can also use nearly identical wording and land on different account classifications, which creates a real operating problem.
Speed covers time to first response, total completion time, throughput, and behavior under concurrent load, and different workflows tolerate wildly different amounts of each. A live customer conversation needs an immediate first response. An overnight classification batch can tolerate several minutes without anyone noticing. Set the minimum acceptable quality first and only compare speed among candidates that already clear that bar. A faster response that gets corrected half the time provides no real value at all.
Cost starts with published token rates, and those rates rarely represent what the workflow will cost to run. The number that matters is total operating cost divided by accepted, completed results. That figure folds in retries, human review, corrections, and failed tool actions rather than counting every generated response as a success. A model with a lower per-request price can still lose this comparison outright once its retry rate gets factored in.
Turn seven dimensions into one decision with the Model Fit Score
Score each candidate from one to five on all seven dimensions, weight them, and total them out of 100 points. A workable starting distribution gives task performance 25 points and data handling 15. Integration support and output consistency each get 15, with context performance, speed, and cost at 10 apiece. Shift the weights with the workflow. A real-time support assistant can weight speed more heavily. A financial analysis workflow can weight consistency and data handling more heavily, and a high-volume classification task can weight cost more heavily.
Apply mandatory gates before any score gets calculated, and they exist specifically because a weighted average can hide a disqualifying weakness. A model should leave the candidate list if it fails the approved data arrangement, the required deployment region, or the minimum usable context. It should also leave if it fails the required tool support, the maximum acceptable response time, or a minimum task-performance floor. These are the requirements you cannot trade away for a better number somewhere else on the scorecard.
Let the seven dimensions carry different weight by function
Marketing needs speed and format consistency for channel adaptation. An original article workflow needs argument quality, source accuracy, and controlled claims instead. The same team can reasonably run two different models for two different jobs.
Sales research needs retrieval, source attribution, and account history, while a customer-facing workflow needs tighter control over claims and tone. Either way, the model has to distinguish evidence from recommendation and recognize when the available signal cannot support a confident next step.
Finance needs numeric fidelity and mechanical validation more than fluent prose. A variance analysis workflow specifically needs several plausible explanations without ever presenting an unsupported cause as a confirmed fact. Separate extraction accuracy, calculation accuracy, and explanatory quality in your evaluation. Folding all three into one combined score can hide a real numeric error.
Operations needs consistency and cost control on the routing path, and reasoning and source use on the incident-analysis path. Across both, a model that routes an unknown case to a person is worth more than one that classifies it confidently and wrongly.
Know that different workflows can justify different models
Nothing requires one model to serve every use case in your company. A fast, cheap model can handle routine classification while a stronger reasoning model takes the difficult exceptions. A long-context model can handle document analysis, and a specialized model can handle code or structured extraction. This portfolio approach improves both cost and task fit. It also creates a real operating responsibility: a record of which model supports which workflow, why you chose it, and its fallback if unavailable. A routing layer that directs cases by complexity or risk needs to stay measurable and explainable, not only assumed to work.
Give production models version control and a reassessment trigger
Models change through new releases, revised defaults, and retired endpoints. Google’s own documentation distinguishes stable, preview, and continuously updated aliases precisely because the “latest” reference can shift under a workflow without warning. Pin a controlled model reference wherever the platform allows it in production workflows. The model record should include the provider, the exact model identifier and version, and the prompt version. It should also include the evaluation set it passed and the next scheduled review date.
Trigger reassessment after any material change: a new model or prompt version, a new data source or integration, or a broader user group. Reduced human review and a material incident should trigger it too. A model selected six months ago may still be the right choice today. That conclusion should rest on current evidence, not on the fact that nobody has revisited it since launch.
Run this 30-day process to move from contract to controlled pilot
Week one. Define the workflow contract in full, then build the acceptance rubric and the mandatory gates before anyone starts testing a model.
Week two. Build the representative evaluation set: normal cases, edge cases, long-context cases, sensitive cases, and high-consequence cases. Attach reference answers or scoring rubrics, then select three to five candidates.
Week three. Run the controlled comparison. Tune an appropriate prompt for each model, then run the full test set under equivalent conditions. Repeat the important cases and record quality, consistency, latency, and failures for each.
Week four. Complete the weighted score, estimate cost at real production volume, and confirm data and security requirements. Test the leading candidate inside the complete workflow before recording the final decision, fallback, and reassessment date.
What you tell them at the end
General benchmarks can start the search. Provider claims can suggest where to look. Published context limits can establish basic eligibility, and token prices can support an early forecast. None of that decides production fit. Your company gets more value from a well-tested model that fits its process than a highly ranked model chosen on reputation.
The evaluation itself becomes a reusable capability. Your team learns how to define quality, build a real test set, control versions, and reassess performance on a schedule. That capability outlasts whichever model wins this particular comparison.
