Teams pick a model by running an eval and picking the winner. That is a reasonable first step and a poor final one, because the eval only measures the model you already have.

The question that matters is not "which model is best today" but "what does it cost to be wrong".

Price is a moving target

Input and output token prices have fallen repeatedly, but not uniformly, and not for the capabilities that tend to be load-bearing. A product whose unit economics depend on a specific price point has a problem that a spreadsheet does not surface.

Model a rate change of two or three times in either direction before you commit. If the feature stops working at triple the price, that is a real constraint, not a theoretical one.

Deprecation is the bigger risk

Providers retire models. The usual lead time is generous, but structured outputs, tool calling and fine-tuned model identifiers are all tied to specific model versions, and those are the parts that break first.

Keep an abstraction layer between your product and the provider. Not a large one -- just enough that swapping the model is a configuration change rather than a refactor.

Run the eval against candidates, not just the winner

Build the eval early enough that you can run it against every model you are considering, and again when prices or capabilities change. A team that has an eval can switch providers in a week. A team that does not starts from zero.

Also test the failure modes you will actually see in production: malformed output, refused requests, and truncated context. Capability benchmarks rarely cover these, and they are what users hit.