In April, Meta drew criticism for allegedly gaming the benchmark scores of its Llama 4 AI model. Critics accused Meta of manipulating the system by submitting an experimental, unreleased version of the model to inflate its ranking on AI leaderboards.
I believe Llama 4’s benchmarking criticism is mostly unfounded. First, Meta did not disguise the fact that Llama 4 was categorized as an experimental model. In fact, it was only outranked by a Google model that was also experimental.
But here’s what is most important: Any practical users of foundation models will test and verify performance for their own use case, with Llama 4 likely to win on openness, transparency, cost and other factors. Whether Llama 4 wins or loses will be dependent on its practical capabilities in live cases, not on benchmark rankings and ELO scores.
Indeed, the blowback underscores the importance of a key conversation: What really matters when it comes to AI benchmarks, and how can organizations determine which models are best for them?
A simple test
We’re clearly in a new era of AI. Our old AI benchmarks have been overcome by even the most basic language models. To muddle things even more, the benchmark questions and their answers are often well-known on the internet, which means they have been incorporated into the training for new models. ‘Model Card Inflation’ results when the AI knows what’s on the test!
So, what sort of benchmarking should organizations look to when evaluating different models? The best methodology comes from your own business. Here’s a good starting point: Prepare a series of prompts, and what you would consider great responses to them. Then, provide those prompts to a model and consider whether it got the answer right.
It’s that simple. An intern can set up a test framework that you can use on each new model that’s released. Those questions are also great examples of business logic that you want an LLM to know when it is working for you, so keep them handy in all your AI projects. And make it a living document; you always have something new to teach AI.
One situation in which third-party AI benchmarking has value for the enterprise is when you’re asking an AI for original work ‘off the cuff,’ without great examples. In that case, look for benchmarks that keep their questions secret, or ones that generate answers dynamically.
An ELO-style contest – based on a rating system originally developed for chess – can be objectively useful, serving to ‘evolve’ the benchmark through the work of the participants. Just know that some of the things benchmarks test may not be required for your business (chess player AI, for example). Check what’s being measured in the benchmark’s description. I would recommend reviewing Hugging Face’s list of major benchmarks to find one that aligns with your priorities and business requirements.
An intern can set up a test framework that you can use on each new model that’s released.
Any benchmarking system that measures things you don’t care about is probably irrelevant for evaluating models. For example, the popular LMArena gets massive amounts of publicity, but the differences between its models are somewhat esoteric. Compare that to lesser-known but useful marks like Credo AI’s Enterprise Model Trust Score that may be a lot more relevant to you.
In fact, there are critical enterprise issues that currently affect all AI models. These include hallucinations, SQL generation, instruction-following, and trustworthiness. Tracking progress against indicators of those issues is essential for most use cases, table stakes in comparison to many of the more fine-grained attributes that foundation models publish such as ‘P(not-stereotype | not unknown),’ a technical fairness metric from research.
Remember that cost, latency, availability, content filtering, streaming throughput and other ‘mechanical’ aspects of a language model may be more important for your use case than benchmarks on the model card. Often, a simple task that needs to be performed well repeatedly can be achieved with a lesser model fine-tuned in a way that overcomes many of the ‘out of the box’ benchmark results.
The key is to qualify models according to your enterprise requirements. For example, are they supported on your hyperscale cloud stack? Are they fast enough for your use case? Are they cheap enough for your budget? Then take a look at the leaderboards where those models rise to the top. You’ll likely learn something about how good they are, or how good they could be.
Are benchmarking claims trustworthy?
What about first-party benchmarking claims? Are they reliable? Generally speaking, the answer is yes, for the most part. We can trust these benchmarks in the same sense that we can trust the stated fuel usage of an automobile published by carmakers or the Energy Star rating of a refrigerator.
There is enough scrutiny on them for companies to take only limited liberties in their measurement strategies. That said, there are counter-examples such as Volkswagen, which famously engineered its fuel economy tests to deceive the world for years about just how good their diesel engines really were.
Would OpenAI or X stretch the truth a little too far in the process of raising billions? Caveat emptor. The only way to be sure is to test the model on your use case to know how well it will perform for you.
Choosing the right AI model has always been tricky. The sprawling ecosystem of AI benchmarks isn’t delivering clarity for most organizations. These systems and leaderboards vary in usefulness, but none of them promise the perfect prescription on their own.
The best benchmarking begins in your own business. Start by crafting some prompts and ideal responses, then weigh that vs. what you get against various models. Beyond that, prioritize the metrics that matter the most for your use case – such as speed, cost and other factors. Remember, the landscape changes fast so keep checking new options even after you’ve settled on one. Keep the software stack nimble as well and you’ll never make a choice you can’t improve upon later.
