Artificial Analysis launches Optima to benchmark AI models on your own data

Artificial Analysis, an AI benchmarking organization known for independent LLM evaluations and benchmarks like GDPval-AA and AA-Briefcase, has launched a new platform called Optima. The premise: public benchmarks compare models on predefined tasks and criteria, but that doesn't necessarily show which model works best for a particular use case. Optima is meant to close that gap by letting users build benchmarks tailored to their own workflows, using their own data. It is available now.

There are three ways to supply source material. Users can upload existing evaluation datasets, either from their own files or from Hugging Face, or import AI agent traces from platforms such as Arize, Braintrust, or Langfuse. Developers can also install a skill that gathers information from their coding environment and past sessions. Users without that kind of data can instead describe their intended use case and provide sample inputs and outputs; Optima then generates suggested test inputs, evaluation criteria, and example tasks, which users can review and refine through feedback before running the actual benchmark.

Optima offers two scoring approaches. The first is rubric-based evaluation against objective criteria. The second is a pairwise comparison method, the same one Artificial Analysis uses for GDPval-AA and AA-Briefcase: users evaluate a sample of response pairs and mark which answer they prefer, and Optima derives a full ranking across the test dataset from those preferences.

Beyond raw model quality, Optima tracks cost per task and time per task as standalone comparison dimensions, so a team can check whether a quality gain is actually worth a model's higher cost or longer processing time. For agentic applications, raw token price alone says little: a cheaper model can end up costing more overall if it needs more attempts, fails more often, or requires extra cleanup work, so cost per completed task is often the more meaningful number. Early testers illustrate the point: one team built a benchmark for finance and accounting agents and found a model that could cut costs by a factor of ten without major quality loss; others tested which model best matched the writing style of lawyers, or most accurately identified elements in a proprietary image dataset.

Optima charges only the actual token costs of the models used, with no markup, according to Artificial Analysis. Rubric-based evaluations cost $0.125 per criterion per model, and pairwise evaluations cost $0.375 per comparison. At the start of benchmark creation, at each benchmark run, and at each evaluation round, the platform holds a balance against a cost estimate; billing is then based on the actual usage and evaluation costs incurred.

Optima is aimed at a well-documented problem with AI benchmarks. An analysis by Epoch AI found that benchmark results depend on implementation details that are rarely disclosed: different prompt wording and temperature settings caused the same model to score noticeably differently depending on configuration, and for agentic benchmarks like SWE-bench, swapping only the scaffold, meaning the agent's control software and tool environment, accounted for up to 15 percentage points of difference. A broader study examining 445 benchmark papers from leading AI conferences found more systemic issues: nearly all had methodological weaknesses in at least one area, including unclear definitions, unrepresentative samples, and missing statistical validation, and only about 10 percent of the benchmarks studied used complete real-world tasks that reflected actual application scenarios. Key concepts like reasoning or alignment were often poorly defined, limiting how much weight their conclusions can bear.

Optima can address the problem of a general benchmark failing to capture a specific use case, but the deeper methodological challenges of benchmarking don't go away. Even a benchmark tailored to a user's own tasks is only as useful as how precisely the target capabilities are defined, how representative the test cases are, and how carefully the evaluation is implemented and documented. Cost and time per task also don't show what an output is actually worth to a business: a cheap, fast AI workflow can still be inefficient if its results need heavy rework or add little value to the process they're part of.

Key facts

Artificial Analysis, known for benchmarks like GDPval-AA and AA-Briefcase, has launched Optima, a platform for building custom AI benchmarks from your own data, workflows, or a description of your use case; it is available now.
Optima compares models on cost per task and time per task as well as quality; early testers used it to find a model that cut costs by a factor of ten on finance and accounting agent tasks without major quality loss.
Source material can come from existing evaluation datasets, Hugging Face, AI agent traces from Arize, Braintrust, or Langfuse, or a skill that reads a developer's coding environment and past sessions.
Scoring uses either rubric-based evaluation, priced at $0.125 per criterion per model, or pairwise comparison, priced at $0.375 per comparison, both billed at actual token cost with no markup.
An Epoch AI analysis found that swapping only the agent scaffold caused up to 15 percentage points of difference on benchmarks like SWE-bench, and a broader study of 445 benchmark papers found only about 10 percent used complete real-world tasks.

Why it matters

General-purpose benchmarks compare AI models on predefined tasks and criteria, but that doesn't reveal which model actually works best for a specific job. Optima is built to close that gap, letting a team benchmark models against its own data, workflows, or a description of its use case rather than relying on a generic leaderboard. The pitch lands against real trouble with existing benchmarks: an Epoch AI analysis found that undisclosed factors like prompt wording and temperature settings can make the same model score differently, and on agentic benchmarks like SWE-bench, swapping only the scaffold, meaning the agent's control software and tool environment, accounted for up to 15 percentage points of difference. A broader study of 445 conference benchmark papers found nearly all had at least one methodological weakness, and only about 10 percent tested complete, realistic tasks. A tool that lets anyone build a benchmark around their own problem responds to that credibility gap directly, though it doesn't resolve the deeper issues underneath it.

Who it affects

Teams choosing which AI model to use for a specific job, rather than researchers comparing models in the abstract. Artificial Analysis's own examples are practical: a finance and accounting team benchmarking agents to find one that cuts costs by a factor of ten without major quality loss, a team checking which model best matches how its lawyers write, and a team testing which model most accurately reads a proprietary image dataset. Developers get a dedicated path in: a skill that pulls benchmark material straight from their coding environment and past sessions. Existing Artificial Analysis benchmark users are also relevant, since Optima reuses the same pairwise comparison method behind GDPval-AA and AA-Briefcase.

How to use it

Optima is available now. There are three ways to supply source material: upload an existing evaluation dataset from your own files or from Hugging Face; import AI agent traces from platforms like Arize, Braintrust, or Langfuse; or, for developers, install a skill that gathers information from your coding environment and past sessions. Without ready-made data, you can instead describe the use case and give sample inputs and outputs; Optima generates suggested test inputs, evaluation criteria, and example tasks, which you review and refine before running the benchmark across models. Scoring is rubric-based against objective criteria, or pairwise, where you judge a sample of response pairs and Optima derives a full ranking from your preferences. Pricing has no markup on model costs: $0.125 per criterion per model for rubric evaluations, $0.375 per comparison for pairwise ones. The platform holds a balance against a cost estimate at the start of benchmark creation, at each run, and at each evaluation round, then settles based on actual usage.

How solid is it

Every figure here, including the $0.125 and $0.375 evaluation prices and the factor-of-ten cost cut found by early testers, comes from Artificial Analysis describing its own new product. None of it is independently verified in the source, and the specific models Optima compares and the early testers behind the finance, legal, and image-dataset examples are not named. Artificial Analysis does have a track record in this area: it already runs GDPval-AA and AA-Briefcase, and it grounds the case for Optima in outside research, an Epoch AI analysis plus a broader study of 445 benchmark papers. That broader study, unlike the Epoch AI work, is not attributed to a named author or institution in the source, and neither piece of outside research is evidence that Optima's own user-built benchmarks will be sound; the source itself says that depends on how each one is designed.

Risks and caveats

The source is explicit that a custom benchmark is only as good as its design: whether it is methodologically sound and captures actual business value still depends on how the user builds it, and Optima does not remove the deeper challenges of benchmarking, how precisely the target capabilities are defined, how representative the test cases are, and how carefully the evaluation is implemented and documented. Cost and time per task are also not the same as business value: a cheap, fast AI workflow can still be inefficient if its output needs heavy rework or adds little value to the process it serves.