Glonce

Netlify compares Claude, GPT, Gemini, Kimi and GLM on credits and quality

modelsThe Glonce editorsHacker News🔥56Glonce rating56Impact35Novelty32Relevance88Surprise32Players48Credibility78
Netlify compares Claude, GPT, Gemini, Kimi and GLM on credits and quality

Netlify has launched a partnership with OpenRouter that adds two new capabilities to its platform. First, any project built on Netlify can now reach any model on OpenRouter through Netlify's AI Gateway, widening the model choice available to end-user-facing AI features in web apps. Second, Netlify's own in-product coding assistant, Agent Runners, the chat prompt box inside Netlify that lets a developer build a new project from scratch or iterate on an existing one, expands its selection of frontier coding models to include Kimi K3, GLM 5.2 and DeepSeek V4, open to everyone. Agent Runners runs a full coding agent rather than a pared-down one; until now it supported Claude Agent, OpenAI Codex and Gemini CLI, each optimized to run models from its own provider. To properly drive the newly available range of models, Netlify has added OpenCode, a popular open-source agent, as a further choice. Netlify also gives these agents extra skills and project context so they know which Netlify capabilities, such as Netlify Database, the AI Gateway, or Identity, are available, and when and how to use them.

More model choice raises an obvious question: which one is actually right for a given job, and whether a cheaper option is missing out on something a pricier one offers. Netlify's answer combines two different comparisons. Internally, it evaluates models automatically with AXIS, a tool it has recently open-sourced: AXIS runs prompts that build a site and then iterate on it, and scores the result on functional correctness, for instance whether the generated site wires up Netlify Database when the use case calls for one, and stays simple, with no database, when it does not. A model that falls behind on those internal scores is not offered in Agent Runners at all; if a model fails to apply one of Netlify's built-in skills correctly, or works but at what looks like an inflated credit cost, Netlify treats that as a problem with the skill to fix, not with the model.

For this post, though, Netlify ran a second, more subjective comparison meant to show what the generated sites actually look like and how many Netlify credits each model burns to get there, with every model run on its default settings. It set up three test cases: a one-page site for a local coffee shop, later extended with a seat-reservation request; a shared to-do list app that needs a live database from the start, later extended with a per-item photo upload; and a 'What can I cook' app that calls Netlify's AI Gateway to turn a list of ingredients into a recipe suggestion. Each model was run three times per test case. This post covers only the first case, the coffee-shop site; results for the to-do list and recipe apps are promised in follow-up posts. The full report, including links to each generated site, is hosted on a companion site Netlify built for it.

The coffee-shop prompt read: 'Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself,' the last line added as a hint that no content management system was needed. Netlify's default skills also carry general UI guidance meant to avoid known problems, such as generic AI-slop visuals, and to get each model reasoning about a visual identity that fits the brief. One of Claude Opus 5's three runs cost 1,055 credits, about four times more credits than any other single run in the whole test; its other two runs cost 253 and 249 credits, which is why Netlify calls the Opus average 'heavily slanted upwards'. For scale, Netlify's free plan includes 300 credits, its Personal plan 1,000, and its Pro plan 3,000, with extra credit packs for Pro priced at $10 per 1,500 credits. The 1,055-credit page is, in the post's own assessment, delightful and full of detail, down to a 'stamp-like' element with a coffee bean at its center built as an actual animatable text element, a custom map at the bottom, and dark mode that works out of the box, though its two-font, two-color heading is called hip but perhaps 'cliche by now.' The other two Opus runs are described as not bad either, since vector graphics of this kind sit near the frontier of what current LLMs can do, a frontier the post itself says is 'not in a very good place yet' next to image or text generation. More generally, the post says Opus tends to run off with excessive credit usage compared to its own typical baseline more than other models do, without that translating into a guaranteed better or worse result. Claude Sonnet 5's three runs cost 81, 245 and 103 credits, 143 on average; the post still finds delightful detail in each but less of it and less content overall, with simpler vector graphics that would not really suit a live site. The post is careful to add that none of this says anything about a model's ability to write complex code or answer harder questions, since that is not what this particular test asks of it.

GPT 5.6 Sol, which the post calls OpenAI's Opus-class model, is run by Netlify on low effort by default, an intentionally more economical alternative to Opus that Netlify says still gives 'pretty darn good' results, though the effort level is user-adjustable and Netlify's default may change over time. Its three runs cost 173, 158 and 92 credits, 141 on average, and the post judges that, at least in this scenario, OpenAI's top-tier model in low-effort mode beats Anthropic's mid-tier Claude Sonnet 5 on basic design sense, with richer content and no odd vector shapes, though its images read as a bit generic. GPT 5.6 Terra, one tier down from Sol in OpenAI's own lineup, which runs Sol, then Terra, then Luna from top to bottom, cost 43, 23 and 49 credits, 39 on average, and surprisingly did not show the same size of quality drop that Opus to Sonnet did: its visual language is simply different, not necessarily worse, if simpler on content, with minor glitches, a missing image in one run, low-contrast text over an image in another, that the post calls nothing seriously wrong overall. At this point, the author's own preference, given only a very vague sense of the desired look for a project, is to run the same prompt through both Opus 5 and GPT 5.6 Terra for two very different but worthwhile takes.

Gemini 3.1 Pro and Gemini 3.6 Flash, described as not being of the same model generation, show it. Gemini 3.1 Pro averaged 53 credits, and, in the post's words, did what was asked and 'really nothing more,' with results the post considered not worth linking to. Gemini 3.6 Flash cost 109, 91 and 111 credits, 103 on average, and reads like 'a whole new generation,' with results more in line with other modern models and noticeably more effort put into content, though the post adds that, while all models repeat themselves to some degree, Gemini seems to repeat itself even more.

Kimi K3, which the post says is marketed mainly as a frontier model for long-horizon agentic tasks and built more to rival Fable 5 than Opus 5, a positioning the post says various benchmarks and reviews confirm, cost 125, 95 and 86 credits, 102 on average, but did not particularly stand out among the others on this narrow, design-led task; the post says doing the model justice would need a wholly different set of prompts built around a more complex web app, promised for a follow-up post. Kimi K2.7 Code, called a big step back in model architecture from K3, averaged a very low 19 credits, and, despite some hype about Kimi's visual capabilities around the K2.6 launch, showed little to see on design or content. GLM 5.2's three runs cost 15, 42 and 24 credits, 27 on average; surprisingly, aside from a shared 'love for maple' running through all three pages, they looked very different from each other, in the post's words 'as if coming from a few different models,' and given GLM's low cost, the post suggests running it a few times before settling on a result. Being a text-only model that does not take image inputs, GLM 5.2 in its current iteration cannot do something the post never gets to specify: the captured text ends mid-sentence at that exact point, before it can finish the thought, cover DeepSeek V4 (named earlier as newly available in Agent Runners but not shown being benchmarked here), or offer any closing verdict across the nine models it does show results for.

Key facts

  • Netlify's new OpenRouter partnership opens its AI Gateway to any OpenRouter model and adds Kimi K3, GLM 5.2 and DeepSeek V4 to Agent Runners' coding model lineup, alongside a new open-source OpenCode agent.
  • Using its AXIS evaluator plus a separate, more informal three-run-per-model test, Netlify compared nine models on an identical one-page coffee-shop-site prompt; this post covers only that first of three planned test scenarios.
  • One of Claude Opus 5's three runs cost 1,055 credits, about four times any other single run in the test; its other two runs cost 253 and 249 credits, versus 141 to 143 credits on average for GPT 5.6 Sol and Claude Sonnet 5.
  • The cheapest results came from GLM 5.2 (15, 42 and 24 credits) and Kimi K2.7 Code (19 credits on average), while Kimi K3, built more to rival Fable 5 than Opus 5, averaged 102 credits without standing out on this design-led task.
  • Netlify's plans include 300 credits on Free, 1,000 on Personal and 3,000 on Pro, with extra Pro credit packs priced at $10 per 1,500 credits.

Why it matters

The story is really two things at once. First, Netlify's OpenRouter partnership makes real what had been a hypothetical: Agent Runners, Netlify's built-in coding agent, now reaches trendy open models like Kimi K3, GLM 5.2 and DeepSeek V4, plus anything else on OpenRouter, alongside the existing Claude, OpenAI and Gemini options, with a new OpenCode agent added to drive them properly. Second, more choice creates exactly the FOMO question the post opens with: is a cheaper model missing something a pricier one has, or is the pricier one worth its extra cost. Instead of leaving that to guesswork, Netlify backs the launch with actual numbers, both an internal evaluator, AXIS, that gates which models even qualify for Agent Runners, and a public, run-by-run comparison of what nine of those models build and what they cost in Netlify's own credits for an identical task.

Who it affects

Anyone building with Netlify's Agent Runners, or deciding whether to, especially developers watching credit spend against a Free (300 credits), Personal (1,000) or Pro (3,000, plus $10 per 1,500-credit top-up) plan. It also affects builders of end-user-facing AI features, who can now call any OpenRouter model through Netlify's AI Gateway rather than a fixed provider list. Because this particular post scopes its comparison to one simple job, a static one-page brochure site with no database, it speaks most directly to that use case; Netlify says the more demanding ones, a database-backed to-do app and an AI-Gateway-calling recipe app, are covered in posts still to come.

How to use it

The practical numbers: on the coffee-shop prompt, GLM 5.2 and Kimi K2.7 Code were the cheapest at 27 and 19 credits on average, GPT 5.6 Terra next at 39, then Gemini 3.1 Pro at 53, Kimi K3 at 102 and Gemini 3.6 Flash at 103, then GPT 5.6 Sol and Claude Sonnet 5 close together at 141 and 143, with Claude Opus 5 the most expensive and least predictable, at 253 and 249 credits for two runs and 1,055 for a third. OpenAI's GPT 5.6 line runs Sol, then Terra, then Luna from top to bottom, and Netlify runs Sol on low effort by default specifically as a cheaper alternative to Opus, though that effort setting can be changed by the user. The post's own tip: with only a vague idea of the desired look, run the same prompt through both Opus 5 and GPT 5.6 Terra to get two different, useful directions before committing to one.

How solid is it

This is Netlify grading its own product with its own prompts, on a sample of three runs per model for one narrow use case, a static one-page site with no database, and the post is explicit that this says nothing about a model's ability to write complex code or reason through harder problems, since that is not what was tested. It is also incomplete on its own terms: the post's title promises 11 models, but the captured text only names and gives credit figures for nine, Claude Opus 5, Claude Sonnet 5, GPT 5.6 Sol, GPT 5.6 Terra, Gemini 3.1 Pro, Gemini 3.6 Flash, Kimi K3, Kimi K2.7 Code and GLM 5.2; the other two are never identified in what was captured, which cuts off mid-sentence in the GLM 5.2 section before showing any DeepSeek V4 results or offering a closing ranking. It never says whether GPT 5.6 Luna, the tier below Terra, was tested in this scenario at all. No individual author, executive or spokesperson is named anywhere in the text; it is written in first-person plural with occasional unattributed first-person singular asides. Unlike the other eight models, the captured text gives no stated average for Claude Opus 5, only three individual figures and a note that the average skews high; the captured text gives Gemini 3.1 Pro and Kimi K2.7 Code an average but no per-run breakdown.

Risks and caveats

Netlify is comparing its own coding-agent product, and by extension its new OpenRouter partner models, using its own prompts, its own AXIS scoring rubric and its own credit pricing, so this is a vendor's self-comparison, not an independent benchmark, and the credit figures are specific to Netlify's own pricing rather than a portable cost measure. The one limitation flagged for GLM 5.2, tied to it being a text-only model with no image inputs, is never actually stated: the post cuts off mid-sentence right after naming the cause, before saying what GLM 5.2 actually cannot do. And because the entire comparison rests on one simple, static-site prompt, explicitly not the long-horizon agentic workload a model like Kimi K3 is said to be built for, none of it should be read as a general ranking of these models' coding ability.

“Opus does have a tendency to run off with excessive credit usage (compared to its "typical" baseline) more than other models. It does not guarantee a worse or better outcome, though.”

— the post's author, writing for Netlify