Glonce

AI inference costs fell about 2.5 orders of magnitude in a year, blog post argues

infrastructureThe Glonce editorsHacker News🔥74Glonce rating74Impact68Novelty68Relevance100Surprise55Players45Credibility70

A blog post titled "tokens too cheap to meter" (jyn.dev) argues that the price of running machine learning inference is falling by several orders of magnitude a year with no sign of slowing, and predicts that within the next year or two LLMs will be woven into computing as infrastructure rather than sold as a standalone product, and that within 3 to 6 years frontier-quality models will run locally on commodity hardware.

The author breaks the decline into several independent factors. GPUs themselves keep getting more power-efficient: a logarithmic efficiency-over-time chart has a slope of 1.3, meaning GPU power efficiency roughly doubles every two years, a pace the author compares to Moore's Law in the 1960s. Separately, the cost to complete a given task with a model (not the raw per-token price) has fallen sharply: a "pareto frontier" chart of model quality versus task cost shows the frontier moving down and to the right over 2025, and by 2026 the cost axis is about two orders of magnitude cheaper for similar quality, with models like Claude Fable-5.1 at the expensive-and-smart end and GPT-5.6 Luna at the cheap end of the chart the author cites.

Inference engines, the software that runs a trained model on a GPU, are also improving fast, especially for "serving" workloads (live requests) rather than "offline" batch processing. vLLM went from version 0.5.4 in September 2024 to 0.11.1 in December 2025, about a 40% efficiency gain (Joules per token) in 15 months. NVIDIA reported up to 50% efficiency improvements on its MLPerf stack between versions 2.0 and 2.1, and Intel reported a 2.4x throughput increase purely from software improvements between MLPerf versions 6.0 and 6.1 on fixed hardware.

Architectural changes add further gains. Mixture-of-Experts (MoE) designs, which activate only specialized "expert" layers per input instead of the whole model, let a model be up to 7x smaller (6 billion down to 0.8 billion parameters) while matching the same benchmark performance as a dense model, though the author notes MoE helps hosted inference more than local machines since all experts must still fit in memory. For local use, "Mamba" and Mamba-transformer hybrid architectures cut the RAM needed to hold context: the Nemotron-H-47B model, quantized to 3 to 4-bit weights, can hold over a million tokens in 32 GB of VRAM, while a comparable-quality Llama-3.1 60B model would need almost 120 GB for the same number of tokens.

The post also covers specialized, non-generative models. TypeSafe AI launched a product called Jev this week (relative to the post), which cannot generate text but only returns a probability for a fixed set of options, for example judging whether a shell command violates a system prompt. Jev's pricing page lists input tokens for existing generative LLMs at $0.20 to $10 per million tokens with output tokens roughly 5x more expensive, versus $0.042 per million input tokens ($42 per billion tokens) for System One plus Jev, with output tokens free. The author illustrates this as about 3 cents to read five average books (about 80,000 words each) or $42 to read a tenth of a thousandth of every book ever written. This pricing is already cheap enough that developer tools call Jev directly: a tool called jgrep lets users pipe text through natural-language filters from the command line and, measured on 994 Hacker News titles, took 4.6 seconds and $0.012 to generate one description (the same time for three at once), returning a probability in about 200 milliseconds for roughly a thousandth of a cent. An open-weight counterpart called Laya can be faster and more accurate than Jev once fine-tuned, but performs poorly out of the box, requires real ML expertise to tune, must be self-hosted, and only supports contexts up to 512 bytes.

Putting the pieces together, the author estimates models are about 100x more cost-efficient per task than a year ago, hardware is about 1.3x more energy-efficient per token, and inference engines are about 1.4x more energy-efficient per token, combining to roughly 2.5 orders of magnitude of token-cost decline over the last year. On top of raw cost, new architectures let 5x or more tokens fit in the same RAM for local models, and specialized products like Jev and Laya can cut cost by another one to two orders of magnitude on top of that. The author stresses all of these trends are still immature and far from diminishing returns.

Key facts

  • The post estimates about a 2.5 order-of-magnitude drop in LLM token cost over the last year, from roughly 100x more cost-efficient models, 1.3x more energy-efficient hardware and 1.4x more energy-efficient inference engines
  • GPU power efficiency is doubling roughly every two years (logarithmic slope of 1.3); vLLM gained about 40% serving efficiency between its September 2024 and December 2025 releases, NVIDIA's MLPerf stack gained up to 50% between versions 2.0 and 2.1, and Intel saw a 2.4x throughput gain between MLPerf 6.0 and 6.1
  • Mixture-of-Experts architectures let a model shrink 7x (6B to 0.8B parameters) at equal benchmark performance, while Mamba-hybrid models like Nemotron-H-47B hold over a million tokens in 32 GB of VRAM versus almost 120 GB for a comparable Llama-3.1 60B model
  • TypeSafe AI's new classifier product Jev prices input tokens at $0.042 per million ($42 per billion) with free output, versus $0.20 to $10 per million for existing generative LLMs; a devtool called jgrep built on it judged 994 Hacker News titles in 4.6 seconds for $0.012
  • The author predicts LLMs become embedded computing infrastructure within a year or two and frontier-quality models run locally on commodity hardware within 3 to 6 years

Why it matters

The post's central claim is that inference is getting cheap fast enough to change what AI is for: not a metered resource people ration, but a default ingredient that gets baked into ordinary software and tools, the way a search index or a database is. If the trend holds, the bottleneck shifts from how many tokens you can afford to how good and how accessible those tokens are.

Who it affects

Developers building AI-powered tooling (the jgrep example pipes natural-language judgments straight into shell pipelines), companies deciding whether to run models locally versus via API, GPU and inference-engine vendors racing on efficiency (NVIDIA, Intel, the vLLM project), and anyone evaluating narrow classifier products like Jev or Laya instead of full generative LLMs for yes/no or scoring tasks.

How to use it

The concrete products named are Jev, a proprietary classifier from TypeSafe AI priced at $0.042 per million input tokens with free output, used for tasks like judging whether a shell command is destructive or triaging pull requests by reading every comment; and Laya, an open-weight alternative that can be run and fine-tuned locally but needs real ML know-how, self-hosting, and is limited to contexts up to 512 bytes. The jgrep command-line tool built on Jev lets users filter text streams (including live tail -f output) with natural-language queries.

How solid is it

The author backs the claims with cited benchmark charts and version-to-version comparisons: vLLM release notes, NVIDIA and Intel MLPerf results, a pareto-frontier chart of model quality versus cost across 2025 and 2026, and specific model specs (Nemotron-H-47B versus Llama-3.1 60B memory use). The jgrep numbers are stated as directly measured on a set of 994 Hacker News titles. The piece is a personal technical analysis rather than a peer-reviewed study, and some of the charts and figures are sourced from third parties the post links to rather than the author's own measurements.

Risks and caveats

The multi-year predictions, LLMs as ubiquitous infrastructure within a year or two and local frontier-quality models within 3 to 6 years, are the author's own extrapolation, not a guarantee. The cheapest pricing cited comes from a single vendor's, TypeSafe AI's, own pricing page for a newly launched product. Laya, the open alternative to Jev, performs poorly without fine-tuning, demands ML expertise to use well, and is capped at 512-byte contexts, so it is not a drop-in substitute. MoE architectures mainly help hosted inference and do less for local machines, since all experts still need to fit in memory.

“It returns a probability in about 200 ms for about a thousandth of a cent, which is fast and cheap enough to sit in a pipe. jgrep reads lines as they arrive, judges them concurrently and prints matches in input order, so it works on tail -f as well as on files. Measured on 994 Hacker News titles: 4.6 seconds and $0.012 for one description, and the same time for three descriptions at once.”

— jgrep (tool self-description, quoted in the post)