Glonce

Five startups pitch alternatives to transformers in LLMs

researchThe Glonce editorsMIT Technology Review🔥74Glonce rating74Impact70Novelty65Relevance100Surprise55Players65Credibility75
Five startups pitch alternatives to transformers in LLMs

The transformer architecture, introduced by Google researchers in the 2017 paper "Attention Is All You Need," powers every major LLM on the market today. Its core mechanism, dense attention, compares every word in a passage of text with every other word to encode meaning, and the cost of that comparison grows fast: a 10,000-word document can require a transformer to perform 50 million multiplications. That cost shows up as money and power. OpenAI president Greg Brockman has said the company is set to spend $50 billion on computing this year, and the International Energy Agency projects that data centers' total electricity consumption will double by 2030. Dense attention also makes it hard for transformers to hold large amounts of context, a problem that gets worse as reasoning models write and reread their own chains of thought. MIT Technology Review's "What's Next" series profiles five startups building around these limits.

Subquadratic, based in Miami, says it has built the first sparse attention mechanism, a technique that runs calculations on only some word pairings instead of all of them, that rivals top mainstream LLMs on tasks including search and coding. Cofounder and CEO Justin Dangel calls transformers "one of the most important innovations in the history of computer science." The company's model, SubQ, decides on the fly which words in a given passage matter; Subquadratic says thousands have joined its waitlist and it plans wider availability soon. The claim of matching top LLMs' quality is described in the piece as one that "some people in the industry remain skeptical" about.

Manifest AI, based in San Francisco, replaces attention altogether with what it calls power retention: instead of tracking everything in a context window, the model keeps a rolling summary and drops less relevant information as new input arrives. The company says the ten-year-old idea of retention has been updated to finally stand up to transformer-based LLMs. Manifest AI has converted the open-source coding model StarCoder into a power retention version called PowerCoder, and released a model called Brumby that it says rivals some versions of Alibaba's Qwen.

Liquid AI, an MIT spinout in Cambridge, Massachusetts, pairs transformers with its own liquid neural networks, inspired by worm brains, to build what cofounder and CEO Ramin Hasani calls liquid foundation models (LFMs). Its recent LFMs mix 20% transformer components with 80% liquid neural network components, a ratio worked out by a separate in-house AI system that searches combinations of network types for the best tradeoff of performance and efficiency. The models run on car chips for automakers including Mercedes and on a $50 Raspberry Pi, are free to any organization with annual revenue under $10 million, and have been downloaded almost 34 million times. Liquid AI says its new models match the performance of rivals four times their size, including versions of Qwen and Google's Gemma. Hasani argues transformers are only a start: "Your brain is an AGI system, and it operates with 20 watts of power. How is it possible? We can get a lot more innovative."

Inception, based in Palo Alto, builds LLMs with diffusion, the technique behind most image and video generators, which produces a whole block of text at once rather than word by word. Cofounder and CEO Stefano Ermon, also a Stanford researcher, worked out with two Stanford colleagues in 2024 how to make diffusion work on text; their first model matched the performance of OpenAI's 2019 GPT-2 while running 10 times faster. Inception's latest model, Mercury 2, is claimed to perform as well as some of OpenAI's 2023 GPT-4 models while again running 10 times faster. Diffusion LLMs still use transformers to encode meaning, Ermon says, but by generating many tokens at once they make transformers do more work per computation. Google is also experimenting with diffusion and has built a prototype called Diffusion Gemma, but Ermon says the competition is "validating."

A fifth startup, Pathway, is introduced as the most radical of the group. It has built a model called Dragon Hatchling, named after dragons in Terry Pratchett's novel The Colour of Magic that materialize when thought about hard enough. Its headline result is solving more than 97% of a benchmark of over 250,000 hard sudoku puzzles, a set on which several leading LLMs from top labs reportedly solved none. The source text is cut off mid-sentence before explaining what Pathway's approach is or why the sudoku result matters.

Key facts

  • Dense attention, the mechanism at the core of transformers, can require about 50 million multiplications to process a 10,000-word document. Separately, OpenAI is set to spend a projected $50 billion on computing this year, according to OpenAI president Greg Brockman.
  • Subquadratic claims its sparse attention model SubQ rivals top mainstream LLMs on search and coding, though the piece notes some in the industry remain skeptical of the claim.
  • Liquid AI's hybrid liquid-neural-network models (20% transformer, 80% liquid network) run on a $50 Raspberry Pi, are free for organizations under $10 million in annual revenue, and have racked up almost 34 million downloads.
  • Inception's diffusion-based Mercury 2 is claimed to match some of OpenAI's 2023 GPT-4 models while running 10 times faster, building on a 2024 Stanford result that matched GPT-2 at the same speed multiple.
  • Pathway's Dragon Hatchling model solved over 97% of a 250,000-plus sudoku benchmark on which several leading LLMs from top labs reportedly solved none.

Why it matters

Transformers are nine years old and, per the piece, are starting to show their age: the dense attention mechanism that made them powerful also makes them expensive to run and limited in how much context they can track well, a problem that compounds as reasoning models reread their own chains of thought and as agents pass output between each other. With OpenAI alone projected to spend $50 billion on compute this year and the IEA forecasting that data center electricity use will double by 2030, the cost of scaling transformers further is becoming a structural constraint on the industry rather than a minor inefficiency. That is the opening five startups are trying to exploit with new architectures rather than incremental patches.

Who it affects

The story centers on four companies attacking different parts of the problem: Subquadratic (sparse attention), Manifest AI (power retention), Liquid AI (liquid neural networks paired with transformers), and Inception (diffusion-based text generation), with a fifth, Pathway, introduced at the end. Their comparisons and rivals are the AI incumbents whose models run on standard transformers, including OpenAI's GPT-2 and GPT-4 lines and Alibaba's Qwen; Google appears twice, both as a target for comparison (its open-source Gemma) and as a competitor pursuing the same idea (a prototype called Diffusion Gemma). Liquid AI's automotive customer Mercedes and any organization running LLMs on constrained hardware or a tight budget are also directly affected, since several of these architectures are pitched specifically at cheaper or more efficient inference.

How to use it

Availability varies by company. Liquid AI's models are the most accessible: they run on a $50 Raspberry Pi, are downloadable (almost 34 million downloads to date), and are free to any organization with annual revenue under $10 million. Manifest AI has released PowerCoder (a power retention version of the open-source StarCoder model) and Brumby, both already out. Subquadratic's SubQ is not yet generally available: the company says thousands have joined its waitlist and it plans wider release soon. The piece does not state how Inception's Mercury 2 or Pathway's Dragon Hatchling can currently be accessed, or give pricing for any of the paid tiers implied by Liquid AI's revenue-based free threshold.

How solid is it

Every performance claim in the piece comes from the companies themselves: Subquadratic on SubQ rivaling mainstream LLMs at search and coding, Manifest AI on Brumby rivaling some versions of Qwen, Liquid AI on matching models four times its size, and Inception on Mercury 2 matching some GPT-4 models at 10 times the speed. The article flags skepticism only around Subquadratic's claim, noting explicitly that some in the industry doubt it; it does not report independent benchmarking, peer review, or third-party verification for any of the four companies. Dragon Hatchling's sudoku result is the one figure framed as a hard, measurable benchmark score rather than a comparative claim, though the source cuts off before explaining what the benchmark tests or why the score is considered significant.

Risks and caveats

The article itself frames this as an early, unsettled field: multiple startups are chasing the same problem from different angles, and it says plainly that some will fail. None of the four are shown facing off against each other or against incumbents in a shared, independent test, and Google's own Diffusion Gemma prototype shows that established labs are exploring at least one of these approaches too, which could blunt any edge a startup holds. The source text available for this story is cut off mid-sentence just as it begins to explain Pathway and Dragon Hatchling in more depth, so whatever it says about that startup's approach or the sudoku result's broader significance is not reflected here. No founding dates or funding figures are given for any of the five companies.

“The entire AI industry is built on transformers. They are one of the most important innovations in the history of computer science, and they've changed the world.”

— Justin Dangel, cofounder and CEO of Subquadratic