Glonce

Kog aims to speed up LLM inference on existing GPUs

infrastructureThe Glonce editorsTechCrunch🔥62Glonce rating62Impact60Novelty45Relevance90Surprise40Players50Credibility80
Kog aims to speed up LLM inference on existing GPUs

Kog, a French inference-software startup, is trying to prove that ordinary datacenter GPUs still have far more performance locked inside them than enterprises are using. The company hit the front page of Hacker News in May with a tech preview arguing that extremely fast single-request decoding is possible on standard GPUs that companies already own, demonstrated on AMD MI300X and Nvidia H200 hardware. CEO and solo founder Gael Delalleau told TechCrunch the preview brought in 200 tangible business leads.

The demo itself showed 3,000 tokens per second for a single request, but only on a small, purpose-built model with about 2 billion parameters, called Laneformer 2B, which Kog has since open sourced. The company's public goal is far more ambitious: 30x faster LLM inference. Reaching that with full-sized large language models, whose scale makes fast decoding much harder, is the leap Kog still has to make. Delalleau says he is confident the same approach will work on LLMs, arguing that newer GPUs carry growing memory bandwidth that goes largely unused, and that the idea GPUs are poorly suited to decoding is a misconception.

Early interest points to software engineering as the first real use case: developers using tools such as Claude Code sometimes wait hours for results, and Anthropic already charges a price premium for a faster Fast Mode. Kog also has design partners building prompt-to-game and prompt-to-app products, for whom faster inference translates directly into more revenue. The company found that its prospective customers are not willing to fine-tune small models themselves, so since the May launch it has focused entirely on accelerating larger models instead.

Delalleau contrasts Kog's approach with ZML, another French startup that built hardware-agnostic software bypassing Nvidia's CUDA to run fast inference across competing chips; he says Kog goes deeper, comparing its focus to Stanford lab Hazy Research. His own path to the problem is unusual: he studied solid-state physics at Ecole Polytechnique, then worked in offensive cybersecurity, reaching the DEFCON capture-the-flag finals four times. He says that background taught him to reverse-engineer systems down to assembly and binary code and repurpose them for goals they were not designed for, a mindset he now applies to squeezing performance out of GPU hardware at the level of physical constraints.

That depth comes at a cost: Kog spends several weeks to months per new GPU model doing low-level engineering research on it, which limits how many chips an 11-person team can support for now. Longer term, Kog wants to feed its methodology into agent-based pipelines that could cover more chips and models automatically, a direction Delalleau links to Europe's push for its own AI hardware and software capability. Kog's seed round was co-led by Varsity VC, the firm run by Kamel Zeroual, Delalleau's former co-founder at his earlier startup Stribe; the company is also backed by cloud provider Scaleway, France's Bpifrance and the French Tech 2030 program. Delalleau says the near-term milestone is implementing a first major model at 10x speed, which he expects around September, as the proof point needed to demonstrate customer traction and raise a Series A round.

Key facts

  • Kog's May tech preview, claiming fast single-request decoding on off-the-shelf AMD MI300X and Nvidia H200 GPUs, generated 200 tangible business leads, according to CEO Gael Delalleau.
  • The demo reached 3,000 tokens per second per request, but only on the small, now open-sourced 2 billion-parameter Laneformer 2B model; Kog's stated overall goal is 30x faster LLM inference.
  • Kog now targets a 10x speedup on its first major large language model, which Delalleau expects around September, as the milestone needed to demonstrate customer traction and raise a Series A.
  • The 11-person startup's seed round was co-led by Varsity VC, run by Delalleau's former Stribe co-founder Kamel Zeroual; Kog is also backed by Scaleway, Bpifrance and French Tech 2030.
  • Delalleau, a former offensive-security researcher and four-time DEFCON capture-the-flag finalist, says each new GPU model takes his team several weeks to months of low-level engineering work.

Why it matters

Inference speed and cost have become a central bottleneck in deploying AI, underlined by the market's warm reception of purpose-built inference chipmaker Cerebras at its May IPO. Kog is betting the opposite way: that conventional, already-deployed GPUs still hold large amounts of untapped performance that software alone can unlock, without enterprises buying new specialized hardware.

Who it affects

The clearest early beneficiaries are software engineers who rely on AI coding tools such as Claude Code and currently wait hours for results, plus Kog's design partners building prompt-to-app and prompt-to-game products, where faster inference means more usable output and more revenue. Longer term, GPU operators and enterprises running AI workflows on hardware they already own are the target customers.

How to use it

The Laneformer 2B model used in Kog's demo has been open sourced. The company's inference engine, the Kog Inference Engine (KIE), is aimed at enterprises and Kog's design partners rather than sold as a general consumer product, and no pricing has been disclosed.

How solid is it

The headline 3,000 tokens-per-second figure comes from a small, purpose-built 2 billion-parameter model, not from a large language model, so the promised 30x inference speedup for LLMs remains unproven. The 10x milestone on a first major model is a self-reported target Delalleau expects around September, not yet an achieved result.

Risks and caveats

Kog is a solo-founder company of 11 people, and its seed round's size was not disclosed beyond being co-led by Varsity VC. The company's own account shows its first go-to-market idea, fine-tuning small models for customers, did not find demand, forcing a pivot toward larger models. Its optimization method requires weeks to months of manual engineering per new GPU, which caps how many chips the team can support while it tries to prove the approach on LLMs.

“GPUs have a bright future”

— Gael Delalleau, CEO of Kog