WebLLM runs LLMs in the browser using WebGPU acceleration

WebLLM is an open-source JavaScript and TypeScript engine, built by the MLC LLM project, that runs large language model inference entirely inside a web browser. Everything happens client-side: there is no server component, and the engine is accelerated through WebGPU, the browser's hardware acceleration API.

The engine exposes an interface that is fully compatible with the OpenAI API, so an application already built against OpenAI's chat completions calls can point at WebLLM's locally running models with the same code. Supported functionality includes streaming responses, JSON mode structured generation (implemented in the WebAssembly portion of the model library for performance), logit level control, and seeding; function calling is listed as a work in progress.

WebLLM natively supports several model families, including Llama 3, Phi 3, Gemma, Mistral, and Qwen2, with Qwen2 available in 0.5B, 1.5B, and 7B parameter sizes. Custom models can be compiled into MLC's own format and integrated the same way. The project ships as the npm package @mlc-ai/web-llm, installable via npm, Yarn, or pnpm, and can also be imported directly from a CDN (jsdelivr's esm.run) for use in browser based playgrounds such as JSFiddle, CodePen, or Scribbler.

For caching downloaded model weights between sessions, WebLLM supports four cache backends, configurable through AppConfig.cacheBackend: the browser's Cache API (the default), IndexedDB, the Origin Private File System (OPFS), and an experimental Chrome only Cross Origin Storage API backend that requires a separate browser extension.

To keep the page responsive during inference, WebLLM provides Web Worker and Service Worker integrations that move the heavy computation off the main thread. The documentation specifically warns that a Service Worker's message handler must be instantiated at the top level of the worker script during its initial evaluation, not inside an activate or message listener, because the browser can restart an already active worker without firing another activate event; skipping that step can silently break the handler after such a restart. The project also documents example integrations for building Chrome extensions on top of WebLLM.

Key facts

WebLLM runs LLM inference entirely in the browser using WebGPU for hardware acceleration, with no server component.
Its interface is fully compatible with the OpenAI API, supporting streaming, JSON mode, logit level control, and seeding; function calling is still a work in progress.
It natively supports model families including Llama 3, Phi 3, Gemma, Mistral, and Qwen2 in 0.5B, 1.5B, and 7B parameter sizes, plus custom MLC-format models.
It offers four cache backends for storing downloaded model weights: the browser Cache API, IndexedDB, OPFS, and an experimental Chrome Cross Origin Storage backend.
It ships as the npm package @mlc-ai/web-llm and includes Web Worker and Service Worker support to keep inference off the browser's UI thread.

Why it matters

WebLLM lets developers run language model inference directly in a user's browser, with no server side inference cost and nothing sent to a remote API, while keeping the same OpenAI style interface developers already build against. Because it relies on WebGPU, it can use a user's local GPU without a native install. Client side inference of this kind removes a whole layer of backend infrastructure that a server hosted LLM app would otherwise need.

Who it affects

Web developers adding AI features such as chatbots or assistants to browser applications without standing up an inference backend; builders of Chrome extensions on top of WebLLM; developers working in CDN based, no build environments like JSFiddle, CodePen, or Scribbler. WebLLM is described as a companion project to MLC LLM, which targets universal LLM deployment across hardware, so the same underlying model compilation work carries over.

How to use it

Install the npm package @mlc-ai/web-llm with npm, Yarn, or pnpm, or import it directly from a CDN URL. Create an engine with CreateMLCEngine(model), then call engine.chat.completions.create() with OpenAI style messages, optionally passing stream: true for streaming output. For production interfaces, move inference into a Web Worker or Service Worker so the main thread stays responsive, and pick one of the four cache backends to control how model weights are cached between page loads. The source gives no pricing or licensing terms; it is distributed as an open source npm package.

How solid is it

The source is the project's own README and documentation, not an independent benchmark or third party review. No performance, speed, or latency figures appear anywhere in the text, so no throughput or quality claims can be checked against the source itself. The text also draws no comparison to other in-browser or server based inference engines, so how WebLLM stacks up against alternatives is left unaddressed by the material at hand.

Risks and caveats

Function calling is explicitly marked as a work in progress, not a finished feature. The Service Worker integration carries a documented pitfall: if its handler is not instantiated at the top level of the worker script during initial evaluation, a browser initiated restart of an already active worker can silently break message handling, since no activate event fires again in that case. The experimental Cross Origin Storage cache backend depends on installing a separate, non-default browser extension and does not support programmatic cache deletion. The source states no release date, version number, or team and funding information beyond the companion project link to MLC LLM.

“WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration.”

— WebLLM README