gpu-lexer highlights syntax in any language via a WebGPU model in a 27.5KB library
A new project called gpu-lexer offers syntax highlighting without a traditional grammar for each language. The library first splits source code into simple parts: words, whitespace, newlines and symbols. A small WebGPU model then combines local and whole-file context to label each part as one of a fixed set of types (plain, comment, string, number, keyword, type, function, constant or operator). Adjacent labels of the same type are merged into the highlight spans the library returns. Because the model works from surrounding context rather than a hand-written grammar, its creator says it can guess token types for a language or syntax it never saw during training, and the live demos cover 75 languages. The whole library is 27.5KB. Usage is a single async call: importing highlight from gpu-lexer and awaiting highlight('source code') returns an array of objects, each with a type, a start index and an end index. The project page is explicit about the model's limits: on files held out of training, 12.57% of the current model's token labels differ from those Shiki would assign. The page stresses that this figure measures agreement with Shiki, not objective correctness, and that unseen languages or real-world code may show higher disagreement than that. The author frames gpu-lexer as an experiment rather than a finished, grammar-equivalent highlighter.
Key facts
- gpu-lexer is a 27.5KB JavaScript library that highlights source code using a small WebGPU model instead of a per-language grammar
- The model labels tokens as plain, comment, string, number, keyword, type, function, constant or operator based on local and whole-file context
- On files held out of training, 12.57% of the current model's token labels differ from Shiki's, a figure the source describes as agreement with Shiki rather than objective correctness
- Live demos on the project site cover 75 languages
- The project page describes gpu-lexer as an experiment, not a grammar-equivalent highlighter, and warns unseen languages or real-world code may diverge more
Why it matters
Syntax highlighters normally need a hand-written grammar per language, which is why obscure or new languages often go unsupported by tools like Shiki. gpu-lexer replaces that grammar with a small WebGPU model that infers token types from context, which is a different approach to a problem that has mostly been solved with parsers and regular expressions rather than machine learning.
Who it affects
Developers who build code editors, documentation sites or other tools that render highlighted code, and anyone maintaining or choosing between highlighters such as Shiki, are the direct audience. It is also relevant to users writing in niche or newer languages that lack a maintained grammar for existing highlighters.
How to use it
The library exposes one function: importing highlight from gpu-lexer and calling await highlight('source code') returns an array of spans, each carrying a type, a start offset and an end offset. The project page links to live demos covering 75 languages so the output can be inspected directly rather than only read about. The source text gives no pricing, license terms or installation instructions beyond the import shown.
How solid is it
The only benchmark given is agreement with Shiki: on files excluded from training, 12.57% of the model's token labels differ from what Shiki would assign. The page itself frames this as a measure of agreement rather than correctness, since Shiki's own labels are not treated as ground truth. No comparison to other highlighters, no accuracy figure against a labeled correctness standard and no performance or speed benchmark is given in the source.
Risks and caveats
The source states directly that unseen languages or real-world code may produce a higher disagreement rate than the measured 12.57%, and calls the project an experiment rather than a finished, grammar-equivalent highlighter. The page text does not name an author, company or institution behind gpu-lexer, does not give a release date, and does not describe the model's architecture, parameter count or training process beyond the library's overall 27.5KB size.
“This is an experiment, not a grammar-equivalent highlighter.”
— gpu-lexer project page