Glonce

Grep beats the Language Server Protocol on tokens, study finds

researchThe Glonce editorsarXiv🔥75Glonce rating75Impact65Novelty70Relevance100Surprise75Players30Credibility80

Coding agents burn most of their context budget on retrieval: finding the right code before they can act on it. Lexical retrieval, plain grep, is universal, instant and needs no setup, but it is noisy, unable to tell a definition from a call from a comment. Semantic retrieval through the Language Server Protocol (LSP) is precise and typed, but it needs a running, indexed server and pays a round trip per symbol. The claim that semantic retrieval is more token-efficient is, the authors write, asserted almost everywhere and measured almost nowhere: no public source isolates the LSP-versus-lexical token delta for an agent at equal task success.

The paper sets out to fix that. It defines a single metric, tokens-to-success, specifies a five-arm ablation designed to isolate semantic retrieval from confounding factors, and maps three pre-stated failure modes onto measurable variables. It then reports a preliminary study on Python and TypeScript repositories, run with three Claude models: Opus 4.8, Sonnet 4.6 and Haiku 4.5.

On symbol-named localization tasks, the LSP costs more tokens than grep, between 6% and 118% more, and the models mostly ignore it when it is optional: semantic tool use sits at just 0-6% there. On reference-completeness tasks the picture flips only partway. The LSP buys precision but not token savings, and it cannot raise the recall ceiling set by how thorough the agent already is; it reduces tokens only for the weakest of the three models. Even so, tool choice tracks the task rather than a fixed preference: the same models that skip the LSP on localization reach for it about half the time on reference tasks, unprompted.

The gap is starkest on edits scored by real test execution, specifically multi-file renames. Grep solves them perfectly. A location-only LSP fails three-quarters of them by missing a call site. A more complete LSP setup, index-warmed and enriched with each reference's source line inline, the way production LSP-MCP servers work, recovers most of that gap but still cannot close it, because a rename has to touch comments and strings that semantic references leave out.

The authors' conclusion is not that agents should always reach for the LSP, but that retrieval choice should be an adaptive router keyed on task class, model capability and how noisy lexical search is expected to be for that task.

Key facts

  • On symbol-named localization tasks, using the LSP instead of grep costs a coding agent 6% to 118% more tokens.
  • Models mostly skip the LSP when it is optional for localization, using it only 0-6% of the time, but reach for it about half the time on reference tasks, unprompted.
  • On reference-completeness tasks, semantic retrieval improves precision but saves tokens only for the weakest of the three tested models.
  • On real multi-file renames scored by test execution, grep solves them perfectly while a location-only LSP fails three-quarters of them by missing a call site.
  • Even a complete, index-warmed, text-enriched LSP setup narrows but cannot close that renaming gap, since it excludes comments and strings a rename must also touch.

Why it matters

The efficiency case for wiring a Language Server into a coding agent is repeated constantly in agent-tooling discussions but, per the paper, had never been measured head to head against grep at equal task success. This study builds a metric and an ablation designed specifically to isolate that comparison, and its headline finding cuts against the common assumption: LSP-based retrieval is often the more expensive option, not the cheaper one.

Who it affects

Anyone building or configuring coding-agent tooling, including LSP-MCP server integrations, and anyone deciding whether an agent pipeline needs a running, indexed language server at all. The study tests three Claude models of different capability, Opus 4.8, Sonnet 4.6 and Haiku 4.5, so the effect is not uniform across a single tier of model.

How to use it

The paper's own recommendation is not LSP-always but a router that picks retrieval mode by task class and model capability. For symbol-named localization, grep is both what models already choose and the cheaper option. For multi-file renames specifically, grep alone already solves the task; adding a location-only LSP does not help and can miss call sites, and even a richer, text-enriched LSP setup only narrows the gap because renames also touch comments and strings that semantic references exclude.

How solid is it

This is a preliminary study, explicitly framed as such, built on a formal five-arm ablation meant to isolate semantic retrieval from confounding variables. It covers Python and TypeScript repositories and three Claude models; the source text does not give a repository count, sample size, or task count beyond that description.

Risks and caveats

The paper does not give a specific number for how many tokens the LSP saves on reference-completeness for the weakest model, only that a saving exists there and nowhere else. It also names three pre-stated failure modes as mapped to measurable variables without spelling those modes out in the material available here. As a preliminary study on a limited set of repositories and models, the findings may not generalize to other languages, codebases, or model families.