Simon Willison ships condense-json 1.0 for deduplicating JSON

Simon Willison released condense-json 1.0, a Python library he first wrote about a year and a half ago, after applying what he describes as sensible and non-disruptive fixes before committing to the 1.0 label. The library scans a JSON document for strings or substrings that also appear in a separate replacements object supplied by the caller, and swaps each match for a compact reference in the form {"$r": [...]}, splicing in a {"$": ""} marker wherever the replacement text occurs. A companion function, uncondense_json, reverses the process, expanding the references back into the original strings using the same replacements object. Willison built the tool to make it cheaper to store JSON that repeats data already present in a related structure. He says he uses it to shrink the SQLite logs his LLM command line tool generates, and points to pull request #1586 as the latest step in that ongoing use.

Key facts

condense-json reached its 1.0 release, roughly a year and a half after Willison first wrote the library.
The library replaces JSON strings that match entries in a supplied replacements object with a compact {"$r": ...} reference syntax.
uncondense_json reverses the transformation, restoring the original strings from the condensed output and the same replacements object.
Willison uses the library to reduce the size of SQLite logs produced by his LLM tool, citing pull request #1586 as its latest application.
The 1.0 release consists of fixes Willison calls sensible and non-disruptive; no details of those fixes are given.

Why it matters

condense-json addresses a narrow but real cost problem: JSON logs, especially from LLM tools, often repeat the same strings (system prompts, tool schemas, boilerplate) across many records. A reference syntax that swaps repeated text for short pointers cuts storage without changing what the data represents, and the round trip through uncondense_json means nothing is lost.

Who it affects

Willison names one direct use: his own LLM command line tool, which logs conversations to a local SQLite database. Anyone storing JSON with a similar duplication pattern, such as logging pipelines that repeat shared fields across many records, could apply the same technique.

How to use it

The library exposes two functions. condense_json(input_json, replacements) takes a JSON document and a replacements object mapping keys to strings to look for, and returns a condensed version with matches swapped for {"$r": ...} references. uncondense_json(condensed, replacements) reverses it. The project is open source and Willison links pull request #1586 in the LLM project as a working example of the pattern in production use.

How solid is it

The library predates this release by about a year and a half, and Willison frames the 1.0 tag as a maturity milestone rather than a new feature, applying only fixes he calls sensible and non-disruptive. No details of those fixes, no benchmark of space saved, and no release date are given in the source.

Risks and caveats

The write-up gives no performance numbers, so the actual storage savings in Willison's SQLite logs, or in any other use case, are not quantified. The technique depends on the caller supplying an accurate and stable replacements object; nothing in the source describes error handling for mismatched or missing replacements.

“The idea is to make it easier to store JSON that includes duplicated data from other related structures.”

— Simon Willison