Lemmalog turns LLM memory into a Datalog analysis engine

The author has spent months running LLM agents for vulnerability research: navigating large codebases, explaining unfamiliar subsystems, exploring attack surfaces. The agents are good at this, but once an investigation runs a few hours, the model starts losing track of what has actually been established. It proposes an approach that was already ruled out, forgets that an assumption turned out to be false, or keeps reasoning from an observation that is no longer valid. Telling the model a fact is wrong does not reliably make it stop believing everything that depended on that fact.

Existing LLM memory systems store past observations, embed them, and retrieve the most relevant pieces on demand. That works reasonably well for recalling what was said, but not for tracking what is currently true. In the author's worked example: the agent establishes that the attacker controls object_a, object_a points to object_b, and object_b is a kernel object, from which it concludes the attacker controls a kernel object. Two hours later, debugging in LLDB shows object_a does not actually point to object_b. The memory store now holds both the original chain and the correction, and a retrieval-based system has to hope the model, working from a retrieved subset, notices which conclusions no longer hold.

This looked to the author like a program-analysis problem: a set of facts, rules that derive further facts from them, and a fixed point representing everything currently known, with existing techniques for updating only the affected results when one input fact changes, instead of rerunning the whole analysis. Rather than asking the LLM to reconstruct its entire investigation from a transcript every time something changes, he built the same machinery for agent memory: a Datalog engine called Lemmalog.

Datalog stores facts such as controls(attacker, object_a), points_to(object_a, object_b) and kernel_object(object_b), plus rules such as "controls_kernel_object(Attacker) if Attacker controls ObjectA, ObjectA points to ObjectB, and ObjectB is a kernel object," from which the engine derives controls_kernel_object(attacker). If points_to(object_a, object_b) later turns out to be false, the engine knows exactly which derived fact depended on it and retracts that conclusion automatically, rather than relying on the model to notice.

In Lemmalog's design, the LLM handles the fuzzy part: turning something like "LLDB shows that the freed object is later reused as the destination of the write" into structured facts such as freed(object_a) and reused_as(object_a, write_target). Lemmalog then handles the deterministic part, running those facts through rules to produce derived facts, so the model is no longer responsible for repeatedly working out the consequences itself.

Removing facts turned out to be harder than adding them. In a small example with facts a and b, and rules deriving c from either a or b, removing a alone must not remove c, since b still supports it; only removing both should. This matters in vulnerability research because a conclusion such as "candidate_3 is exploitable" can remain valid through an independent path even after one particular exploit primitive turns out not to work. Lemmalog tracks how each fact was derived and updates that support when something changes, which also lets it answer why a conclusion is true, by walking the chain of underlying observations and rules back down. If an agent claims "we already established this pointer is attacker-controlled" and no such chain exists in Lemmalog, the claim is not actually part of the maintained state. The author is explicit that this does not stop an LLM from hallucinating during fact extraction, but it makes it much harder for an unsupported conclusion to silently stick around afterward.

Replacing a fact is not always the same as deleting it either: if the system first believes a given exploit primitive is viable and later finds it is not, both states stay useful, so Lemmalog attaches validity intervals to facts (for example, viable from one timestamp to another, then not_viable from that point on), letting the same store answer both "is it viable now" and "why did we previously think it was viable," without keeping two contradictory facts around for the model to sort out.

The author contrasts this with vector databases, which he says answer a different question: given a query, what past information is relevant, rather than given everything learned so far, what is currently true. A vector store can retrieve a statement because it is semantically relevant even after it has been disproven, with no inherent way to know that other conclusions depended on it and should no longer be treated as valid; retrieval and Lemmalog's fact maintenance are complementary, and the author currently uses both together.

He describes the overall system as a kind of compiler: the LLM is the front-end, turning source code, debugger output and natural-language notes into structured facts; Lemmalog is the intermediate representation and analysis engine, running those facts through deductive rules to produce maintained state; a further LLM call can turn that state back into natural language, propose the next experiment, or act on it. The engine as described already supports incremental evaluation, retractions, provenance, temporal facts, aggregations, entity reconciliation, hybrid retrieval and demand-driven queries. The retrieved text breaks off immediately after the author poses the central open question, "Does it actually make LLMs better?", without giving a benchmark or a concrete before-and-after comparison.

Key facts

• Lemmalog is a Datalog engine: it stores facts and rules, computes a fixed point of derived conclusions, and updates only the affected conclusions incrementally when one input fact changes, instead of making the LLM reconstruct the whole investigation.
• Each derived fact keeps provenance, a record of which observations and rules produced it, so the system can explain why a conclusion holds, and a claim with no supporting chain is treated as not actually part of the maintained state.
• Because a conclusion can have more than one independent supporting derivation, retracting one underlying fact removes only the dependent conclusions that lose all their support, not everything ever derived from it.
• Facts carry validity intervals, so a belief that is later superseded (for example, that an exploit primitive is viable) stays queryable for why it was once believed, instead of being deleted outright.
• The author frames the design as a two-part compiler, an LLM front-end extracting structured facts from messy input and Lemmalog's deterministic engine maintaining everything derived from them, but the retrieved text ends before he answers whether the approach measurably improves LLM performance.

Why it matters

LLM agents used for hours-long investigations tend to accumulate stale or contradictory facts under typical store-and-retrieve memory, and telling the model a fact is wrong does not reliably clear out what it inferred from that fact. Reframing agent memory as program analysis, facts plus rules plus fixed-point evaluation, lets an update to one observation automatically invalidate only what actually depended on it, using techniques that already exist in that field, rather than hoping the model notices during a retrieval-based prompt reconstruction.

Who it affects

The author built Lemmalog for his own LLM-driven vulnerability research, long codebase investigations and attack-surface exploration done with agents, but the technique generalizes to any long-running agent task where facts get established, revised or invalidated over time, and to anyone building agent memory who currently relies purely on embedding-based retrieval.

How to use it

The post describes the design in detail, a Datalog-style database split between an LLM front-end that extracts structured facts from natural language, source code and debugger output, and a deterministic engine that applies rules, tracks provenance and maintains validity intervals, but the retrieved text gives no release date, license, repository link or availability, so there is nothing yet to install or point code at.

How solid is it

The description is detailed and internally consistent, with worked examples for derivation, retraction and provenance, but every concrete figure in the post, the two-hour gap and the five dependent conclusions, is an illustrative hypothetical rather than measured data from a real investigation. The retrieved text cuts off exactly at the author's own question, "Does it actually make LLMs better?", before any benchmark or before-and-after comparison is given.

Risks and caveats

The author states plainly that provenance tracking does not stop an LLM from hallucinating during the fact-extraction step itself, only that it makes an unsupported claim harder to silently persist afterward, so the fuzzy front-end that turns messy input into structured facts remains an unaddressed source of error in the design as described.

“It does not inherently know that the statement was disproven two hours later, or that five other conclusions depended on it and should therefore no longer be considered valid.”

— the post's author, on pwning.systems