TigerBeetle tests replica internals with protocol-aware DST
TigerBeetle, the distributed financial-transactions database, published a blog post (with a companion talk) describing protocol-aware deterministic simulation testing (DST), a way of checking a distributed system's correctness from the inside rather than only from the outside.
The post starts from two kinds of invariants any distributed system must uphold: safety (nothing bad ever happens, such as two nodes returning different answers to the same request) and liveness (something good eventually happens, such as a request eventually getting a response). TigerBeetle uses Viewstamped Replication (VSR) for consensus. Its liveness invariant is staying responsive as long as a majority of replicas are online; as an illustration, in a 3-replica cluster the majority is 2, so the system tolerates up to 1 fault. Its safety invariant is strict serializability, the strongest database isolation level, meaning that if one operation finishes before another starts, the database's recorded order must reflect that.
Until now, TigerBeetle checked these invariants mainly with black-box methods: Jepsen-style generative testing that injects faults and probes the system only through its user-visible API. Deterministic hypervisors such as Antithesis are another example of that same black-box category, cited in the post as an illustration rather than a method TigerBeetle says it ran. The post quotes TigerBeetle's own prior Jepsen evaluation: 'Integrating Viewstamped Replication with flexible quorums and protocol-aware recovery does not appear to have compromised the key invariant of Strong Serializability.' TigerBeetle argues that outside-in testing alone is not enough for what it calls foundational infrastructure, because it cannot see invariants that never surface at the API boundary.
TigerBeetle is built for both logical determinism (all database code executes deterministically, with no multithreaded concurrency in the control plane, as in FoundationDB) and physical determinism (replicas converge to a byte-by-byte identical on-disk state). That combination lets the team run the real consensus and storage code inside a simulator called the VOPR, on a single machine, with simulated time sped up by orders of magnitude, so scenarios that would take months in production surface in minutes and any bug found can be reproduced deterministically.
Protocol-aware DST extends the VOPR to inspect each replica's internal consensus and storage state directly. For consensus safety, production already crashes a backup replica if its write-ahead log diverges from the primary's, downgrading a safety violation to an availability one; protocol-aware DST goes further, asserting on every single committed request that its checksum matches across every replica that has committed it, rather than only at periodic commit-message checkpoints. For storage safety, production checks a lighter-weight checkpoint_id checksum and crashes a diverging backup; protocol-aware DST additionally checksums the Manifest, the index over each replica's on-disk LSM tree, across every level and every table, asserting that the tree structure is completely consistent across replicas. The source text available for this story cuts off mid-sentence while describing a further physical storage check, so the post's remaining content is not covered here.
Key facts
- TigerBeetle's deterministic simulator, the VOPR, is now protocol-aware: it checks safety and liveness invariants inside each replica's consensus and storage state, not just through the database's external API.
- TigerBeetle's safety invariant is strict serializability and its liveness invariant is staying responsive while a majority of replicas are online; in a 3-replica cluster the majority is 2, so it tolerates up to 1 fault.
- A prior Jepsen evaluation, quoted in the post, found that integrating VSR with flexible quorums and protocol-aware recovery 'does not appear to have compromised' Strong Serializability.
- For consensus safety, protocol-aware DST asserts that a committed request's checksum matches across every replica each time it commits, going deeper than the production check that only fires on periodic commit messages.
- For storage safety, protocol-aware DST checksums the Manifest metadata of every LSM-tree table across all levels to assert byte-for-byte structural consistency across replicas, beyond the lighter checkpoint_id check used in production.
Why it matters
Distributed systems are hard to verify because the state space of possible interleavings across machines is vast, and black-box tools like Jepsen and deterministic hypervisors like Antithesis only probe a system through its external, user-visible API. TigerBeetle argues that for foundational infrastructure, such as a database that financial applications build on top of, that outside-in view is not enough: invariants can be violated deep inside a replica's consensus or storage logic without ever surfacing at the API boundary. Protocol-aware DST closes that gap by giving the simulator direct visibility into each replica's internal state.
Who it affects
This is aimed at engineers building or evaluating distributed, consensus-based systems, particularly ones like TigerBeetle where financial applications depend on strict serializability guarantees. It is a description of TigerBeetle's own internal testing methodology, not a feature or product change for TigerBeetle's users.
How to use it
There is nothing to install or buy here: it is an engineering methodology described in a blog post and an accompanying talk, running inside TigerBeetle's existing deterministic simulator, the VOPR. Teams building their own deterministic distributed systems could apply the same pattern, checksumming committed requests and on-disk index structures across replicas inside a simulator, rather than relying only on black-box fault injection from outside the API.
How solid is it
TigerBeetle already has a Jepsen evaluation, quoted in the post, that found no evidence of the Strong Serializability invariant being compromised. The same categories of check described for protocol-aware DST, write-ahead-log checksum matching and checkpoint_id validation, also run live in production, where a divergence causes TigerBeetle to crash the affected backup replica and downgrade a safety violation to an availability one. Protocol-aware DST runs equivalent checks more exhaustively inside the simulator, asserting on every commit and checksumming the full Manifest across all LSM levels rather than sampling at commit-message boundaries. All of this is TigerBeetle's own account of its own methodology; the source text cuts off before the article's concluding section, so what follows is not covered here.
Risks and caveats
The description comes entirely from TigerBeetle about its own system, so it is a vendor's self-assessment rather than independent verification. Protocol-aware DST does not replace black-box testing; the post frames it as an addition layered on top of the Jepsen-style testing TigerBeetle already runs, the same black-box category that also includes deterministic hypervisors such as Antithesis. The approach also depends on a specific design decision, logical and physical determinism built into the database from the ground up, that most existing distributed systems were not built with and cannot easily retrofit.
“Integrating Viewstamped Replication with flexible quorums and protocol-aware recovery does not appear to have compromised the key invariant of Strong Serializability.”
— TigerBeetle's Jepsen evaluation report, quoted in the post