OpenAI's Pando Git backend cuts median push latency nearly 5x
OpenAI has described Pando, a new backend for serving Git repositories that it built for its own monorepo. The reason, per the company: over the past year the size and demands on the monorepo grew at a staggering pace, and Codex made far more concurrent work possible. More engineers' changes mean more branches, pushes, merges, and fetches and clones from development environments, research jobs and CI. The demands compound: one engineer can now have several changes ready for CI at once, and each push can set several jobs fetching the same commit.
Starting around the end of last year, the systems behind the monorepo struggled to keep up. OpenAI reports slow fetches and pushes and failed merges, with reliability at times falling below 90%. It asked whether it had hit a limit of Git itself or of the way it served the monorepo, and chose to keep Git as the foundation and rebuild the backend. The aim was to handle pushes to different branches in parallel and serve the same history to many readers at once. OpenAI also says it wanted to contribute improvements from its copy of Git to upstream, so the wider community benefits.
Pando has two stated goals: horizontally scalable writes, optimized for concurrent pushes to many branches, and maximal read throughput, so clients can download fast enough to saturate their available network bandwidth. OpenAI says it met them while staying compatible with the Git protocol and client, so engineers and agents adopted it in existing workflows. It says Pando was built without interrupting engineering work or changing how the team uses Git, and that it now powers all development in the monorepo. In production, Pando cut median push latency by nearly 5x and delivers nearly a petabyte of Git data per day.
The post compares Pando with two conventional approaches OpenAI evaluated while designing it. Both were modeled; the text does not say they were run in production. The first uses three-phase commit (3PC) across full repository replicas. A pushed packfile is sent to all replicas in parallel, then a coordinator needs a majority to accept the ref update through lock-and-check, pre-commit and commit rounds. Reads are served locally and clients see a consistent view of all references. The cost is that refs are updated one at a time, each push needs multiple round trips, and replica disk performance and communication latency sit on the push path. The second orders updates through a shared write-ahead log (WAL), a manifest in blob storage updated by compare-and-swap. Packfiles upload in parallel, but even pushes to unrelated branches must update the same manifest, so the compare-and-swap round-trip time limits how often a manifest can be published. Batching helps but queues updates and adds latency, a bigger blob store does not remove the bottleneck, and replicas must still read and index packs before serving them.
Pando instead gives up two things most Git systems treat as requirements: repository-wide consistency (a point-in-time snapshot of all refs, which forces a single coordination point) and unique packfiles for every client (which costs CPU and disk I/O per reader). In their place it offers a relaxed, causal consistency model for writes and reuses packfiles for reads. When a client pushes to a branch, Pando stores the incoming packfile as an immutable blob in a blob store. Each time the branch moves, it records a ref version in a metadata store: the new tip, the pack that supplied it, and any earlier ref versions the pack depends on. Both stores are horizontally scalable.
Fetching a commit or ref means replaying a set of ref versions (packfiles) to the client, reconstructing the state of the client that created the version being fetched. Updates to individual refs are serializable, and clients only conflict when pushing to the same ref at the same time. Updates to unrelated branches proceed in parallel. The tradeoff is that a reader looking at several branches can see them at different points in time; Pando does not provide a point-in-time state of all references during Git operations or an ordering of ref updates within a clone or fetch. On reads, the server reuses packfiles across fetches rather than building one per client, so a client may download some objects it already has, but the server can serve the same bytes to many clients with little computation. The post also gives a primer on Git objects, refs and packfiles, and says it covers what comes next and how OpenAI will keep contributing to Git for agent-scale work.
Key facts
- OpenAI built Pando, a new backend for Git repositories, because Codex-driven concurrent work overloaded its monorepo systems; reliability at times fell below 90%, with slow fetches and pushes and failed merges.
- Pando now powers all development in the monorepo, stays compatible with the Git protocol and client, and was rolled out without interrupting engineering work.
- OpenAI reports nearly 5x lower median push latency in production and nearly a petabyte of Git data delivered per day.
- Pando trades repository-wide consistency and per-client packfiles for causal consistency on writes and reused packfiles on reads, with serializable updates per ref.
- OpenAI modeled and rejected two conventional designs, three-phase commit across replicas and a shared write-ahead log manifest, because pushes to unrelated branches still contend.
Why it matters
Agent-driven coding changes the load on source control. OpenAI's account is that Codex turned each engineer into a source of many parallel branches, pushes and CI fetches, and that the Git serving layer, not Git itself, became the bottleneck. The post is a concrete example of a large engineering organisation redesigning its backend around that load. Its central design choice is notable: drop repository-wide consistency and per-client packfiles, two things most Git systems assume, while keeping the Git protocol and client unchanged.
Who it affects
Most directly OpenAI's engineers and agents, who use Pando for all monorepo development. Beyond that, it is relevant to platform and developer-tooling teams that run large Git monorepos or expect agents to push and fetch at high volume, and to Git hosting and tooling maintainers. OpenAI says it wants to contribute client improvements from its copy of Git to upstream so the wider community can use them.
How to use it
The post does not say Pando is open source or available to others, so there is nothing to adopt directly. What readers can take away is the design reasoning: coordinate updates per branch rather than per repository, store pushed packfiles as immutable blobs, record ref versions with their dependencies in a scalable metadata store, and reuse packfiles across fetches. Teams weighing 3PC or a shared WAL manifest can use OpenAI's account of where each design bottlenecks: round trips per push for 3PC, and a single manifest updated by compare-and-swap for the WAL approach.
How solid is it
This is a first-party engineering post from OpenAI, so the figures are the company's own. The nearly 5x median push latency reduction is stated without a named baseline or absolute latencies, and the below-90% reliability figure applies only to unspecified moments before Pando, with no stated measurement method. The petabyte-a-day figure is a current delivery rate. The 3PC and WAL comparisons are of designs OpenAI modeled, and the text gives no benchmark numbers for them. The explanation of the architecture and its consistency tradeoffs is detailed and internally consistent.
Risks and caveats
The consistency tradeoff is the main one, and OpenAI states it plainly: a reader looking at several branches can see them at different points in time, and Pando gives no point-in-time view of all refs and no ordering of ref updates within a clone or fetch. Reused packfiles mean clients may download objects they already have. The design suits OpenAI's workload as described, with many writers on different branches; the post does not show how it would fare elsewhere. No monorepo size, push rate or engineer count is given, so the scale of the gains is hard to judge from outside.
“Instead, Pando provides a relaxed consistency model (causal consistency) to scale writes, and reuses packfiles to scale reads.”
— OpenAI, "Scaling Git at OpenAI"