OpenAI scales Habitat storage to serve 1 billion ChatGPT users

OpenAI engineers Jon Lee, Chaomin Yu and Ben Ries, all Members of Technical Staff, write the first of a two-part account of how they scaled Habitat, the internal storage platform behind every OpenAI product, from someone logging in, to checking Codex settings, to starting a new ChatGPT conversation. Habitat now handles more than 70 million requests every second, supports products used by over 1 billion people each week across almost 40 geographic regions, and serves more than 500 petabytes of data. That scale came from relentless growth: OpenAI's usage grew more than 10x year over year for three straight years, which the authors say forced a running series of tactical decisions rather than one stable plan built for a fixed target size. Habitat itself started small, launching at DevDay 2023 as a simple Python client-side library that let product engineers store and retrieve data from a single database, Azure Cosmos DB, without handling schema lookup, routing, authorization, encryption or connection pooling themselves. It caught on quickly across OpenAI's product teams, even without any central push away from self-serve Postgres and Cosmos DB, and teams kept extending the shared library themselves with features like client-side caching, compression and encryption.

That client-side model hit its limits by the middle of 2025: as the library grew more complex and OpenAI's number of services increased, backward-compatible protocol changes became impractical. The authors illustrate the cost with one migration: to shrink the blast radius of a regional outage, they wanted to move critical data sets onto Azure Cosmos DB accounts distributed across regions. That required adding routing logic to the client behind a feature flag, coordinating a rollout across dozens of services (which took days), adding shadowing to confirm the sharding logic was correct (another couple of days), and fixing a bug that surfaced along the way (another couple of days). When the flag was finally ready to flip, one team rolled its service back to a previously buggy client for unrelated reasons, and caused the very outage the team had worked so hard to avoid. That experience, and the general brittleness of coordinating client-library changes across so many services, is why OpenAI pulled Habitat out of the client and into its own standalone service: a single point of control for deployments, observability and platform improvements, and a single chokepoint to enforce access control, run audit logging, and protect user data from external, internal and even agent actors.

Turning Habitat into a service meant keeping Python at its core, even though the team knew a high-throughput Python service would add network latency and CPU and memory costs a client-side library never had, and that Python's inefficiencies would not hold up at 100x scale, making an eventual rewrite 'almost certain.' They call this a deliberate 'strategic incursion of technical debt': the immediate goal was unblocking product teams and reaching platform stability, not saving on compute cost. Part of that bet was on OpenAI's own coding models: the team wagered that by the time a full migration off Python became necessary, tools like Codex and GPT would make that rewrite achievable, a bet the authors say has since proved correct.

Running that Python service at Habitat's scale means fighting tail latency non-stop. An average user request can trigger hundreds of separate database calls, and the slowest one is the one the user actually feels, so the team's central challenge became managing those tail latencies rather than average ones. Asyncio helps Python juggle I/O-bound work concurrently, but it does nothing for CPU parallelism, since Python's GIL (its Global Interpreter Lock) still serializes CPU-heavy work such as routing, compression, encryption, checksumming, health checks, request shadowing and hedging. Early traces showed requests stalling not because the downstream database was slow, but because the coroutine handling the response was waiting to be rescheduled, producing scheduling jitter of up to hundreds of milliseconds and, in some edge cases, several seconds under high load. The team's response was operational: watch how busy the asyncio loop itself is, not just standard CPU, memory, network and disk metrics, and keep each process handling only a small number of concurrent requests, scaling out through many more worker processes instead of fewer busier ones.

One concrete cause of that scheduling delay, found through live CPU profiling after the service's initial launch, was Statsig, the feature-flag tool Habitat used to manage configuration and run tests like A/B experiments. By default, Statsig polled for a refreshed configuration every minute with no jitter, and that configuration bundled every production rule for every service, not just the ones a given pod needed. An earlier architectural choice to run up to 8 Python worker processes per pod, meant to push CPU usage higher and cut latency, meant that every pod, every minute, hit a moment where all of its workers stalled in-flight requests to parse that one giant configuration file. Once profiling pinned this down, the fix was straightforward: ship a smaller, targeted config per service, lengthen the refresh interval, and add jitter to that and other periodic background tasks.

A second failure mode came from connection pooling. With client-side pooling, a single client process making many concurrent requests might open only a handful of connections to Habitat's servers, sending its whole load onto a handful of processes; before the team fixed load balancing, some 'tail' processes were absorbing 5 to 10 times the concurrent requests of an average one. They found this through an incident where, even after they stopped the client that had overloaded part of the service, a subset of processes stayed degraded well past the burst, taking on more and more traffic until the team restarted them by hand, a pattern some teammates recognized from prior experience as 'metastable failure.' Capping the maximum time a connection could stay in reuse limited the degradation and confirmed the connection pool was the cause; further investigation traced it to a default in Python's aiohttp library, whose TCPConnector reuses the most recently returned connection first (LIFO), a default the authors call normally reasonable because it lets extra connections opened for bursty traffic idle out naturally. The post does not go on, in the material available here, to state exactly how the connection-pooling problem was ultimately handled at Habitat's scale. The authors flag a second post to come, covering multi-tenancy reliability at scale, their layered strategy for read performance, and how they scaled the partnership with Azure Cosmos DB, none of which this first post covers.

Key facts

Habitat now handles more than 70 million requests per second and serves over 500 petabytes of data for products, including ChatGPT, used by more than 1 billion people a week, across almost 40 regions, up from a simple Python library OpenAI launched at DevDay 2023.
OpenAI's usage has grown more than 10x year over year for three straight years, which the authors say forced a series of tactical fixes rather than one stable plan for a fixed target scale.
By mid-2025 the client-side library had hit its limits, so OpenAI pulled Habitat into its own standalone service after a routine migration, coordinated across dozens of services, still triggered an outage when one team rolled its client back for unrelated reasons.
The team kept Habitat's new service in Python despite the overhead, calling it a deliberate 'strategic incursion of technical debt,' betting that improvements in OpenAI's own Codex and GPT models would make an eventual rewrite achievable, a bet the authors say has since proved correct.
Engineers traced tail-latency spikes to two causes: the feature-flag tool Statsig reparsing a giant config every minute across up to 8 Python processes per pod, and a default in Python's aiohttp library that let a handful of server processes absorb 5 to 10 times the average load during traffic bursts.

Why it matters

This is less an announcement than a rare, detailed look at what it actually costs to keep AI products fast at OpenAI's scale. Habitat is the layer that decides whether logging in, opening ChatGPT, or checking Codex settings feels instant or sluggish, and the post shows that problems at this scale are not solved by better models but by unglamorous systems engineering: event-loop scheduling, connection-pool defaults, and feature-flag polling intervals. It also documents a real strategic choice, staying on Python on purpose, betting that OpenAI's own coding tools would later make the expensive rewrite achievable, which the authors say has since proved correct.

Who it affects

Directly, OpenAI's own platform and product engineering teams, who build on Habitat rather than talking to Azure Cosmos DB or other storage directly. Indirectly, everyone who uses an OpenAI product: the piece frames Habitat as the shared layer behind logging in, ChatGPT conversations, Codex settings, and GPTs, serving over 1 billion people a week. More broadly, engineers anywhere running high-throughput Python services, or using Statsig or aiohttp, since two of the concrete failure modes described, an unjittered feature-flag poll and aiohttp's default connection-reuse order, are defaults many other services share, not something unique to OpenAI's stack.

How to use it

There is no product or price here, but the post reads as an operations checklist. Treat the asyncio event loop's own busyness as a first-class metric alongside CPU, memory, network and disk usage. Prefer many worker processes handling few concurrent requests each over fewer processes handling many, since Python's GIL means a busy process stalls everything queued behind it. Add jitter to any periodic background task, especially config polling, so pods do not all stall at the same moment. And if requests pool onto a handful of processes under bursty load, check the client's connection-reuse policy, aiohttp's LIFO default among them, before assuming the servers themselves are undersized.

How solid is it

This is a first-party account: an official OpenAI engineering blog post carrying three named authors, all Members of Technical Staff, describing their own system, with specific throughput, data-volume and process-count figures rather than vague claims. That also means the numbers are self-reported and not independently audited. The account is candid about its own failure, including an outage caused by the team's own migration process, which supports its credibility, but the text available for this retelling cuts off mid-explanation of the aiohttp connection-reuse problem, before stating what change resolved it. The authors also flag this as part one of a two-part series, with multi-tenancy reliability, read-performance layering and the Azure Cosmos DB partnership explicitly deferred to a second post not covered here.

Risks and caveats

No date or duration is given for the outage caused by the rolled-back client, no exact count backs 'dozens of services', and no exact duration is given for the several rounds that each took 'a couple of days'. The excerpt available for this retelling does not state how the aiohttp connection-reuse issue was ultimately fixed beyond capping the maximum connection-reuse duration as a diagnostic test; it breaks off before reaching a stated resolution. Everything here is OpenAI's own retrospective of its own infrastructure: there is no external benchmark or third party confirming the throughput or scale figures.

“When the average user request results in hundreds of database calls, the slowest database call is the one the user feels.”

— Jon Lee, Chaomin Yu and Ben Ries, OpenAI, in the Habitat engineering blog post