Cloudflare reclaims 100TB of RAM by shrinking consistent hashing

Cloudflare's Performance team investigated a ticket filed by an engineer named Ivan, who found excessive memory usage from pingora-ketama, the open-source library that handles consistent hashing inside Cloudflare's internal load balancer, the Pingora Backend Router (PBR). Consistent hashing maps servers and tasks onto the same numeric output space so that requests can be routed to the right server, and added or removed without reshuffling everything; Cloudflare uses it to route cacheable requests by URL so each file is cached in only one place per data center. The problem is that hashing spreads load unevenly: in a worked example with 100 servers, giving each server just one hash point produces a coefficient of variation (a measure of the error margin) of about 99%. The standard fix is to give each server many hash points instead of one; NGINX hardcodes 160 points per server by default, and Pingora inherited the same default, which brings that same 100-server example's error margin down to about 8%.

Zaidoon spotted that the struct Cloudflare used to store each hash point in memory, a hash value plus an index, was taking up more space than it needed to. Simply shrinking the index field's type did nothing, because Rust's alignment rules force a struct's size to be a multiple of its largest field, so the struct still occupied 8 bytes. Repacking the hash and index into a raw 6-byte array with accessor functions, rather than using the riskier #[repr(packed)] attribute, avoided the alignment padding and cut the memory used for consistent hashing by 25%.

The team then worked out an exact formula for how the standard deviation of a server's assigned range shrinks as the number of hash points per server grows, something existing sources apparently only approximated. That derivation showed the number of hashes generated per server could be cut by 90% without introducing appreciable error. The one caveat is that Cloudflare's hashes are 32-bit numbers, not points on a truly continuous ring, so collisions become possible, and the probability of a collision rises quickly once the hash count per server gets large; a simulation for data centers with 2048 servers showed the resulting error rate climbing somewhere between 10,000 and 100,000 hashes per server, well above the ranges Cloudflare actually uses. Between the struct repacking and the reduced hash count, the PBR memory work reclaimed more than 100TB of RAM globally, on top of a separate 100TB reduction Cloudflare's DNS team had already made the previous month. The post's account of how the hash-count change was rolled out to production is cut off mid-sentence in the material reviewed here.

Key facts

A ticket filed by engineer Ivan flagged excessive memory usage from pingora-ketama, the consistent-hashing library behind Cloudflare's internal load balancer, the Pingora Backend Router (PBR).
Repacking the internal Point struct (a hash plus an index) from an alignment-padded 8-byte layout into a packed 6-byte byte array cut memory used for consistent hashing by 25%.
In a 100-server example, moving from one hash point per server to the NGINX/Pingora default of 160 dropped the coefficient of variation, an error margin, from about 99% to about 8%.
A newly derived exact formula for standard deviation with multiple hashes per server showed the hash count per server could be cut by 90% without appreciable error, while 32-bit hash collisions only become a measurable problem between roughly 10,000 and 100,000 hashes per server in a 2048-server simulation.
Combined, the PBR memory work reclaimed more than 100TB of RAM globally, on top of a separate 100TB reduction Cloudflare's DNS team made the previous month.

Why it matters

Cloudflare runs thousands of servers worldwide with every service packed onto every node, so a per-instance memory saving multiplies across the whole fleet. Here a single library, pingora-ketama, was quietly overusing memory in the company's internal load-balancing service, the Pingora Backend Router; fixing it, on top of a separate 100TB cut the DNS team made a month earlier, freed over 100TB of RAM without adding hardware.

Who it affects

Directly, Cloudflare's own infrastructure and performance teams, and any team at the company competing for shared RAM and CPU across the fleet. More broadly, it is relevant to engineers who run consistent hashing at scale, since Pingora and pingora-ketama are open source and NGINX ships the same 160-hash-per-server default that Cloudflare started from.

How to use it

Two concrete, reusable techniques come out of this: pack small fixed-size fields into a raw byte array with getter functions instead of a naively typed struct, since Rust's alignment rules otherwise round a structure's size up to a multiple of its largest field; and, before assuming a hash-per-server count is load-bearing, derive the exact relationship between hash count and distribution error rather than relying on a hardcoded default like NGINX's 160.

How solid is it

The account comes from Cloudflare's own engineering blog and includes a worked numeric example, a named contributor (Zaidoon) for the struct change, and a stated methodology, deriving an exact standard-deviation formula rather than using the approximations the team says most sources rely on. The concrete before-and-after figures given are the 25% memory reduction from the struct repack and the 8% versus 99% error-margin comparison; the text available here cuts off mid-sentence before describing how the hash-count reduction was actually rolled out and validated in production.

Risks and caveats

The post does not state how the total 100TB-plus PBR saving splits between the 25% struct-level reduction and the separate 90% cut in hash count per server, so the two cannot be added independently. No calendar date is given for either change; the DNS team's 100TB reduction is dated only as "last month" relative to this post. The 90% hash-count cut relies on 32-bit hashes staying below the range where collisions start increasing error, which the post's own simulation puts at roughly 10,000 to 100,000 hashes per server for a 2048-server data center, well above the counts actually used here, but a caveat worth carrying forward for anyone applying the same cut at a different scale.