Cloudflare halves DNS build delay and cuts Zone Builder memory 85%

Cloudflare has described a rework of the Zone Builder, the service that turns DNS changes into data its authoritative DNS servers can use. When the author last wrote about the DNS propagation pipeline, in 2022, the Zone Builder received about 250 DNS record changes per second. Today it receives about 700, nearly three times as many, plus about 100 other events such as zone setting changes. The author says the event count was never the real problem. The problem was the kind of work some events needed and how that work grows with zone size. Some zones now hold more than 20 million records, and Cloudflare expects them to keep growing toward hundreds of millions.

Each event starts a clock. A "build" is the work of reading the current database state, converting it into the format used at the edge, and writing any changes to Quicksilver, Cloudflare's distributed key-value store. The pipeline runs in stages: a customer changes a record through the API or dashboard (or another Cloudflare service changes a zone setting), the change is stored in a database, a Kafka event is produced, the DNS Kafka Aggregator combines events from several topics and sends them to the Zone Builder, and the Zone Builder compares its generated keys with Quicksilver and writes only what changed.

In 2022, Cloudflare added per-record builds, which were 150 times faster than full builds on average, and 4,250 times faster on a test zone with one million records. The obvious next step was to do the same for zone settings. That was harder. A record event says what changed. A zone event can come from many Cloudflare products, may not say why the zone changed, and can alter data shared by several zones with the same name. Doing too much work wastes capacity; doing too little can leave stale or incorrect DNS data at the edge. Universal SSL is one example: toggling it updates metadata and can change CAA records at the zone apex.

The fix has two parts: a targeted build planner that chooses the smallest build Cloudflare can prove is safe, and a new scheduler that combines work for the same zone. Together they cut peak memory by 85%. The extra headroom let Cloudflare safely shorten waits the pipeline has always needed for resilience, which halved the time from a DNS change to its data being written to Quicksilver. Zone setting changes no longer rebuild zones with tens of millions of records.

To make this possible, Cloudflare changed the event producer to include the changed zone fields and settings, with previous and new values when available, and traced each setting through the code that generates DNS output. Not every event is that specific: some database functions only update a timestamp, which can be harmless bookkeeping or can mean data changed elsewhere. Until the caller and its purpose are identified, a timestamp-only update still causes a full build. The same goes for missing details and unknown events. In the author's words, the planner never turns uncertainty into less work.

The planner picks from five build types: resources only (specific DNS names, no metadata change; ordinary A, AAAA, TXT and CNAME changes usually take this path); resources and metadata (nameserver and wildcard changes); apex and metadata (settings such as Universal SSL can use this path); purge (remove a zone that should no longer be published); and full zone (rebuild every record when smaller work cannot be proven safe). The governing rule is that every key a targeted build writes must be byte-for-byte identical to what a full build would write; if that cannot be proven, Cloudflare does a full build. Stale keys are removed as in 2022: record events include deleted records and the previous version of updated ones, and the Zone Builder scans Quicksilver for existing keys under each name it rebuilds and removes any the new build did not write.

There is one deliberate exception. A full build holds every database record and every existing Quicksilver key for the zone in memory at once, and the largest zones do not fit. Above a fixed record-count threshold, the Zone Builder never escalates to a full build; it rebuilds the apex, the metadata and every name identified by the queued events. Cloudflare accepts the tradeoff because a full build of such a zone cannot finish and, while it retries, holds back every other change on its Kafka partition. Zone holds, which stop the same name being added to another account, reduce the risk in many same-name cases. For the rare cases that remain, such as enabling a zone on another network, individual names can be rebuilt manually.

The post also covers data shared across zones. Quicksilver stores each zone's record keys separately but keeps one metadata key shared by every Cloudflare zone with that domain name; it works like a directory of the internal zone IDs the DNS servers may use when answering. Several zone objects can briefly exist for one public domain during onboarding and offboarding, and a priority policy picks a small, bounded set to publish. A TXT change in one zone touches only that zone's record keys. Nameserver and wildcard changes also update the metadata key. At execution time the Zone Builder re-reads the current list, rebuilds the whole key from that fresh view rather than patching an old copy, and writes the triggering zone's records before the metadata that refers to them.

Because a build may wait while more events arrive, planning happens twice. The event creates a conservative work list, and just before execution the Zone Builder reads a fresh publication snapshot: current zone settings, whether this deployment should publish the zone to its network, and which same-name zone IDs belong in the metadata key. Fresh state can turn a targeted build into a full build, turn a build into a purge, cause a build to be skipped if the triggering zone no longer belongs in the metadata key, or make a metadata build reconstruct the shared key from the current candidate set. The fresh read comes from read-only database replicas. Because Kafka delivery and replication move independently, the DNS Kafka Aggregator delays events long enough to cover normal replication lag, and a final database check rejects the read if the replica is still behind.

Finally, record work and full-zone work used to run on different schedulers that could not combine each other's work. Cloudflare replaced them with one unified scheduler: the planner decides what must be built, and the scheduler decides when it can run. If an A record change at www.example.com and a wildcard change at *.example.com arrive before a build starts, the scheduler merges them into one build covering both names and the metadata update. Merging only adds work: if any event requires the complete zone, the combined build becomes a full build, and a purge overrides every rebuild because its desired result is that the zone is no longer published.

Key facts

• Cloudflare says its targeted build planner and unified scheduler together cut Zone Builder peak memory by 85% and halved the time from a DNS change to its data being written to Quicksilver.
• Load has grown from about 250 DNS record changes per second in 2022 to about 700 today, plus about 100 other events; some zones now hold more than 20 million records.
• The planner chooses among five build types, and the safety rule is that a targeted build must write keys byte-for-byte identical to what a full build would; if that cannot be proven, it does a full build.
• Above a fixed record-count threshold, the largest zones never get a full build, because a full build cannot fit in memory; the Zone Builder rebuilds the apex, metadata and the names named in queued events instead.
• Planning happens twice, from the event and again against a fresh database snapshot just before the build, and one scheduler now merges work for the same zone.

Why it matters

DNS changes are only useful once they reach the edge, so the delay between a change and its data landing in Quicksilver is the number that matters to customers. Cloudflare reports halving that delay while cutting peak memory by 85%, at a time when event volume has nearly tripled since 2022 (about 250 to about 700 record changes per second) and zones are heading from 20 million records toward hundreds of millions. The post is also a worked example of a general trade: replacing a simple 'rebuild everything from current state' design with targeted work, and paying for it in the effort of proving each shortcut safe. The author says that proof took most of the work.

Who it affects

Customers with very large zones benefit most: zone setting changes no longer rebuild zones with tens of millions of records. Anyone whose DNS changes pass through Cloudflare's pipeline is affected by the shorter time to reach Quicksilver. Engineers running event-driven pipelines that rebuild derived state from a source of truth, whether with Kafka, read replicas or a key-value store, are the other audience for the design lessons.

How to use it

This is an engineering write-up, not a product or tool you can adopt. The transferable ideas are stated in the post. Make events say what changed, including previous and new values. Classify every setting by what output it can affect. Define correctness against the full rebuild, so a shortcut is allowed only if its output is byte-for-byte identical. Treat uncertainty as a reason to do more work, never less. Plan once from the event and again against fresh state just before execution. Merge queued work for the same zone, and let merging only add work.

How solid is it

The figures are Cloudflare's own measurements, reported by the operator of the system in a first-party engineering post. The 85% reduction is for peak memory and is credited to the planner and scheduler together. The post states the delay improvement only as 'halved'; the before and after times in seconds are not given in the portion of the article available here, and neither is the record-count threshold for skipping full builds. The 2022 baselines (150 times and 4,250 times faster per-record builds) come from the author's earlier post. The available text breaks off partway through the scheduler section, so later material is not covered here.

Risks and caveats

The design accepts a deliberate tradeoff: above the record-count threshold, the Zone Builder never does a full build, so it relies on the apex, metadata and event-identified names being enough. Cloudflare says many of the cases that would otherwise need a full build involve several zones sharing one name, and that zone holds reduce that risk. For the rare cases left, such as enabling a zone on another network, names must be rebuilt manually. A timestamp-only update still triggers a full build until its caller and purpose are identified, so some events still cost full-zone work. The post notes that choosing too little work can leave stale or incorrect DNS data at the edge, which is why so much of the design is about proving safety.

“The planner never turns uncertainty into less work.”

— Cloudflare engineering blog post on DNS targeted zone builds