Oxide publishes design for its rack-level key hierarchy
Oxide Computer has published RFD 301, a design document ('Request for Discussion,' the company's public engineering-proposal format) that works out a rack-level key hierarchy for protecting the secrets stored inside an Oxide server rack. The stated goal is that an attacker who walks off with only a subset of a rack's sleds or drives should not be able to recover any useful information. Oxide's existing building blocks are a Trust Quorum (specified in a separate document, RFD 238) built on Shamir secret sharing: a single rack-level secret is split by a dealer process into N unique key shares and distributed over encrypted point-to-point "sprockets" sessions to each sled's bootstrap agent, along with the platform identities of the other quorum members so each sled can verify who it is talking to. Any sled can then reconstruct the rack secret once it collects K of the N shares from other members. As the document puts it, without obtaining K shares, no information about the rack secret can be learned.
Today those individual key shares sit unencrypted on each sled's M.2 drive, so an attacker needs to physically steal at least K such drives to rebuild the rack secret, which Oxide calls infeasible without significant time and disruption during physical access. A planned future step will "seal" the shares behind the sled's root of trust so they only decrypt at boot; that would raise the bar to stealing K whole sleds and being able to boot them, and the document notes that the physical weight of K sleds alone makes this prohibitive for a casual attacker. This sealing has not been built yet, it is described only as a future plan.
From the rack secret, Oxide derives keys for the data it actually needs to protect: everything ends up in files or databases on each server's U.2 drives. Rather than use hardware-based full-disk encryption, which the team says it distrusts because of unclear vendor implementations and the complexity of managing keys across multiple drive vendors, Oxide encrypts almost an entire ZFS zpool at the dataset level, carving out an exception for the Crucible storage service, which already encrypts its own data. Each U.2 drive gets its own key, so that compromising one drive's key does not expose any other drive. The RFD notes that ZFS's own key rotation, handled inside the kernel module, lets that per-drive key change without having to re-encrypt the whole dataset.
The document weighs two ways to hand keys down the hierarchy: deriving a child key from a parent (nothing needs to be stored, but every downstream key changes whenever the parent does) versus wrapping, meaning encrypting one key with another (the wrapper key can rotate without forcing child keys to change, at the cost of having to store, and possibly replicate, the wrapped key). Oxide's stated design goal is to derive rather than wrap wherever it can. On reconfiguration of the trust quorum, meaning sleds are added or removed, the system always generates a brand-new rack secret rather than reusing the old one with new shares, specifically so that a sled that once held the rack secret and was later removed for being compromised cannot decrypt any data produced after its removal.
The RFD then lays out the hard constraints this creates for the reconfiguration protocol, which runs as a two-phase commit (detailed separately in RFD 238). Sleds that belong to both the old and new quorum groups need to hold both the old and new rack secrets at the same time so they can re-key every drive's ZFS wrapper key. Because not every sled learns of a new configuration simultaneously, and because a reconfiguration can have "false starts" where a proposal is distributed but never committed, the new key shares have to go out during the protocol's prepare message rather than at commit time, and a sled cannot change a drive's wrapper key until its new rack secret is actually committed.
Key facts
- RFD 301 builds a key hierarchy rooted in a shared "rack secret" split by Shamir secret sharing across a trust quorum of sleds; a sled needs K of N shares to reconstruct it, and no information about the secret leaks with fewer than K.
- Key shares currently sit unencrypted on each sled's M.2 drive, so an attacker must physically steal at least K drives; a planned future step will seal shares behind the root of trust so only booting K whole sleds would work instead.
- Data at rest on each U.2 drive is protected with per-disk ZFS native encryption rather than hardware full-disk encryption, which the team says it distrusts over vendor-implementation and key-management complexity.
- Every trust-quorum reconfiguration or key-share rotation generates a brand-new rack secret, so a sled removed for compromise cannot decrypt data produced after its removal.
- Sleds in both the old and new quorum must hold both rack secrets at once to re-key every drive, and new key shares must go out during the two-phase-commit's prepare step rather than at commit time.
Why it matters
The RFD tackles a concrete physical threat: someone walking off with a handful of drives or server sleds from a data center rack. Oxide's answer ties data-at-rest encryption to the presence of enough of the rack's own hardware, via a Shamir-secret-sharing trust quorum, rather than to a single master key or to vendor hardware full-disk encryption, which the document says the team distrusts because of unclear implementations and the complexity of managing keys across multiple drive vendors.
Who it affects
Directly, this is aimed at Oxide's own engineers designing and reviewing the rack's security architecture, and at anyone operating or evaluating an Oxide rack who needs to know what happens to encrypted data if hardware is stolen or a quorum sled is compromised. More broadly, it is a public reference for security engineers and researchers working on distributed key management, Shamir secret sharing or hardware-rooted trust systems, since Oxide publishes its RFDs openly.
How to use it
RFD 301 is a public design document at rfd.shared.oxide.computer/rfd/0301, part of Oxide's open 'Request for Discussion' process; it is not a product and carries no price or license. It is useful as a worked reference for teams building similar systems: it walks through the concrete trade-off between deriving keys from a parent secret, which never has to be stored but changes every downstream key when the parent rotates, and wrapping keys, which can be rotated independently but must be stored and replicated, and it explains how ZFS's native encryption lets a per-drive key rotate without re-encrypting the whole dataset.
How solid is it
This is a first-person engineering design document from Oxide Computer, not an independent audit or a finished specification: it explicitly frames itself as working through open questions about the key hierarchy, and its own diagrams use K = 2 purely as a simplified example rather than a stated production value. Sealing key shares behind the root of trust, which would raise the bar from stealing K drives to stealing and booting K whole sleds, is described only as a future plan, not something already built. The extracted text carries no author, team or publication date, but it also cuts off mid-sentence, so that absence may be an artifact of extraction rather than something missing from the RFD itself.
Risks and caveats
The stronger protection the RFD describes, requiring an attacker to steal and boot whole sleds rather than just drives, depends on sealing via the root of trust, which has not shipped; until it does, the protection in place is against stealing individual M.2 drives. The design also depends on ZFS encryption and on Crucible keys currently stored in CockroachDB, and the document notes that a proper key-management system for certificates and authentication tokens is still planned for after the product's initial release rather than in place today. The reconfiguration protocol itself is presented as an area still being worked through, including how sleds that have not yet learned of a new configuration are handled mid-commit.
“Importantly, without obtaining K shares, no information about the rack secret can be learned.”
— RFD 301, Oxide Computer