Glonce

Oxide ships Kubernetes integrations shaped by customer needs

infrastructureThe Glonce editorsHacker News🔥55Glonce rating55Impact35Novelty40Relevance75Surprise25Players45Credibility85
Oxide ships Kubernetes integrations shaped by customer needs

In late 2024, an engineer joined Oxide as the company's first Solutions Software Engineer, tasked with making it easier to deploy and operate Kubernetes on Oxide. Customers and prospects were already eager to run Kubernetes on the platform, but Oxide had no supported integrations to help them do it. In the first week, the new hire was handed two things to start from: a pull request a customer had already submitted for a Rancher node driver, and an early draft of an internal document, RFD 493, titled 'Initial Kubernetes Integrations.' Rather than designing integrations in the abstract, the team then followed the problems customers actually hit as they moved from provisioning clusters to operating workloads. Different provisioning workflows led to Rancher, Omni and Cluster API; running clusters required reconciling them with Kubernetes; exposing applications revealed networking gaps; and stateful workloads exposed storage constraints that the post names as platform work still ahead, not something already solved.

No single provisioning approach fit every customer, so Oxide ended up publishing three separate integrations. The first was a Rancher node driver, a plugin that teaches Rancher how to create and manage virtual machines on a given infrastructure platform: after learning how Rancher and node drivers work from scratch, the engineer tested and merged the customer's pull request, added CI/CD and documentation, and published it as Oxide's first Kubernetes integration, with a customer already running it in production. The second was an infrastructure provider for Sidero Labs' Omni, which provisions Kubernetes clusters running Talos Linux. Oxide partnered with Sidero Labs and built the provider in seven weeks to have it ready for a joint Oxide+Sidero event at KubeCon North America 2025. That work surfaced several Omni and Talos Linux issues, filed in siderolabs/omni#1633. Sidero Labs' team was eager to help, which the post calls a reminder of Oxide's own internal document on partnership, RFD 68, 'Partnership as Shared Values.' The most memorable bug, siderolabs/talos#11948, was that Talos's filesystem probe only tried to read an ISO 9660 superblock from the NoCloud configuration disk, while Oxide's cloud-init user-data actually uses a FAT12 filesystem. When the ISO 9660 read failed, the probe stopped instead of also trying formats such as VFAT or MS-DOS, so Talos never read Oxide's user-data and never joined the instance to Omni. The real fix would not ship in time for KubeCon, so the interim workaround was to pad the user-data with comments until it grows large enough to be read via an ISO 9660 superblock.

The third integration was a provider for Kubernetes Cluster API (CAPI), which lets operators declaratively create, scale, upgrade and delete clusters through Kubernetes custom resources without a third-party platform like Rancher or Omni. Oxide had wanted a CAPI provider since the first draft of RFD 493, but deferred it because customer demand and engineering capacity did not yet justify the investment. Once both grew, two teammates, Josh and Brandon, took ownership of the work and released Cluster API Provider Oxide, or CAPOx, giving customers a Kubernetes-native way to provision clusters on Oxide. CAPOx also exercises Oxide's other integrations end to end: the Kubernetes Image Builder uses an Oxide Packer plugin to build CAPI-ready VM images, and clusters provisioned with CAPOx use a separately installed Oxide cloud controller manager (CCM) at runtime.

Provisioning integrations create and manage Oxide instances, but on their own they do not reconcile those instances with Kubernetes Node objects; without that reconciliation, a cluster could not reliably tell whether an unreachable node was merely offline or its backing Oxide instance had actually been deleted. Oxide closed that gap by building the CCM, the standard Kubernetes extension point for connecting cluster state to an infrastructure provider's API without adding provider-specific code to Kubernetes itself. Its node controller keeps Kubernetes Node objects synchronized with their backing Oxide instances, recording details like instance IDs and network addresses and reporting whether each instance is running, shut down or gone, so Kubernetes can safely initialize and remove nodes. The CCM itself does not create instances or provision clusters; that stays the job of the Rancher, Omni and CAPOx integrations. What it gives Oxide instead is a durable, shared extension point inside every cluster: new infrastructure-aware controllers can be added to the CCM later, rather than updating every provisioning integration separately.

That same CCM architecture includes a service controller for Kubernetes LoadBalancer Service objects, which is how Oxide addressed the next gap customers hit: exposing applications. Oxide has no native load balancer, but it does have floating IPs, addresses from a rack's external IP pools that can be attached to and detached from instances to make them reachable from outside their VPCs. The service controller uses a floating IP to satisfy a LoadBalancer service: the floating IP delivers traffic to a single Kubernetes node, and the Kubernetes Service dataplane then distributes that traffic to the right pods. Floating IPs are transparent to the guest instance in two ways: Oxide translates an inbound packet's destination address to the instance's internal IP before delivery, and the instance itself has no network interface configured with the floating IP. Because of that translation, the Service dataplane has to treat the node's internal IP, not the floating IP, as the real frontend address, so the service controller publishes two entries in a LoadBalancer service's status.loadBalancer.ingress: the attached floating IP in Proxy mode, and the node's internal IP in VIP mode. In the post's worked example, a client request to floating IP 45.154.216.233 on port 80 is translated to internal IP 172.30.0.5 on port 80 before the Kubernetes Service dataplane selects an endpoint and delivers the traffic to a pod.

Key facts

  • Oxide published three Kubernetes provisioning integrations: a Rancher node driver, an Omni infrastructure provider built with Sidero Labs, and a Cluster API provider called CAPOx.
  • The Omni provider was built in seven weeks so it would be ready for a joint Oxide+Sidero showcase at KubeCon North America 2025.
  • A Talos Linux filesystem-probe bug, tracked as siderolabs/talos#11948, kept Talos from reading Oxide's FAT12 cloud-init user-data because it only tried an ISO 9660 superblock; the interim fix was to pad the user-data with comments until it reads as ISO 9660.
  • Oxide built a cloud controller manager (CCM) to keep Kubernetes Node objects synced with Oxide instances and to back LoadBalancer Service objects with Oxide's floating IPs, since Oxide has no native load balancer.
  • Because floating-IP traffic is address-translated before it reaches a node, each LoadBalancer service gets two status entries: the floating IP in Proxy mode and the node's internal IP in VIP mode.

Why it matters

The post matters less as an announcement and more as an account of how Oxide's Kubernetes support took shape: not from a fixed roadmap but from following, in order, whatever gap customers hit next, provisioning first, then keeping Kubernetes Node objects in sync with real instances, then exposing services externally. That sequence produced concrete decisions that still hold: three separate provisioning paths because no single one fit every customer, a shared cloud controller manager built as a long-term extension point rather than one-off glue code, and a load-balancer design built on floating IPs because Oxide has no native load balancer. It also shows Oxide reaching outside its own codebase, filing and helping fix bugs in a partner's software, Sidero Labs' Talos Linux, rather than only building an adapter around it.

Who it affects

The direct audience is anyone running or evaluating Kubernetes on Oxide: Rancher shops, Omni or Talos Linux shops, and teams that want a Kubernetes-native workflow through Cluster API without a third-party platform. It also affects Sidero Labs and its Omni and Talos Linux users more broadly, since the collaboration surfaced and is fixing bugs in their software, tracked in siderolabs/omni#1633 and siderolabs/talos#11948, not only in Oxide's own integration code. Inside Oxide, it credits specific people behind the work: the post's author, hired as the company's first Solutions Software Engineer, and teammates Josh and Brandon, who built and released the Cluster API provider.

How to use it

The post points to three separate paths, each with its own guide referenced in the text: a Rancher guide for Rancher shops using the Oxide node driver, an Omni guide for Omni or Talos Linux shops using the Oxide infrastructure provider, and a Cluster API guide for teams provisioning through CAPOx. Clusters built through any of these paths also pick up the separately installed Oxide cloud controller manager, which syncs Kubernetes Node objects with Oxide instances and backs LoadBalancer Service objects with Oxide's floating IPs; CAPOx additionally uses the Kubernetes Image Builder with an Oxide Packer plugin to produce CAPI-ready VM images. The source gives no pricing, licensing terms, or version requirements for any of it.

How solid is it

This is a first-person account from inside Oxide's engineering organization, not third-party reporting, but it is unusually specific for a vendor blog post: it names exact GitHub issues, siderolabs/omni#1633 and siderolabs/talos#11948, explains the precise filesystem mismatch behind the Talos bug, FAT12 versus ISO 9660, gives a worked example with concrete IP addresses, and credits named colleagues, Josh and Brandon, rather than speaking only in a corporate we. What it does not do is name the customers involved, give adoption or scale numbers, cite software version numbers, or give exact dates beyond relative markers like late 2024 and seven weeks before KubeCon. Read it as a credible engineering case study, not as independently verified usage data.

Risks and caveats

No customer is named and no adoption numbers are given, so there is no way to gauge from the source how widely any of these integrations are actually used. The real fix for the Talos filesystem-probe bug had not shipped in time for KubeCon North America 2025 according to the post, which describes only the interim workaround, padding cloud-init user-data with comments so it is read via the wrong superblock format, and does not say whether the underlying probe was fixed afterward. Storage for stateful workloads is named only as an open problem, platform work still ahead, so it should not be assumed to be covered by anything described here. As a company-authored post about the company's own integration work, it also naturally presents that work in a favorable light.

“The workaround right now is to pad the user-data with comments to increase its size enough that it uses an ISO 9660 superblock.”

— the post's author, Oxide's first Solutions Software Engineer