Glonce

Pinterest traces Ray training crashes to zombie memory cgroups

infrastructureThe Glonce editorsHacker News🔥46Glonce rating46Impact45Novelty35Relevance55Surprise35Players45Credibility75

In early 2025, Pinterest's ML platform team told the Kubernetes platform team (PinCompute) that their Ray based training jobs, which often run for hours on expensive GPU hardware, were crashing intermittently from what their logs showed as loss of network connectivity. The ask was simple: make it stop. It took PinCompute more than three months to find the real cause, and the trail ran through network drivers, CPU profiling and a completely unrelated bug before it ended.

The first clue was that every crash correlated with a reset of the AWS ENA network driver, logged as a transmit queue paused for too long. AWS's own ENA reference docs describe the reset as a rare, self healing event triggered by CPU starvation: if the driver's threads do not get CPU time for a stretch of time (hardcoded to 5 seconds in AWS's ENA kernel driver), the device resets itself. The crashes were costly: some Ray workloads saw their success rate drop by more than 25%, burning GPU hours and slowing model development.

The team's first theory was inefficient memory allocation causing heavy page faulting, so they tried transparent huge pages, swapped in jemalloc, pinned training jobs to specific CPU cores with taskset, and steered network interrupts to other cores. None of it stopped the resets, which struck unpredictably, sometimes minutes into a job and sometimes hours in. Rebooting affected machines did help, but only for about a week before the resets came back.

A second clue deepened the mystery: the resets only hit machines in a single AWS Availability Zone, even though PinCompute's zonal Kubernetes clusters looked identical across zones, same Kubernetes version, same system image. AWS support checked and ruled out a hardware issue on their end.

To find the actual culprit, the team moved from broad perf snapshots, which only showed the expected ML job computation, to mpstat readings of each of the GPU machines' 96 vCPU cores individually. That surfaced a single core, in one case core 39, pegged at 100% system CPU for several seconds at a time, lining up with the network resets. To catch a reset in the act, PinCompute reserved a set of machines, ran training jobs on them, and captured rolling two minute perf snapshots overnight; the next day they used Netflix's Flamescope tool to zoom into the moment of a captured reset. The trace showed kubelet, Kubernetes' lightweight node agent, spiking to about 6.5% of total CPU (versus its usual sub 1% baseline) right before a reset, spending most of that time inside the mem_cgroup_nr_lru_pages system call, meaning it was iterating over the host's memory cgroups.

A post on Oracle's blog about "zombie" memory cgroups pointed the team at the actual number: on a resetting machine, /proc/cgroups reported 68,680 kernel tracked memory cgroups, while a filesystem scan of /sys/fs/cgroup/memory/ found only 240 actually in use. Kubelet was churning through tens of thousands of dead cgroup entries on every pass, and when that churn landed a network thread on the same starved core, the ENA driver reset.

The source of the zombies turned out to be a container nobody meant to run: docker ps showed an amazon/amazon-ecs-agent container that was always only seconds old. Pinterest's GPU nodes used the AWS Deep Learning AMI (Ubuntu 20.04) as their base image, and that AMI sets up the Amazon ECS agent as a default systemd unit, even though Pinterest runs Kubernetes, not ECS. With no ECS cluster to join, the agent failed to start, crashed, and restarted in a loop for days, and each crash left behind more memory cgroup wreckage. Disabling the ECS agent's systemd unit and rebooting all affected machines to purge the accumulated zombies fixed the problem: memory cgroup counts stayed stable and Ray training jobs went back to their expected success rate.

The Availability Zone puzzle turned out to be a coincidence layered on top of a separate, pre-existing bug. The two zones were not actually running identical bootstrap configurations: the unaffected zone's cluster was fetching the same Kubernetes binary from a different URL, and that mismatch caused a metric emitted at the end of the bootstrap script to fail, which marked the whole bootstrap as failed. Because the ECS agent's systemd unit depended on a successful bootstrap, this unrelated failure accidentally prevented the ECS agent from ever starting in that zone, which meant it never built up zombie cgroups there either. PinCompute was already working on fixing that URL mismatch independently, and fixing it would eventually have brought the reset problem to the previously unaffected zone as well.

Key facts

  • The investigation ran more than three months, triggered by Ray training jobs on Pinterest's GPU Kubernetes nodes crashing intermittently, with some workloads seeing their success rate drop by more than 25%.
  • Root cause: a repeatedly crashing Amazon ECS agent, started by default from Pinterest's AWS Deep Learning AMI even though Pinterest runs Kubernetes rather than ECS, left 68,680 kernel tracked memory cgroups behind versus only 240 actually in use.
  • Per core profiling with mpstat and temporal perf snapshots analyzed in Flamescope showed kubelet's CPU use spiking to about 6.5% (from a normal sub 1% baseline) as it iterated the zombie cgroups via mem_cgroup_nr_lru_pages, at times pinning a single core to 100% and starving the ENA network driver past its hardcoded 5 second pause threshold, which forced a device reset.
  • Rebooting affected machines cleared the zombie cgroups and restored stability for about one week before the count built back up and resets resumed, since the ECS agent kept crashing and being restarted.
  • A separate puzzle, that only one AWS Availability Zone showed the problem, turned out to be an unrelated bootstrap bug: a mismatched binary URL made node bootstrap fail there, which incidentally stopped the ECS agent's systemd unit (which depended on a successful bootstrap) from ever starting.

Why it matters

The story is a detailed public account of how a stray default in a cloud vendor's base image, one nobody at Pinterest asked for or noticed, degraded GPU training throughput for months before it was traced. It shows how a low level resource leak (zombie cgroups from a crashing container) can masquerade as a network driver problem, and how far a team had to dig, from high level metrics down to per core CPU profiling and system call level tracing, before the real mechanism became visible.

Who it affects

Directly, Pinterest's own ML platform and PinCompute (Kubernetes platform) teams, whose Ray based training jobs were crashing and burning GPU hours. More broadly, it is relevant to any team running Kubernetes, rather than ECS, on AWS EC2 instances built from the AWS Deep Learning AMI, or on any AWS instance using the ENA network driver, since the same default ECS agent and the same reset behavior would apply to them too.

How to use it

The team's own takeaways: audit what processes and systemd units actually run on your base OS images rather than trusting the defaults; build fleet wide metrics to catch transient, hard to reproduce failures; set up reproducible, isolated environments for iterative debugging; invest in temporal profiling tools, such as continuous perf capture paired with a time travel viewer like Flamescope, rather than relying on one off snapshots; and when two environments look identical but behave differently, assume there is a real configuration difference and keep digging rather than trusting that they match.

How solid is it

Published on Pinterest Engineering's official Medium blog under a byline of ten named engineers, ranging from Staff Software Engineer to Principal Engineer, a Director of Engineering and a Senior Engineering Manager. The account includes command output, log lines, graphs and a walkthrough of the debugging steps in sequence, which supports it as a detailed first hand engineering post rather than a summary.

Risks and caveats

The article gives no total dollar cost or GPU hour count for the compute lost to the crashes, only the qualitative 'significant slowdown' and the more than 25% success rate drop for some use cases. It does not say how many machines, nodes or Ray clusters were affected fleet wide, does not give an exact date for when the fix shipped or the investigation concluded, and does not state whether AWS has since changed the Deep Learning AMI's default of enabling the ECS agent, or whether other AWS customers running the same AMI have hit the same zombie cgroup issue.

“it may take as little as one CPU core to be heavily utilized and block an unlucky network thread that was scheduled onto that core”

— Pinterest engineers, in the Pinterest Engineering blog post