A false smoke alarm nearly flooded Jane Street's NYC datacenter

This is a first-person blog post by a member of Jane Street's systems team, who describes himself and his colleagues as the firm's blue-collar OCaml programmers. The author is not named in the article text. The post gives no date for the incident.

The setup: Jane Street insisted on keeping its machine room close to the trading floor. The author gives two reasons. One was intellectual property, since the firm did not want servers stolen by spies who could learn its trading strategies. The other was technology risk: out-of-control automated trading could always be stopped by running across the hall and pulling power cables. He credits a partner who had served on a nuclear submarine with instilling much of the firm's risk culture. As the firm grew, the machine room became a sizable datacenter with an online UPS, Computer Room Air Conditioner (CRAC) units, refrigerator-sized network switches, a subterranean diesel backup generator and about 10 rows of server cabinets, 10 cabinets deep, all inside 1 New York Plaza, an otherwise normal office skyscraper.

The author lived in a 200 square foot studio, a 9 minute walk from his apartment door to his desk on the trading floor, after earlier commuting an hour or more each way from Queens. One evening, as he was about to get into his apartment building's elevator, the Hong Kong office phoned: the smoke alarm in the NYC datacenter had gone off. Contractors working on a prep room next to the datacenter had kicked up dust while cutting into material, and the dust set off the alarm.

Why that was dangerous: sprinklers over a machine room like this usually are not kept full of water. A separate system detects smoke, and then the pipes fill with pressurized water, but nothing sprays yet. Each sprinkler head has its own heat-sensitive plug that opens only when the air around it gets hot enough. The catch was that the smoke alarm was wired to shut down the CRAC units, to avoid feeding a real fire with circulating air. With cooling off, a room full of servers heats up, and heat is exactly what opens the sprinkler heads. The author's mission was to pull the power cord on every device in the room before the temperature passed 150 F and the sprinklers destroyed everything.

He ran down Water Street, rode the elevator up 33 floors at 1 NYP, used his keycard on the office door and the machine room door, and stepped into what felt like a sauna. He went row by row, cabinet by cabinet, yanking power cables out with both hands. Then he watched the temperature monitor: 141 F, 142, 143, 143.5, 143.6, and it stopped. The room went quiet. He notes the machine room had been kept across the hall so someone could always pull the cords, and he doubts anyone had pictured doing it to save the servers from the sprinklers.

The damage was limited. The major NYC markets were closed for the day, so downtime was minimal and systems were up the next morning. Meetings followed, which he describes as matter-of-fact and level-headed, more like engineering students arguing over what went wrong than a hunt for whose career was over. The outcome: the sprinklers would stay, procedures and surveillance would be tightened, and servers would continue to be hosted in the office building next to the trading floor. He adds that this lasted until Hurricane Sandy, which he calls a story for another day.

The fix he built: the Hong Kong team had caught the problem only by luck, when someone happened to see on a display that server after server showed a climbing temperature condition. No single server was set to alert for being a little hot, because that happens from time to time. But an across-the-board rise on many servers signals a general cooling failure. The next morning he wrote a monitor in OCaml that, every minute, asks every server in each location directly for the temperature from its onboard sensors, takes the average, and if it is elevated, sends an alert to everyone on the systems team to check for datacenter overheat. He says alerting on CRAC shutdown or a dedicated room temperature sensor was possible, but that telemetry was nowhere near as reliable, and unknown unknowns can still hurt, so asking the fleet itself was the most durable option.

Key facts

• Dust from contractors working on a prep room next to Jane Street's NYC datacenter at 1 New York Plaza set off the smoke alarm, which was wired to shut down the CRAC cooling units.
• With cooling off and the sprinkler pipes pressurized, the author feared the room would pass 150 F, the heat that opens the sprinkler heads, so he ran to the office and pulled the power cord on every device.
• The monitor's last readings were 141, 142, 143, 143.5 and 143.6 F before the temperature stopped climbing; downtime was minimal because the major NYC markets were closed, and systems were back the next morning.
• After the incident the sprinklers stayed, procedures and surveillance were tightened, and servers stayed in the office building next to the trading floor.
• The author then wrote an OCaml monitor that checks every minute the onboard-sensor temperature of every server in each location, averages it, and alerts the systems team if it is elevated.

Why it matters

The story shows how safety systems that each make sense alone can combine into a hazard. A sprinkler setup whose pipes usually stay empty and fill only on smoke still sprays only on heat, and a smoke alarm that shuts off cooling to starve a fire of air makes heat inevitable in a room full of servers. The author also shows a design habit at work: Jane Street kept servers close enough to the trading floor that a person could physically pull the power as a last resort, and that is what saved the day here. Finally, the fix is a useful monitoring idea: no single hot server is alarming, but a fleet-wide average rise points to a cooling failure.

Who it affects

Anyone who runs or hosts servers in a room with sprinklers, especially inside an ordinary office building, and operations teams who rely on per-machine alerts. It is also a reference point for firms that keep machine rooms next to trading or operations floors for physical control. Nothing in the post is about AI.

How to use it

The practical takeaway is the monitor the author built. Each minute, ask every server in each location for its onboard-sensor temperature, take the mean, and alert the whole systems team if it is elevated. The author prefers this to alerting on CRAC shutdown or a dedicated room sensor because that telemetry was nowhere near as reliable. He wrote his version in OCaml; the post gives no code and no threshold. The other lesson is to check what your smoke detection is wired to, including whether it cuts cooling, and to know how your sprinkler system arms and triggers.

How solid is it

This is a personal blog account written from memory, and Jane Street is not quoted and has not confirmed it. The post gives no year or date for the incident. The technical explanation of how such sprinklers work is the author's general description ("usually"), not a documented specification of this building. The author says the story was written by human hands, with AI doing proofreading and light edits, and the tone is anecdotal and humorous in places.

Risks and caveats

The account does not say whether the sprinklers ever discharged; the stated outcome is that the temperature stopped climbing after the power was pulled. It gives no dollar figure and does not say whether servers were damaged or data lost. Pulling power on a whole room is a drastic step that worked here because the markets were closed. The post also does not say how many servers the later monitor covers or whether it ever fired. Treat it as a vivid lesson rather than a verified incident report.

“the smoke alarm is wired to shut down the CRAC units(!)”

— From the blog post by a member of Jane Street's systems team