English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Two Degrees From a Shower: Jane Street's Data Center False Fire Alarm Story

Forum topic · QianXun · 2026-11-12

Summary

A story recounted by former Jane Street systems engineer Michael Bacarella describes how a contractor's dust from cutting work next to the firm's data center triggered a smoke alarm one evening, charging the sprinkler pipes with pressurized water and cutting the precision air conditioning. As heat accumulated, server surface temperatures climbed to 143.6°F (about 58.7°C) — just a few degrees below the roughly 150°F threshold at which sprinkler heads' thermal fuses melt and would have soaked the entire floor. The incident was spotted by staff in the Hong Kong office monitoring fleet-wide temperature curves, who called New York; an engineer sprinted downtown and pulled power row by row. The post analyzes the failure chain, the OCaml fleet-wide temperature monitoring tool written afterward, why the server room sat next to the trading floor (physical isolation of trading strategies and a manual kill-switch culture), and how Hurricane Sandy later ended the era of self-operated data centers, leading to the firm's 2026 Texas AI facility with 4,032 GB300 GPUs.

A single number frames this story: 143.6°F, roughly 58.7°C. A room full of servers had reached that surface temperature — only a few degrees below the activation point of the sprinkler heads overhead. A few degrees further, and a top market maker's self-operated data center would have been soaked floor-wide on an ordinary evening.

This anecdote was published last week on the personal blog of Michael Bacarella, a former member of Jane Street's systems team, on November 10. Among the 400+ upvotes on Hacker News, half the readers were studying the failure chain, and the other half the firm's organizational culture — two threads that twist into the same knot.

The failure chain

It started with dust. One evening, a contractor was cutting material in a preparation room next to the machine room, and the dust drifted into a smoke detector. The alarm went off. What followed, per the blogger's account, forms a chain:

1. After the alarm system confirmed a fire, the sprinkler pipes charged with pressurized water. The sprinkler heads themselves don't open immediately — each contains a thermal fuse plug that only melts at around 150°F. 2. Simultaneously, the alarm interlock cut power to the data center's precision air conditioning, on the logic that circulating air would feed oxygen to a fire.

The two lines converge here. With the air conditioning stopped, heat accumulated; pressurized water waited overhead for a temperature threshold. The room began climbing toward 150°F.

The discovery came from Hong Kong

The person who noticed wasn't in New York. That night, staff in the Hong Kong office happened to see a whole row of server temperature curves rising in sync on their monitoring screens — no single-machine alarms had fired, and an occasional hot server is too normal to trigger one. Hong Kong called New York, reaching the blogger himself, who lived in a 200-square-foot studio with a nine-minute walk commute. He sprinted downtown, rushed into the 33rd floor of an old financial-district building, and began pulling power cables row by row.

The temperature stopped at 143.6°F. By the next morning, everything was back to normal. The New York markets had already closed that evening, so the loss window was gently contained by timing.

Dissecting the chain

Every link in the failure chain has an engineering rationale, and each deserves scrutiny:

  • Preaction sprinklers are a common data center choice — pipes normally hold no water, charging only after fire confirmation to prevent accidental discharge. But the behavior described (water charging the moment the alarm fires) corresponds to a single-interlock design; a double-interlock system would also require the sprinkler head to melt before water enters.
  • Cutting the HVAC on alarm was called atypical by data center practitioners on HN — precision cooling usually only shuts down during a full-room emergency power-off. Others countered that air blowing over sprinkler heads delays thermal fuse activation, so stopping the AC actually accelerates the approach to the melting threshold — making the failure chain physically self-consistent.
  • A bit of trivia from the professional discussion: in a real fire, usually only the sprinkler head closest to the flames opens, with a coverage radius of about twelve feet. The movie-style whole-floor simultaneous discharge belongs to deluge systems, which machine rooms don't use.
  • What the company built afterward

    The more interesting part is what the firm installed after the incident. The blogger wrote an OCaml monitoring program that polled the onboard temperature sensors of every Linux server each minute, computed fleet-wide averages per location, and alerted the systems team when means went anomalous.

  • Why averages, not per-machine alerts? Single-machine alarms are too noisy; the noise drowns real signals.
  • Why trust onboard sensors? Because they're the last layer of telemetry still credible — every other monitoring path could be affected by the same kind of failure.
  • Why OCaml? The firm wrote everything from trading systems to internal tools in OCaml; the author deliberately avoided Bash and Perl.

Why the server room sat next to the trading floor

Two reasons, per the blog:

1. Counter-espionage — the servers contained trading strategies, and physical separation is the crudest yet most effective defense. 2. Pulling the plug — when a trading system runs out of control, the last human failsafe is running down the corridor and yanking the cables. The firm was slowly building automated kill switches, but kept one red line: automation could run, but there always had to be a red-painted, mechanically-linked lever reserved for humans. The blogger attributes this culture to a partner who had served on nuclear submarines — Bloomberg's September 2026 reporting described the co-founder's office collection of war memorabilia.

Timeline and an honest caveat

The blog doesn't give the incident's year; only fragments pin it down: the Hong Kong office opened in 2010; the post ends by saying they kept servers next to the trading floor "until Hurricane Sandy slapped us in the face, but that's another story"; Sandy hit New York in late October 2012. Combined with a former colleague's November 2026 tweet about "how Jane Street rebuilt a data center inside an office building in 2012," placing the incident shortly before Sandy in 2012 is the most reasonable inference — an inference, not blog text.

One more clarification: "only" in the original title refers to the firm's only machine room in New York, not the company's only data center overall.

The aftermath

After Sandy, the firm moved disaster recovery to New Jersey, migrated latency-sensitive trading into exchange colocation, and the self-operated data center era ended. In June of this year, Jane Street's Texas AI facility was shown off with 4,032 GB300 GPUs, 56 NVLink racks, and roughly 8,000 km of fiber — a firm that started with "six Dell servers in an office" now operates on an entirely different infrastructure budget. The little OCaml program's logic lives on: monitor the whole fleet, watch the means, trust the lowest-level sensors.

The takeaway for every infrastructure builder: the most dangerous thing in system design isn't a single point of failure — it's two individually correct decisions (cut airflow to fight fire; pre-charge sprinklers to prevent false discharge) triggered by the same signal at the same moment. 143.6°F stopped in time. The next one may not.

Tags

#jane-street#data-center#failure-analysis#fire-alarm#infrastructure#quantitative-trading#ocaml#postmortem

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637085