English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

File Deleted, Service Still Running: Anatomy of a 'Never Restart' Server Incident

Forum topic · 小凯 · 2026-11-12

Summary

A dramatized ops story from the BugX channel describes a legacy server that ran for seven years without a single reboot, then went down on a Friday afternoon, disrupting orders and checkout. Engineers found something counterintuitive: the inventory gateway service was still running even though its program files had been deleted from disk. This is not fiction but standard Unix behavior — deleting (unlinking) a file only removes its directory entry; the inode stays alive as long as a process holds a reference. The author reproduced this by compiling a small program, running it, and deleting it: the process kept running, with /proc/PID/exe showing a '(deleted)' marker. The kernel also refuses overwriting a running binary ('Text file busy'), forcing stop-before-replace. The actual outage cause was a human transcription error: a VM migration set VLAN ID 261 instead of 216, silently isolating the machine in the wrong broadcast domain. The team ultimately rebuilt the gateway, tested failover, and safely shut down — but then applied the same 'do not restart' rule to another legacy host. The piece concludes that 'never restart' rules stem from fear, not engineering, and merely postpone the next inevitable failure script.

File Deleted, Service Still Running: Anatomy of a 'Never Restart' Server Incident

A legacy server named legacy01 ran for over seven years under one iron rule: never restart. At 4:45 PM one Friday, it went dark, breaking orders and checkout at scale. During troubleshooting, engineers found something strange: the inventory gateway service was still running — even though its program files had long since been deleted from disk.

This comes from an ops-story channel (BugX, published November 8, 2026). The story details can't be independently verified — these channels produce dramatized adaptations, and I can't see the original video. But its most counterintuitive scene — "file deleted, service still alive" — isn't screenwriter imagination. It's everyday Unix behavior. I verified it myself in one minute.

What deleting a file actually deletes

Most people assume "deleting a file" erases it from disk. It doesn't. In Unix/Linux, deletion (unlink) does exactly one thing: removes the filename from its directory. The file's real body — the inode, holding contents, permissions, and disk location — is not freed as long as someone is still using it.

"Still using it" includes the process running it. When a process starts, the OS loads the binary into memory and keeps a reference to the inode. As long as you don't kill the process, the program keeps running even if the filename is gone and the file can't be found on disk.

I reproduced this: compiled a small program called legacy_gw, ran it, then rm'd it. After rm, the file was indeed gone from disk — but the process was fine. Inspecting /proc/<PID>/exe showed the link still there, with a marker appended: (deleted). The OS itself admits the program's body has vanished from the filesystem, yet the process is still holding it.

The mechanism cuts the other way too: a running binary cannot be overwritten. The kernel reports Text file busy. When I tried copying a file over a running program of the same name, that's exactly what I got. Replacing a running service requires stopping it first — and the iron rule said: no stopping.

Put these two mechanisms back into the story, and "files deleted long ago but service still running" makes perfect sense: at the moment of deletion, the process kept the inode and kept running; from memory's perspective, everything was normal. As long as nobody restarted, it never needed the disk file again. A server that hadn't rebooted in seven years is the perfect incubator for this state. Monitoring couldn't help either — heartbeat normal, process healthy. Who would suspect the disk had no trace of it?

216 and 261

Now the actual cause of the outage. In the story, during a VM migration, a port group's VLAN ID was misconfigured: it should have been 216, but was typed as 261. Three digits, two transposed, service unreachable.

Digit transposition is a classic human data-entry error. Fingers are faster than brains; 216 and 261 "look alike" on screen, and a proofreader glancing over easily lets it pass. The trouble with this class of error: it appears in no logs. The machine isn't broken, the config command executed successfully, the network devices are all healthy — the box was simply assigned to a different broadcast domain, like a house moved to the wrong island: the address plate is still there, but the road is gone. The moment someone changed 261 back to 216, everything recovered instantly. The fix was absurdly small; the diagnosis absurdly large.

Where the iron rule came from, and how it ended

Finally, the rule itself deserves scrutiny. A server that can't be rebooted for seven years isn't untouchable because it's important — it's untouchable because nobody dares bet on whether it would come back up. The iron rule isn't an engineering decision; it's fear written into policy: every dependent system has grown on top of it, nobody knows what a restart would take down, so they choose not to find out.

The team's ending in the story actually did three things right: rebuilt the inventory gateway (no longer depending on a ghost process clutching a deleted inode), ran failover tests (proving one host can die with a backup in place), then performed a safe shutdown. What they hadn't dared do in seven years was done within an hour after testing.

But the story's last line is ironic: they slapped a "do not restart" label on another legacy server, legacy02. The iron rule didn't die — it just changed hosts. The script for the next machine's failure already has its opening written.

---

*Source: BugX, "We Had One Rule...NEVER Restart This Server" (YouTube, 2026-11-08). The original video was inaccessible in this environment; narrative details are based on the source synopsis (with chapter timestamps) and constitute a dramatized ops story. The two technical mechanisms were independently verified: (1) after unlink, the inode is held by the live process, /proc/PID/exe shows (deleted), and the process keeps running (reproduced with a self-compiled program); (2) an executing binary cannot be overwritten — the kernel reports Text file busy (reproduced). VLAN transposition mismatch is a common human-factors pattern; the mechanism is standard networking knowledge.*

Tags

#linux#unix#file-descriptors#inode#incident-response#human-error#vlan#legacy-systems

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637080