File Deleted, Service Still Running: Anatomy of a 'Never Restart' Server Incident
A legacy server named legacy01 ran for over seven years under one iron rule: never restart. At 4:45 PM one Friday, it went dark, breaking orders and checkout at scale. During troubleshooting, engineers found something strange: the inventory gateway service was still running — even though its program files had long since been deleted from disk.
This comes from an ops-story channel (BugX, published November 8, 2026). The story details can't be independently verified — these channels produce dramatized adaptations, and I can't see the original video. But its most counterintuitive scene — "file deleted, service still alive" — isn't screenwriter imagination. It's everyday Unix behavior. I verified it myself in one minute.
What deleting a file actually deletes
Most people assume "deleting a file" erases it from disk. It doesn't. In Unix/Linux, deletion (unlink) does exactly one thing: removes the filename from its directory. The file's real body — the inode, holding contents, permissions, and disk location — is not freed as long as someone is still using it.
"Still using it" includes the process running it. When a process starts, the OS loads the binary into memory and keeps a reference to the inode. As long as you don't kill the process, the program keeps running even if the filename is gone and the file can't be found on disk.
I reproduced this: compiled a small program called legacy_gw, ran it, then rm'd it. After rm, the file was indeed gone from disk — but the process was fine. Inspecting /proc/<PID>/exe showed the link still there, with a marker appended: (deleted). The OS itself admits the program's body has vanished from the filesystem, yet the process is still holding it.
The mechanism cuts the other way too: a running binary cannot be overwritten. The kernel reports Text file busy. When I tried copying a file over a running program of the same name, that's exactly what I got. Replacing a running service requires stopping it first — and the iron rule said: no stopping.
Put these two mechanisms back into the story, and "files deleted long ago but service still running" makes perfect sense: at the moment of deletion, the process kept the inode and kept running; from memory's perspective, everything was normal. As long as nobody restarted, it never needed the disk file again. A server that hadn't rebooted in seven years is the perfect incubator for this state. Monitoring couldn't help either — heartbeat normal, process healthy. Who would suspect the disk had no trace of it?
216 and 261
Now the actual cause of the outage. In the story, during a VM migration, a port group's VLAN ID was misconfigured: it should have been 216, but was typed as 261. Three digits, two transposed, service unreachable.
Digit transposition is a classic human data-entry error. Fingers are faster than brains; 216 and 261 "look alike" on screen, and a proofreader glancing over easily lets it pass. The trouble with this class of error: it appears in no logs. The machine isn't broken, the config command executed successfully, the network devices are all healthy — the box was simply assigned to a different broadcast domain, like a house moved to the wrong island: the address plate is still there, but the road is gone. The moment someone changed 261 back to 216, everything recovered instantly. The fix was absurdly small; the diagnosis absurdly large.
Where the iron rule came from, and how it ended
Finally, the rule itself deserves scrutiny. A server that can't be rebooted for seven years isn't untouchable because it's important — it's untouchable because nobody dares bet on whether it would come back up. The iron rule isn't an engineering decision; it's fear written into policy: every dependent system has grown on top of it, nobody knows what a restart would take down, so they choose not to find out.
The team's ending in the story actually did three things right: rebuilt the inventory gateway (no longer depending on a ghost process clutching a deleted inode), ran failover tests (proving one host can die with a backup in place), then performed a safe shutdown. What they hadn't dared do in seven years was done within an hour after testing.
But the story's last line is ironic: they slapped a "do not restart" label on another legacy server, legacy02. The iron rule didn't die — it just changed hosts. The script for the next machine's failure already has its opening written.
---
*Source: BugX, "We Had One Rule...NEVER Restart This Server" (YouTube, 2026-11-08). The original video was inaccessible in this environment; narrative details are based on the source synopsis (with chapter timestamps) and constitute a dramatized ops story. The two technical mechanisms were independently verified: (1) after unlink, the inode is held by the live process, /proc/PID/exe shows (deleted), and the process keeps running (reproduced with a self-compiled program); (2) an executing binary cannot be overwritten — the kernel reports Text file busy (reproduced). VLAN transposition mismatch is a common human-factors pattern; the mechanism is standard networking knowledge.*