The Curse of the Lock: How a Hidden Memory-Free Bug Made AMD Servers Stop Dead

Forum topic · ✨步子哥 · 2026-01-29

Summary

During a pre-Singles' Day load test, a fleet of AMD Turin servers suffered a system-wide CPI spike from under 1 to 3–4, degrading all online and offline workloads. Investigation using AMD microarchitectural counters and topdown analysis traced the issue to frontend fetch stalls caused by split-lock-triggered bus locks. The root cause was a rare race condition: a C++ SQL engine process linked jemalloc while dlopen-ing libpython.so with RTLD_DEEPBIND, which bound Python's malloc/free to glibc. A concurrent malloc_trim call initialized ptmalloc, and a fork temporarily swapped __free_hook to glibc's free_atfork; a Python interpreter thread then freed a jemalloc-allocated block through glibc's _int_free, computing a bogus arena pointer (0xffffffff). The subsequent futex wait on that misaligned address crossed a cache line, triggering a bus lock that serialized instruction fetch across all CCDs. Intel parts contain the impact to one physical core, but AMD Turin does not. Mitigations included forcing isolation mode with tcmalloc for at-risk jobs, upcoming kernel split-lock detection for AMD, and vendor commitments to limit bus-lock impact in future microarchitectures.

Shortly before the Double 11 shopping festival, newly deployed AMD Turin servers began exhibiting a strange illness: system-wide CPI (cycles per instruction) jumped from a normal value below 1 to 3–4. Instruction counts were unchanged, but every instruction took three to four times longer to complete. Online containers saw CPU utilization soar, offline tasks (including a kata container providing a full VM for ODPS) were squeezed, and all business performance degraded simultaneously. The issue was classified as top priority.

Symptoms: All Cores Stalled Together

Monitoring showed CPI rising simultaneously on every pod and every physical core. Contrary to typical CPI regressions caused by memory bandwidth or latency bottlenecks, L3 cache and memory access rates *decreased*. AMD microarchitectural counters and topdown analysis revealed:

Extremely high L1 instruction cache miss rates
Heavy instruction fetches from remote CCDs
Severe frontend fetch stalls blocking instruction dispatch

Stopping the rund VM inside the kata container with SIGSTOP immediately restored the machine, isolating the problem to a business process within it.

> What is a split lock? An atomic operation crossing a cache line (usually 64 bytes) boundary. Both Intel and AMD treat it as a performance poison: in older architectures it triggers a bus lock that serializes instruction fetch across the entire SMP system.

Locating the Culprit: A Python UDF's Hidden Lock Contention

The affected jobs shared two traits: a C++ SQL execution engine calling Python UDFs on the critical path for string processing. perf showed the problem thread spending 100% of its time in __lll_lock_wait_private → __x86_indirect_thunk_rax → __x86_sys_futex, and the futex's uaddr was 0xffffffff — a classic split-lock signature. Inside the rund VM, perf stat -e ls_locks.bus_lock finally captured bus lock events from that thread.

Reproduction

A minimal test program locking an address offset 15 bytes into a 64-byte-aligned allocation (a cross-cache-line lock) reproduced the disaster:

AMD Turin: CPI of *all* cores rose to 3–4
Latest Intel: only the physical core (and its hyperthread sibling) performing the split lock was affected
Older Intel Skylake: same as AMD — whole machine affected

Root Cause: jemalloc and ptmalloc Fatally Mixed

A core dump analyzed with gdb (in the correct mount namespace) showed threads stuck in Python's list_dealloc → PyMem_FREE → __libc_free → _int_free. The arena pointer (av) passed to _int_free was 0xffffffff:

__libc_free computes p = mem2chunk(mem) = mem - 0x10
arena_for_chunk(p) wrongly took the heap_for_ptr branch, returning 0xffffffff
The mutex at that bogus address caused the misaligned futex wait → split lock → bus lock

The memory was allocated by jemalloc but freed through glibc's ptmalloc path.

The Fatal Chain: RTLD_DEEPBIND + fork + malloc_trim

The process linked both jemalloc and libpython.so. Opening libpython.so with RTLD_DEEPBIND bound Python's malloc/free symbols to glibc rather than jemalloc. Normally jemalloc intercepts frees via __free_hook (je_free), but:

1. A call to malloc_trim triggered ptmalloc_init, setting __malloc_initialized = 1 2. The process called fork 3. During the fork's parent-lock window, __free_hook was temporarily switched to glibc's free_atfork 4. A Python interpreter thread freeing a jemalloc block took glibc's _int_free path 5. Wrong arena determination → av = 0xffffffff → split lock → bus lock → system-wide CPI spike

An extremely rare race requiring perfect timing to trigger.

Why AMD Is Hit Harder

Intel optimized split-lock handling in microarchitecture (proprietary details), confining impact to the local physical core
AMD Turin still triggers a real bus lock, affecting all CCDs
Intel kernels offer split_lock_detect (dmesg #AC warnings and throttling); AMD support requires a pending kernel patch
The host cannot observe bus locks inside VMs due to PMU context isolation

Defense Guidelines

1. Strictly align atomic variables (alignas(64) or better) 2. Place large atomic types at the start of structs 3. Prefer combinations of small atomics over 128-bit atomics 4. Use aligned_alloc instead of plain malloc where alignment matters 5. Verify alignment with static_assert 6. Never place atomics in packed structs

Most importantly: avoid mixing memory allocators, especially in complex dlopen + RTLD_DEEPBIND + fork scenarios.

Resolution

The ODPS team force-enabled isolation mode (uniform tcmalloc) for high-risk jobs; after a two-week gradual rollout the problem disappeared. Long-term, calls to malloc_trim and other functions that trigger ptmalloc initialization will be avoided. On the kernel side, AMD split-lock detection (bus lock #DB reporting in dmesg) is close to being merged. AMD has also committed to suppressing bus-lock impact in its next-generation microarchitecture.

The lesson: in modern complex software stacks, a seemingly trivial symbol-binding difference can, under rare timing, bring an entire machine to its knees. Allocator consistency, hook states during fork, and atomic alignment determine whether systems stay stable at peak load.

References

1. Deep dive into split locks and disasters caused by i++ (Volcengine) 2. RTLD_DEEPBIND documentation and the glibc manual 3. x86/cpu: Add Bus Lock Detect support for AMD (Linux kernel patch) 4. Best practices for avoiding split lock performance contention (Alibaba Cloud) 5. glibc source analysis: malloc/free implementation and arena management