The Curse of the Lock: A Hidden Memory Free Bug That Crippled an Entire AMD Server Fleet

Days before the Double 11 promotion, a fleet of AMD Turin servers suddenly "ran a fever": CPI (cycles per instruction) across all business workloads jumped from under 1 to 3–4, CPU utilization soared, online services were throttled, and offline tasks suffered badly. This was not ordinary performance jitter—it was a near fleet-wide crisis.

The Storm Hits: A Pre-Promotion Performance Nightmare

In late August, the newest AMD Turin machines began showing a strange disease: whole-machine CPI spiked from a normal value (<1) to 3 or even 4. Instruction counts were unchanged, but each instruction took three to four times more cycles. These machines hosted many core business services, plus an offline "kata container" that exclusively occupied all vCPUs to provide a full VM environment for ODPS. With the promotion imminent, this was escalated to top priority.

Symptoms: Every Core Stalls Simultaneously

Monitoring showed CPI rising on all business Pods at the same time, with no physical core spared. Online CPU utilization surged while the offline kata container was squeezed to a standstill.

Normally, whole-machine CPI increases point to memory bandwidth or latency bottlenecks—but this time L3 cache and memory access actually dropped sharply.

Microarchitectural Forensics: Front-End Fetch Breaks Down

Using AMD microarchitectural counters and topdown analysis on the affected machines:

• Very high instruction L1 I-cache miss rates
• Large numbers of instructions fetched from remote CCDs
• Severe front-end fetch stalls; instruction dispatch blocked
• Overall core efficiency collapse

Split lock / bus lock was initially suspected, but perf stat -e ls_locks.bus_lock caught nothing at first. As a last resort, the team SIGSTOP'd the rund VM inside the kata container—the machine instantly recovered. The problem was traced to a business process inside rund.

> What is a split lock? > When an atomic operation crosses a cache line boundary (typically 64 bytes), it becomes a split lock. Both Intel and AMD treat it as performance poison. On older architectures it triggers a bus lock, serializing instruction fetch across the entire SMP system.

Pinpointing the Culprit: A Hidden Lock Contention in a Python UDF

By SIGSTOP'ing high-CPU processes one at a time, a single culprit process was identified. The problematic jobs shared a common trait: a C++ SQL execution engine calling Python UDFs directly on the critical path to process strings.

perf showed the problem thread spending 100% of its time in __lll_lock_wait_private → __x86_indirect_thunk_rax → __x86_sys_futex. pstack pointed to glibc futex lock waits, and critically, the futex uaddr was 0xffffffff—a classic split lock signature. Inside the rund VM, perf stat -e ls_locks.bus_lock finally captured bus lock events from exactly that thread.

Reproduction: One Cross-Line Lock, Whole-Machine Disaster

A minimal test program locked memory at a 15-byte offset on 64-byte-aligned allocations, creating a cross-cache-line lock, then called the extracted __lll_lock_wait_private:

• AMD Turin: all cores' CPI spiked to 3–4
• Latest Intel machines: only the physical core performing the split lock (plus its hyperthread sibling) was affected
• Older Intel Skylake: same as AMD—whole machine affected

This perfectly reproduced the production phenomenon and explained why the issue only erupted on the newest AMD generation.

Root Cause: Fatal Mixing of jemalloc and ptmalloc

Taking a gcore of the problem process and resolving it with gdb inside the correct mount namespace showed threads stuck in the Python interpreter's list_dealloc → PyMem_FREE → __libc_free → _int_free.

Key findings:

• The arena pointer (av) passed to _int_free was 0xffffffff
• av->mutex was consequently 0xffffffff, causing __lll_lock_wait_private to receive an invalid address and trigger a split lock
• In __libc_free, p = mem2chunk(mem) = mem - 0x10, and arena_for_chunk(p) incorrectly took the heap_for_ptr branch, returning 0xffffffff

The memory in question was not allocated by glibc ptmalloc at all—it was allocated by jemalloc, yet wrongly routed into glibc's free path.

The Fatal Chain: RTLD_DEEPBIND + fork + malloc_trim

The business process linked both jemalloc and libpython.so. When libpython.so was loaded via dlopen with RTLD_DEEPBIND, Python's internal malloc/free symbols bound to glibc instead of jemalloc. Normally jemalloc hijacks free via __free_hook (routing to je_free), but:

1. Somewhere the process called malloc_trim, triggering ptmalloc_init and setting __malloc_initialized = 1 2. The process executed a fork at some point 3. During the fork's parent-lock window, __free_hook was temporarily replaced by glibc's internal free_atfork 4. At exactly that moment, a Python interpreter thread freed jemalloc-allocated memory, which went into glibc's _int_free 5. The wrong arena determination produced av = 0xffffffff → split lock → bus lock → fleet-wide CPI explosion

This is an extremely rare race condition requiring a perfect storm of timing.

Why Is AMD Hit Harder?

• Intel recognized the dangers of split locks in the multi-core era and implemented microarchitectural optimizations (trade secrets) that confine the impact to the local physical core
• AMD's Turin generation has not fully closed this gap: split locks still genuinely trigger bus locks affecting all CCDs
• Intel kernels have split_lock_detect, printing #AC warnings in dmesg with optional rate limiting; AMD must wait for later kernel patches
• The host cannot perceive bus locks originating inside VMs, due to PMU context isolation

Defense Guide: Avoiding the Split Lock Trap

1. Strictly align atomic variables (alignas(64) or higher) 2. Place large atomic types at the front of structs 3. Prefer combinations of small atomics over 128-bit atomics 4. Use aligned_alloc instead of plain malloc 5. Check alignment with static_assert 6. Never place atomic types in packed structs

More importantly: avoid mixing memory allocators, especially in complex dlopen + RTLD_DEEPBIND + fork scenarios.

Epilogue: After the Storm

The ODPS team acted quickly:

• Forced isolation mode for high-risk jobs, unifying on tcmalloc
• After a two-week gradual rollout, the problem essentially disappeared
• Long-term: avoid calling malloc_trim and similar functions that trigger ptmalloc initialization

On the kernel side, AMD split-lock detection patches are about to land, bringing dmesg warnings in the next minor release. AMD has also committed to suppressing bus lock impact in the next-generation microarchitecture.

This "curse of the lock" has been broken, but it leaves a deep warning: in a modern complex software stack, a seemingly trivial symbol binding difference can escalate into a fleet-wide disaster under extreme timing. Keeping allocators consistent, watching hook states during fork, and strictly aligning atomic variables—these details determine whether a system stays rock-solid during peak traffic.

------

References

1. Deep Dive into Split Locks: The Disaster i++ Can Cause — Volcano Cloud 2. RTLD_DEEPBIND documentation and glibc manual 3. x86/cpu: Add Bus Lock Detect support for AMD (Linux kernel patch) 4. Best Practices for Avoiding Split Lock Performance Contention — Alibaba Cloud 5. glibc Source Analysis: malloc/free Implementation and Arena Management