MGPUSim and Akita Framework: Multi-GPU Interconnect Architecture, Performance Modeling, and Application Analysis

Overview

This article examines MGPUSim and the Akita framework, two related open-source Go-based tools for computer architecture research. MGPUSim is a cycle-accurate multi-GPU simulator for the AMD GCN3 ISA; Akita is the general-purpose simulation engine on which MGPUSim is built. (An NVIDIA GPU model is under development but not yet stable.)

Key points

MGPUSim: multi-GPU simulator for AMD GCN3

• Models GPU microarchitecture, caches, memory controllers, interconnect networks, and runtime behavior; supports discrete and unified multi-GPU configurations.
• Runs unmodified GPU kernels compiled with the ROCm toolchain.
• Parallel simulation: 3.5x (functional) and 2.5x (detailed timing) speedup on a 4-core CPU with no accuracy loss versus serial execution.
• Validation against an AMD R9 Nano GPU yields a 5.5% average simulation error; per-benchmark error peaks near 20% (e.g., FIR, SC) due to undisclosed hardware details such as pipeline structures and network buffer sizes.
• Validation was performed at two levels: microbenchmarks (L1/L2 cache, DRAM, ALU) and full benchmark suites (AMD APP SDK, Hetero-Mark) on a 2-GPU R9 Nano system.

Akita: a simulator-building engine

• Event-driven core with Smart Ticking (cycle-style programming over an event-driven backend) and parallel execution support.
• Modular component library: caches (write-through/write-back), TLBs, memory controllers, and a flexible NoC model.
• Developer tools: Daisen (web-based post-simulation visualization with hierarchical timelines) and AkitaRTM (interactive live monitoring dashboard for debugging deadlocks and analyzing performance).
• Other simulators built on Akita: Yori (RISC-V, branch prediction and Spectre-style side-channel research), TrioSim (DNN training system modeling), Zeonica (CGRA modeling).

Relationship

MGPUSim is an instantiation of Akita focused on multi-GPU systems. It reuses Akita's event engine, parallel simulation, network model, and component library, focusing its own development on GPU microarchitecture and multi-GPU interconnect modeling.

Network modeling

• Akita's network model targets multiple interconnect types (PCIe, NVLink, Ethernet, InfiniBand), separating public builder APIs from internal implementations for extensibility.
• An event classification mechanism (primary tick events processed before secondary message-delivery events) prevents same-cycle ordering artifacts and improves fidelity.
• Supported topologies: tree, bus, star, mesh, and custom topologies built from Root Complex, Switch, and Endpoint components.
• Endpoints handle message fragmentation into flits and reassembly; switches implement configurable routing and arbitration algorithms (e.g., single-port arbitration emulates a bus; all-port forwarding emulates a crossbar).
• MGPUSim integrates this model for cycle-level interconnect simulation, capturing NUMA effects, remote memory access, and communication overhead.

Optimization strategies evaluated in MGPUSim

1. Locality API — programmer-facing APIs (implemented via a wrapping driver) to explicitly place data buffers and kernel execution on specific GPUs, improving data locality. Average 1.6x speedup (geometric mean) on a 4-GPU system. 2. PASI (Progressive Page Splitting Migration) — transparent hardware mechanism with a Page Migration Controller (PMC), cache-only memory architecture, and an ESI (Exclusive-Shared-Invalid) coherence protocol. Three variants: page migration only; migration + ESI; migration + ESI + page splitting (mitigating false sharing). Average 2.6x speedup (geometric mean) on a 4-GPU system, outperforming the Locality API.

Applications

• Multi-GPU architecture design space exploration (topologies, interconnect technologies, coherence protocols, new MMU/prefetcher/router designs).
• Application performance analysis and optimization using Daisen visualizations and the Locality API.
• Research on next-generation interconnects, including silicon photonics and chiplet-based interconnect.