MGPUSim and Akita Framework: Multi-GPU Interconnect Architecture, Performance Modeling, and Application Analysis

Forum topic · QianXun · 2025-11-20

Summary

MGPUSim is an open-source, cycle-accurate multi-GPU simulator for the AMD GCN3 ISA, written in Go and built on top of the Akita simulation framework. It models GPU microarchitecture, interconnects (PCIe, NVLink-style links), memory hierarchies, and runtime behavior in a unified environment. MGPUSim achieves high fidelity with an average simulation error of only 5.5% versus an AMD R9 Nano GPU, and supports parallel simulation with 3.5x (functional) and 2.5x (detailed timing) speedups on a 4-core CPU while running unmodified ROCm-compiled GPU kernels. Akita is a modular, event-driven framework for building architecture simulators, offering reusable cache, TLB, DRAM controller, and NoC components, plus developer tools such as Daisen (visualization) and AkitaRTM (real-time monitoring); MGPUSim, Yori, TrioSim, and Zeonica are simulators built on it. Akita's network model uses endpoints and switches with configurable routing and arbitration algorithms, supporting tree, bus, star, and mesh topologies, and a primary/secondary event classification scheme ensures timing correctness. MGPUSim also introduces two multi-GPU optimizations: the Locality API, giving programmers explicit control over data/compute placement (1.6x average speedup on a 4-GPU system), and PASI, a transparent hardware mechanism with cache-only memory, an ESI coherence protocol, and page splitting via a Page Migration Controller (2.6x average speedup).

MGPUSim and the Akita Framework: Multi-GPU Interconnect Architecture, Performance Modeling, and Applications

1. Overview

As data-parallel workloads grow in scale and complexity, single-GPU platforms can no longer meet the extreme compute demands of high-performance computing (HPC). Multi-GPU systems, which aggregate the compute and memory capacity of multiple GPUs, have become the mainstream solution — but they bring challenges in microarchitecture design, interconnects, runtime libraries, and programming models. MGPUSim and the Akita framework were created to give researchers a public, comprehensive, and flexible platform for exploring multi-GPU system design.

1.1 MGPUSim: A Multi-GPU Simulator for AMD GCN3

MGPUSim is an open-source, highly flexible, high-performance multi-GPU simulator targeting the AMD Graphics Core Next 3 (GCN3) instruction set architecture. It is written in Go and models GPU microarchitecture, interconnect networks, memory hierarchies, and runtime behavior within one framework.

Core characteristics:

Flexibility — Users can configure discrete or unified multi-GPU systems, freely combining compute units (CUs), caches, memory controllers, and inter-GPU interconnects via platform configuration functions.
Performance — Built-in multi-threaded execution exploits modern multicore CPUs: on a 4-core CPU, functional simulation achieves an average 3.5x speedup and detailed timing simulation 2.5x, with accuracy identical to serial simulation.
Accuracy — Validated against an AMD R9 Nano GPU, MGPUSim shows an average simulation error of only 5.5% across benchmarks.

Key capabilities: cycle-accurate simulation of multi-GPU systems, modeling of inter-GPU data transfers, cache coherence, and remote memory access (including NUMA effects), and direct execution of unmodified GPU kernels compiled with the ROCm toolchain.

Development status: AMD GCN3 simulation is stable; NVIDIA GPU support is under development per the GitHub repository, but not yet stable.

1.2 Akita: A Framework for Building Next-Generation Simulators

Akita is not a single simulator but a general-purpose engine for building computer architecture simulators, emphasizing performance, flexibility, and developer experience.

Event-driven simulation with "Smart Ticking" — events are only processed when they occur, while developers retain an intuitive cycle-by-cycle programming model. The engine supports parallel execution on multicore CPUs.
Modular component library — caches (write-through and write-back), TLBs, memory controllers, and a Network-on-Chip (NoC) component supporting switches, arbitrary topologies, and routing algorithms.
Developer tools — Daisen, a web-based visualization tool producing execution traces and hierarchical timelines; AkitaRTM, an interactive real-time monitoring dashboard useful for deadlock debugging and interactive performance analysis; and ArchSim, which offers Akita simulators as a web service.

Simulators built on Akita include MGPUSim (multi-GPU), Yori (RISC-V, branch prediction and Spectre-style side-channel research), TrioSim (large-scale DNN training performance modeling), and Zeonica (coarse-grained reconfigurable arrays).

1.3 Relationship: MGPUSim as an Akita Instance

MGPUSim is built on Akita rather than developed from scratch — an application/engine relationship. It inherits Akita's event-driven core, parallel simulation, and modular components, letting its developers focus on multi-GPU-specific microarchitecture and interconnect models.

2. Multi-GPU Interconnect Design and Implementation

2.1 Akita's General Network Model

Design philosophy. Akita's network model is designed for generality and extensibility, able to emulate interconnect technologies such as PCIe, NVLink, Ethernet, and InfiniBand. Implementation details (packet formats, routing, arbitration) are encapsulated in Go internal packages, while users interact with simple network-builder APIs — e.g., a PCIe v3 x16 link takes only a few lines of code to configure. New interconnect technologies can be added without changing the public API.

Event handling. Akita classifies events into primary events (component tick events) and secondary events (network message deliveries). Within the same cycle, all primary events execute first, then secondary events. This guarantees components generate traffic before the network delivers it, and that messages arriving in the current cycle cannot be consumed until later — matching real hardware timing.

Topology support. Tree, bus, star, mesh, and custom topologies. PCIe-style trees are built with CreateRootComplex, CreateSwitch, and PlugInDevice; routing in trees is automatic, while mesh/ring topologies use custom routing tables.

Components.

Endpoints — the network interface for devices; they split outgoing messages into flits (one per cycle typically) and reassemble incoming flits into messages.
Switches — routing and forwarding nodes with configurable routing algorithms (route tables, dimension-order, adaptive) and arbitration algorithms (round-robin, priority, random). Arbitration choices change network semantics: single-port arbitration yields a bus, all-port simultaneous arbitration yields a crossbar.

2.2 MGPUSim's Multi-GPU Interconnect

MGPUSim calls Akita's network APIs directly to achieve cycle-level interconnect simulation, tracking each packet hop-by-hop. Each GPU connects to the network through an Akita endpoint; e.g., RDMA requests are packetized, split into flits, routed with computed arrival times, and reassembled at the destination. Switches inherit Akita's configurable routing and arbitration, letting researchers compare strategies' impact on multi-GPU system performance.

3. Performance Modeling and Optimization

3.1 Performance Modeling

MGPUSim collects per-GPU compute time, memory latency, cache hit rates, and inter-GPU traffic/latency, letting researchers classify applications as communication-intensive, identify remote-memory/NUMA bottlenecks, and analyze point-to-point vs. collective (e.g., AllReduce) patterns.

Validation proceeds at two levels:

Microbenchmarks targeting L1/L2 caches, DRAM, ALU, etc. — simulated curves nearly coincide with real hardware (R9 Nano).
Full benchmarks from AMD APP SDK and Hetero-Mark on a 2×R9 Nano system — average error of 5.5%. Peaks up to ~20% (e.g., FIR and SC) stem from undocumented hardware details such as subtle cache pipelines and network buffer sizing.

3.2 Locality API

The Locality API gives programmers explicit control over data and compute placement across GPUs, exploiting a priori knowledge of access patterns to maximize data locality and avoid costly remote access. Implemented as an extended driver that wraps the standard driver (backward compatible), it lets programmers pin buffers to specific GPUs and launch kernels on chosen GPUs. Evaluation on a 4-GPU system shows an average 1.6x speedup (geometric mean).

3.3 PASI (Programmable Accelerator Shared Memory)

PASI is a programmer-transparent, hardware-managed optimization that progressively and automatically improves data placement based on runtime behavior. It introduces:

A cache-only memory architecture, where data can be cached on multiple GPUs without full replication in local memory.
An ESI (Exclusive-Shared-Invalid) coherence protocol supporting shared read/write pages.
A Page Migration Controller (PMC) component with three ports (L2 cache, DRAM, RDMA engine), offering three implementation tiers: (1) page migration only; (2) migration + ESI; (3) migration + ESI + page splitting, which mitigates false sharing when multiple GPUs access different parts of one large page.

On a 4-GPU system, PASI achieves an average 2.6x speedup (geometric mean), outperforming the Locality API.

4. Applications and Evaluation

4.1 Application Scenarios

Multi-GPU architecture design and evaluation — exploring interconnect topologies (fully connected, ring, mesh), interconnect technologies (PCIe, NVLink), and coherence protocols; prototyping new MMUs, prefetchers, or routers before hardware commit.
Multi-GPU program profiling and optimization — using Daisen's traces and timelines to locate load imbalance, excessive/irregular communication, and remote-access hotspots; iterating analyze–optimize–verify with Locality API support.
Novel interconnect research — modeling emerging technologies such as silicon-photonics links and Chiplet-based interconnects, studying topology trade-offs, NoC vs. SerDes options, and energy/performance balances.

4.2 Evaluation Metrics and Results

Primary metrics are execution time, plus bandwidth and latency (network and memory), and microarchitectural counters (cache/TLB hit rates).

Verification compares simulation against real hardware at both microbenchmark and full-benchmark levels. Key results:

Microbenchmarks for L1/L2 caches and DRAM match hardware curves almost exactly.
Full-benchmark average error: 5.5% on a dual R9 Nano system, with peaks of ~20% on FIR and SC attributable to undisclosed hardware details.

An average error of 5.5% is sufficient for most architecture research, establishing MGPUSim as a trustworthy tool for multi-GPU performance evaluation and optimization.