MGPUSim and the Akita Framework: Multi-GPU Interconnect Architecture Deep Dive

MGPUSim and Akita represent significant contributions to computer architecture research, giving researchers powerful capabilities for simulating and analyzing multi-GPU systems. Akita serves as a general-purpose simulator-building engine, while MGPUSim is a specialized multi-GPU simulator constructed on top of it.

As data-parallel workloads grow in scale and complexity, single-GPU platforms can no longer meet the extreme compute demands of high-performance computing (HPC). Multi-GPU systems, which aggregate the compute power and memory capacity of multiple GPUs, have become the mainstream solution — but they introduce challenges in microarchitecture design, interconnect topology, runtime libraries, and programming models.

Key points

• MGPUSim is an open-source, flexible, high-performance multi-GPU simulator targeting the AMD Graphics Core Next 3 (GCN3) ISA, written in Go for fast, parallelized, and accurate multi-GPU system simulation.
• Validated accuracy and speed: average simulation error under 5.5% compared to real hardware; ~3.5x speedup for functional simulation and ~2.5x for timing simulation through multithreaded parallel simulation.
• Akita is a framework for building next-generation architecture simulators with a focus on developer experience. It is not a single simulator but a general engine for creating many kinds of architectural simulators.
• Akita core components:
• Base hardware model library: cache models (write-through/write-back), TLBs, memory controllers, NoC components, switches and interconnect links.
• Developer tooling: Daisen (web-based visualization), AkitaRTM (real-time monitor), ArchSim (simulation-as-a-service platform).
• Relationship: MGPUSim is an instantiation of the Akita framework — an "application on an engine" relationship, not an independently built simulator.
• Interconnect support: Akita's network models underpin multi-GPU interconnect modeling, including PCIe and NVLink topologies.
• Optimization strategies studied with MGPUSim:
• Locality API: up to ~1.6x performance improvement.
• PASI: up to ~2.6x performance improvement.

Comparison: MGPUSim vs. Akita

| Aspect | MGPUSim | Akita Framework | |---|---|---| | Positioning | Multi-GPU simulator for AMD GCN3 | Simulator-building engine for various architectures | | Core function | Multi-GPU simulation, GCN3 ISA support, optimization strategies (Locality API, PASI) | Event-driven simulation core, modular component library, dev tools (Daisen, AkitaRTM) | | Dependency | Built on Akita | Standalone lower-level framework |

Applications and evaluation

MGPUSim is typically used for multi-GPU architecture exploration, interconnect design studies, and evaluating runtime/programming-model optimizations, with evaluation based on comparison against real AMD GPU hardware and parallelized simulation for faster design-space exploration.

References cited in the original post:

• MGPUSim: https://ieeexplore.ieee.org/document/8980359 and https://dl.acm.org/doi/10.1145/3307650.3322230
• Akita: https://sarchlab.org/akita
• MGPUSim tutorial: https://sites.google.com/view/jlabellan/tutorial-mgpusim