[论文] Twelve quick tips for designing AI-driven HPC workflows

论文概要

研究领域: ML 作者: Jamie J. Alnasir 发布时间: 2025-06-11 arXiv: 2506.08630

中文摘要

高性能计算(HPC)集群仍是大规模科学计算的支柱，传统上执行确定性的线性流水线以优化可预测性能。然而，人工智能和基础模型在科学研究中的广泛渗透引入了一种根本性的新计算范式。AI驱动的工作流具有迭代性、数据驱动性和概率性特征，在数据引力、异构资源管理和复杂工作流编排方面带来独特挑战。本指南提供十二条实用建议，帮助研究人员设计高效、可扩展且可复现的AI驱动HPC工作流。通过解决关键系统级瓶颈——如环境可移植性的容器化、作业数组的战略部署、显式反馈循环机制和小文件I/O优化——本文为从刚性执行流水线过渡到自适应智能计算环境提供了框架。这些架构原则广泛适用于分布式环境，特别针对现代计算生物学的资源密集型吞吐量需求。

原文摘要

High-performance computing (HPC) clusters remain the backbone of large-scale scientific computation, traditionally executing deterministic, linear pipelines optimised for predictable performance. However, the pervasive integration of artificial intelligence (AI) and foundation models into scientific research has introduced a fundamentally new computational paradigm. AI-driven workflows are characteristically iterative, data-driven, and probabilistic, introducing unique challenges regarding data gravity, heterogeneous resource management, and complex workflow orchestration. This guide provides twelve practical tips designed to help researchers design efficient, scalable, and reproducible AI-driven HPC workflows. By addressing critical system-level bottlenecks - such as containerisation for ...

--- *自动采集于 2026-06-09*

#论文 #arXiv #ML #小凯