[论文] Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement L...

论文概要

研究领域: ML 作者: Koki Takeshita, Kenji Uebayashi, Hikaru Tomonari, Masaya Tsunokake, Koichiro Yawata, Tatsuya Sasaki, Yusuke Ohtsubo, Kota Dohi, Yasuhiro Sogawa 发布时间: 2026-10-22 arXiv: 2610.26904

中文摘要

近期研究提出了基于可验证奖励的强化学习（RLVR）方法，使LLM能够从自然语言规格生成高质量RTL电路。现有方法通过将多个目标聚合为单一标量奖励来优化电路的功耗、性能和面积（PPA），这使得在不同权衡（如低功耗、低延迟和小面积）下同时生成多样化的帕累托最优解变得困难。然而在实际设计中，各目标的相对重要性随产品需求和设计约束而变化，因此获得多样且高质量的候选解集是有益的，设计者可以在事后选择偏好的权衡方案。本工作首次提出了一种针对基于LLM的电路生成的学习方法，其优化的不是单个电路的质量，而是多次生成所获得的整个电路集合在PPA目标空间中的质量和多样性。所提方法包括：（1）一种基于生成的训练流程，将整个帕累托解集的覆盖改进重新形式化为顺序电路生成问题；（2）基于超体积贡献（HVC）的奖励函数，用于衡量生成的电路对帕累托前沿的贡献。在RTLLM v2基准测试上的评估表明，所提方法相对无PPA优化的基线在相对超体积上提升13.3个百分点，相对单目标强化学习方法提升14.4个百分点，同时在电路多样性方面也表现出优越性。

原文摘要

Recent studies have proposed reinforcement learning with verifiable rewards (RLVR) methods that enable LLMs to generate high-quality RTL circuits from natural-language specifications. Existing approaches optimize circuit PPA (Power, Performance, Area) by aggregating multiple objectives into a single scalar reward, making it difficult to simultaneously generate diverse Pareto-optimal solutions across different trade-offs such as low power, low latency, and small area. However, in practical design, the relative importance of each objective varies depending on product requirements and design constraints; thus, obtaining a diverse and high-quality candidate set is beneficial in that designers can choose a preferred trade-off a posteriori. In this work, we are the first to propose a learning me...

#论文 #arXiv #ML #小凯