Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning for LLMs

Forum topic · 小凯 · 2026-10-22

Summary

Researchers propose a new multi-objective reinforcement learning method for LLM-based RTL (Register Transfer Level) circuit generation that optimizes the quality and diversity of an entire set of circuits in the PPA (Power, Performance, Area) objective space, rather than a single circuit. Existing RLVR (reinforcement learning with verifiable rewards) approaches aggregate PPA objectives into one scalar reward, making it hard to produce diverse Pareto-optimal solutions across trade-offs like low power, low latency, and small area. The proposed method reformulates Pareto set coverage improvement as sequential circuit generation, and uses a hypervolume contribution (HVC)-based reward function to measure each generated circuit's contribution to the Pareto front. On the RTLLM v2 benchmark, it improves relative hypervolume by 13.3 percentage points over a baseline without PPA optimization and by 14.4 points over single-objective RL, while also yielding superior circuit diversity. Paper: arXiv 2610.26904.

Overview

Field: Machine Learning Authors: Koki Takeshita, Kenji Uebayashi, Hikaru Tomonari, Masaya Tsunokake, Koichiro Yawata, Tatsuya Sasaki, Yusuke Ohtsubo, Kota Dohi, Yasuhiro Sogawa Published: 2026-10-22 arXiv: 2610.26904

Abstract

Recent studies have proposed reinforcement learning with verifiable rewards (RLVR) methods that enable LLMs to generate high-quality RTL circuits from natural-language specifications. Existing approaches optimize circuit PPA (Power, Performance, Area) by aggregating multiple objectives into a single scalar reward, making it difficult to simultaneously generate diverse Pareto-optimal solutions across different trade-offs such as low power, low latency, and small area.

However, in practical design, the relative importance of each objective varies depending on product requirements and design constraints; thus, obtaining a diverse and high-quality candidate set is beneficial in that designers can choose a preferred trade-off a posteriori.

Key Contributions

This work is the first to propose a learning method for LLM-based circuit generation that optimizes the quality and diversity of the entire set of circuits obtained over multiple generations in the PPA objective space, rather than the quality of a single circuit. The method consists of:

1. A generation-based training pipeline that reformulates the coverage improvement of the whole Pareto solution set as a sequential circuit generation problem. 2. A hypervolume contribution (HVC)-based reward function that measures the contribution of generated circuits to the Pareto front.

Results

Evaluations on the RTLLM v2 benchmark show that the proposed method:

Improves relative hypervolume by 13.3 percentage points over a baseline without PPA optimization.
Improves relative hypervolume by 14.4 percentage points over a single-objective reinforcement learning approach.
Demonstrates superiority in circuit diversity.

--- *Auto-collected on 2026-10-22*