CAX-Agent: A Lightweight Agent Harness for Reliable MAPDL Automation via Layered Recovery Escalation

Forum topic · 小凯 · 2026-05-19

Summary

This paper introduces CAX-Agent, a lightweight Agent Harness that wraps a large language model around Ansys MAPDL finite-element simulation to improve reliability. The architecture separates execution into three layers (LLM service, agent harness, and solver backend) and adds a recovery ladder that escalates from deterministic rule patching to model-driven regeneration, context-augmented prompting, and human intervention. The authors empirically evaluate three recovery strategies—no_recovery, rule_only, and model_only—across 50 standard structural benchmarks with 3 repeats each (450 runs total). Two blinded human raters scored task completion with strong inter-rater agreement (quadratic-weighted Cohen's kappa = 0.84). Model_only achieved the best completion rate (0.9267), task score (3.59/4), total score (9.16/10), and zero-intervention rate (0.84), outperforming rule_only (0.7733, 3.17/4, 7.03/10, 0.00) and no_recovery (0.6933, 2.74/4, 5.60/10, 0.00) with large effect sizes (Cliff's delta = 0.81–0.87). Findings are limited to deliberately simplified geometries.

Research Overview

Field: Machine Learning / AI for Engineering Simulation
Authors: Chenying Lin, Yichen Hai, Yi He
Published: 2025-05-15
arXiv: 2505.10887

Abstract

Large language models deployed for MAPDL finite-element simulation face practical reliability challenges: without structured execution control, tool encapsulation, and fault recovery, outputs may be inconsistent and task failures are common. The Agent Harness paradigm addresses this by inserting domain-specific orchestration middleware that manages tool lifecycles, workflow state, and recovery escalation. This paper presents the architecture of CAX-Agent, a lightweight agent harness purpose-built for MAPDL automation, and empirically evaluates one of its core components — the recovery ladder. CAX-Agent organizes execution into three layers — LLM service, agent harness, and solver backend — with a recovery ladder that escalates from deterministic rule patching through model-driven regeneration, context augmentation, and human intervention.

Key Findings

Three-layer architecture: CAX-Agent separates concerns into an LLM service layer, an agent harness layer (orchestration middleware), and a solver backend layer (MAPDL).
Recovery ladder design: Escalation moves from deterministic rule patching → model-driven regeneration → context-augmented prompting → human intervention.
Evaluation setup: 50 standard structural benchmarks × 3 recovery strategies (no_recovery, rule_only, model_only) × 3 repeats = 450 task runs.
Human rating protocol: Two raters, blinded and independent; quadratic-weighted Cohen's kappa = 0.84, with 96% of score pairs within 1 point.
Performance results (completion rate / task score / total score / zero-intervention rate):
model_only: 0.9267, 3.59/4, 9.16/10, 0.84
rule_only: 0.7733, 3.17/4, 7.03/10, 0.00
no_recovery: 0.6933, 2.74/4, 5.60/10, 0.00
Effect size: Cliff's delta = 0.81–0.87, indicating large practical differences.
Limitation: Benchmarks use deliberately simplified geometries to isolate the effect of recovery strategies; broader validation on realistic problems is left as future work.

Significance

The work frames LLM-driven engineering simulation as a reliability problem solvable through middleware-style orchestration rather than purely through better base models. The strong gains of model-driven regeneration over rule-only patching suggest that automated regeneration is more valuable than hand-crafted deterministic rules, while still preserving the option to escalate to humans when needed.