Policy-as-Skill: Governed LLM Decision Support with Evidence, Determinism, and Auditability

Forum topic · 小凯 · 2026-09-25

Summary

This arXiv paper (2609.27087) introduces Policy-as-Skill (PaS), a modular runtime that packages governance functions for LLM-based decision support—evidence validation, review routing, version control, and auditability—as executable, versioned policy capabilities. The authors (Mohsenzadegan, Tavakkoli, Kyamakya) evaluate 13 methods with a fixed Gemma4 backend across 600 development tasks. PaS+Audit achieves 53.8% exact accuracy, macro-F1 of 0.346, review F1 of 0.854, citation precision of 1.000, policy-reference recall of 0.984, and audit completeness of 1.000, outperforming an LLM+RAG baseline on most governance and review metrics. Deterministic control further raises aggregate accuracy to 61.2%, though the gain is strongly task dependent, suggesting selective rather than universal application of rule-based intervention. The work targets organizations using LLMs for policy, compliance, risk, and operational decisions where auditability is essential.

Paper Overview

Field: ML Authors: Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya Published: 2026-09-25 arXiv: 2609.27087

Abstract

Organizations increasingly use LLMs for policy, compliance, risk, and operational decision support, requiring evidence validation, review routing, version control, and auditability. We introduce Policy-as-Skill (PaS), a modular runtime that packages these functions as executable, versioned policy capabilities. Thirteen methods are evaluated with a fixed Gemma4 backend on 600 development tasks. PaS+Audit achieves 53.8% exact accuracy, macro-F1 0.346, review F1 0.854, citation precision 1.000, policy-reference recall 0.984, and audit completeness 1.000, outperforming LLM+RAG on most governance and review metrics. Deterministic control raises aggregate accuracy to 61.2% but is strongly task dependent, supporting selective rather than universal rule-based intervention.

Key Results at a Glance

Exact accuracy: 53.8% (PaS+Audit)
Macro-F1: 0.346
Review F1: 0.854
Citation precision: 1.000
Policy-reference recall: 0.984
Audit completeness: 1.000
With deterministic control: aggregate accuracy rises to 61.2%, though gains are strongly task dependent

Takeaway

PaS outperforms LLM+RAG on most governance and review metrics, indicating that packaging governance functions as versioned, executable policy capabilities is a practical approach for auditable LLM decision support. However, deterministic rule-based intervention should be applied selectively rather than universally, given its task-dependent benefits.