Improving Instruction Hierarchy Capabilities in Frontier LLMs (OpenAI Research)

Forum topic · ✨步子哥 · 2026-03-14

Summary

A zhichai.net forum post presents OpenAI research (dated 2026-03-10) on strengthening the instruction hierarchy in frontier large language models. The hierarchy ranks System prompts (security policy, highest priority) above Developer instructions (product constraints), User requests (task instructions), and Tool outputs (untrusted external data). The training approach uses a new IH-Challenge dataset of conversations containing high/low-privilege instruction conflicts, objective Python-script-based scoring of whether higher-level constraints are respected, and safeguards against reward shortcuts such as over-refusal. Key difficulties include confusion between execution failure and hierarchy failure, the subjective nuance of instruction conflicts, and models degenerating into blanket refusal for safety. Experiments on GPT-5 Mini-R show gains of +0.11 on System vs User conflicts (0.84 to 0.95), +0.15 on TensorTrust dev-user benchmarks (0.76 to 0.91), and +0.21 on IH-Challenge over-refusal (0.79 to 1.00). Practical value includes better adherence to safety rules in system prompts without sacrificing helpfulness, and improved resistance to prompt injection by treating tool output as data rather than instructions.

Improving Instruction Hierarchy Capabilities in Frontier LLMs

This forum post summarizes OpenAI research (published 2026-03-10) on enhancing the instruction hierarchy capability of frontier large language models. The initiative is called IH-Challenge: using reinforcement learning to resolve multi-source instruction conflicts and build a robust priority system.

Core Concept: Instruction Hierarchy (System > Tool)

Instructions are ranked in priority order:

1. System — security policy, highest privilege 2. Developer — product constraints, application logic 3. User — explicit requests, task instructions 4. Tool — external data, untrusted source

Training Method

IH-Challenge dataset: constructs conversations containing high/low-privilege instruction conflicts.
Objective scoring: Python scripts objectively determine whether the model respected higher-level constraints.
Avoiding shortcuts: prevents the model from gaming the score simply by over-refusing requests.

Why It Is Hard

Confusion: execution failure is easily misjudged as hierarchy failure.
Subjectivity: instruction conflicts often involve subtle judgment calls.
Degeneration: models can learn the lazy strategy of "refuse everything for safety."

Experimental Results (GPT-5 Mini-R)

| Benchmark | Before | After | Gain | |---|---|---|---| | System <> User Conflict | 0.84 | 0.95 | +0.11 | | TensorTrust (dev-user) | 0.76 | 0.91 | +0.15 | | IH-Challenge (Overrefusal) | 0.79 | 1.00 | +0.21 |

The over-refusal result of 1.00 indicates the trained model perfectly avoided excessive refusal behavior.

Practical Value

Safety and controllability: better adherence to safety rules in system prompts and rejection of violating requests, without sacrificing helpfulness.
Prompt injection resistance: treating tool output as untrusted data rather than instructions, effectively defending against malicious attacks embedded in tool results.

*Source: translated from the OpenAI blog (2026-03-10).*