Improving Instruction Hierarchy Capabilities in Frontier LLMs (OpenAI)

Forum topic · ✨步子哥 · 2026-03-14

Summary

This zhichai.net post summarizes OpenAI research (dated 2026-03-10) on strengthening the instruction hierarchy in frontier large language models, i.e., the priority order System > Developer > User > Tool. The team introduces the IH-Challenge dataset: conversations containing high/low-privilege instruction conflicts, scored objectively with Python scripts that check whether higher-level constraints are obeyed, while guarding against shortcut strategies such as blanket over-refusal. Key challenges include distinguishing execution failures from hierarchy failures, handling the subjective nature of instruction conflicts, and preventing degenerate always-refuse policies. Experiments on GPT-5 Mini-R show notable gains: System<>User conflict resolution improves from 0.84 to 0.95 (+0.11), TensorTrust (dev-user) from 0.76 to 0.91 (+0.15), and IH-Challenge over-refusal avoidance from 0.79 to 1.00 (+0.21). Practical benefits include better adherence to system-prompt safety policies without sacrificing helpfulness, and improved resistance to prompt injection by treating tool outputs as untrusted data rather than instructions.

Improving Instruction Hierarchy Capabilities in Frontier LLMs

This post is a Chinese-language infographic summarizing OpenAI research (published 2026-03-10) on training large language models to respect the instruction hierarchy — a robust priority system for resolving conflicts between instructions from different sources.

Core Concept: The Instruction Hierarchy

Priority order: System > Developer > User > Tool

1. System — safety policies; highest privilege 2. Developer — product constraints and application logic 3. User — explicit requests and task instructions 4. Tool — external data; an untrusted source

Training Method

IH-Challenge dataset: constructs conversations containing high/low-privilege instruction conflicts.
Objective scoring: Python scripts objectively verify whether higher-level constraints were obeyed.
Avoiding shortcuts: prevents the model from gaming the metric via over-refusal (rejecting everything).

Why It Is Hard

Confusion: execution failures are easily misjudged as hierarchy failures.
Subjectivity: instruction conflicts often involve nuanced judgment.
Degeneration: models easily learn a lazy "refuse everything for safety" strategy.

Experimental Results (GPT-5 Mini-R)

| Benchmark | Before | After | Gain | |---|---|---|---| | System <> User Conflict | 0.84 | 0.95 | +0.11 | | TensorTrust (dev-user) | 0.76 | 0.91 | +0.15 | | IH-Challenge (Overrefusal) | 0.79 | 1.00 (perfect over-refusal avoidance) | +0.21 |

Practical Impact

Safety & controllability: models better follow safety rules in system prompts and refuse violating requests without sacrificing helpfulness.
Prompt-injection resistance: tool output is treated as untrusted data rather than instructions, defending against malicious payloads embedded in tool results.

*Source: OpenAI Blog (2026-03-10), as translated on zhichai.net.*