SkillWrapper: Generative Predicate Invention for Task-Level Robot Planning

Forum topic · 小凯 · 2026-05-07

Summary

SkillWrapper, a system from Brown University and the Allen Institute for AI (arXiv 2511.18203), enables robots to autonomously invent symbolic predicates for long-horizon task planning. Instead of relying on expert-written PDDL domains, the system uses a vision-language model to contrast successful and failed skill executions in raw RGB images and propose interpretable predicates that explain the difference. An iterative loop of active data collection, contrastive predicate invention, and operator learning produces PDDL-style operators that plug into classical planners with formal soundness and completeness guarantees. In the Robotouille simulation benchmark, SkillWrapper solved 74.0% of easy and 40.0% of hard tasks and identified 100% of impossible tasks, outperforming ViLa-style direct VLM planning and fixed-predicate baselines on hard tasks. On real Franka Panda and bimanual Kuka robots (cup manipulation, sandwich making), learned models reached 86.7-98.5% VLM classification accuracy, with self-invented predicates classified more reliably than human-defined ones. Open-source VLMs like Qwen3-VL performed comparably to GPT-5. Limitations include full-observability assumptions, over-invention of predicates, and VLM query costs.

SkillWrapper: Generative Predicate Invention for Task-level Planning lets robots discover their own causal logic for unseen long-horizon tasks.

Paper facts

Title: SkillWrapper: Generative Predicate Invention for Task-level Planning
Authors: Ziyi Yang, Benned Hedegaard, Ahmed Jaafar, Yichen Wei, Skye Thompson, Shreyas S. Raman, Haotian Fu, Stefanie Tellex, George Konidaris, David Paulius, Naman Shah
Institutions: Brown University + Allen Institute for AI (AI2)
arXiv: 2511.18203 (v6, 2026-04-13)
Project page: https://yzylmc.github.io/skill-wrapper/

Core problem

Robots have skills (pick, place, open) but lack a causal model of when those skills apply. Traditional TAMP requires experts to hand-write all predicates and operators in PDDL domains, which breaks when environments or objects change. SkillWrapper's answer: let a vision-language model distill semantic predicates directly from raw RGB images, turning pixel-level perception into high-level symbolic logic.

Key idea: Generative Predicate Invention

Rather than using hand-defined predicates like OnTable(cup), the system invents them. It shows the VLM a pair of images — one successful and one failed skill execution — and asks what visual feature explains the difference. The VLM may propose ClearAbove(object). Each invented predicate has natural-language semantics and is validated for planning usefulness before being kept.

System architecture: a three-step loop (Algorithm 1)

1. Active data collection — foundation models propose exploratory skill sequences that maximize information gain, targeting boundary cases the current model does not yet understand. 2. Contrastive predicate invention — find transition pairs with the same skill but different outcomes; the VLM proposes a purely visually-decidable candidate predicate; a scoring function verifies it satisfies preconditions on successes while failing on failures. 3. Operator learning — learn PDDL-style operators (preconditions + effects) from data that plug into classical planners such as MetricFF. The loop repeats, progressively improving the model.

Theoretical guarantees

Unlike VLM-direct planners (e.g., ViLa), SkillWrapper offers formal proofs of soundness (no plans executed when preconditions are unmet) and completeness (any solvable task yields a solution), conditioned on a coverage property the invention algorithm progressively approaches.

Experiments

Simulation: Robotouille (burger making)

| Method | Easy solve | Hard solve | Impossible identified | |---|---|---|---| | Expert Ops (oracle) | 81.0% | 58.1% | 100% | | SkillWrapper | 74.0% | 40.0% | 100% | | System Predicates (8 fixed) | 79.0% | 22.0% | 42.0% | | No Heuristic | 76.0% | 24.0% | 80% | | FM Invent (unfiltered) | 23.3% | 1.7% | 63.3% | | ViLa | 46.0% | 13.9% | 20.0% | | Random exploration | 4.0% | 0% | 100% |

Key findings: SkillWrapper far exceeds all baselines on hard tasks (even beating the 8 fixed expert predicates, which under-cover complex scenes); it identifies 100% of impossible tasks (direct soundness evidence); and achieves lower planning budgets.

Real robots

Franka Emika Panda (single arm): manipulation of cups, plates, sponges, teapots; learned 6 predicates with 1111 grounded instances; 86.7% VLM classification accuracy; failures on gripper_empty/holding with sponges whose color resembles the background.
Bimanual Kuka: sandwich making with peanut butter, knife, bread; 6 black-box skills (LeftArmPick, RightArmPick, Open, Scoop, Spread, Drop); many dead ends (dropped knife unrecoverable, peanut-butter-coated bread uncleanable); improved across iterations, eventually surpassing all baselines; 98.5% classification accuracy.

Key insight: foundation models ground predicate semantics more effectively when the abstractions are self-generated rather than human-defined.

VLM reliability analysis

Domain shift: on Franka, stacked_on accuracy dropped from 100% to 73.3% after changing object instances/configurations (small plates misclassified as saucers); Kuka predicates mostly stayed at 96-100%.
Alternative VLM: Qwen3-VL-235B performs comparably to GPT-5 — more stable on holding (100% vs 60%), slightly weaker on coated (90% vs 100%). The system is not tied to closed-source models.

Honest failure modes

1. Predicate over-invention: an extra predicate like on_cutting_board(item) fragmented the data, causing operator overfitting. 2. Semantic redundancy: synonyms/antonyms are not filtered, burdening VLM classification. 3. Perception error propagation: even at 86-98% accuracy, a single wrong predicate can fail an entire plan due to symbolic rigidity. 4. Objects deviating from visual expectations: small plates mistaken for saucers.

Why it matters

Bridges the perception-symbol gap: the conversion from pixels to symbols becomes automatic, interpretable, and theoretically guaranteed.
Cuts human engineering: no expert-written PDDL domains; robots figure out what their skills can do.
Fundamentally different from VLM-direct planning: ViLa-style methods accumulate errors, lack correctness guarantees, and are black-box. SkillWrapper learns a model first, then plans with classical planners.
Interpretable: learned predicates are human-readable (ClearAbove(x), Coated(Knife)), so you can ask why a pick is deemed impossible.

Limitations and future work

1. Assumes full observability; partially observable scenes (occluded objects) are not handled. 2. Balancing predicate invention against data collection; finer scoring functions needed. 3. VLM cost: every predicate check requires a query; long-horizon tasks may need many. 4. Discrete symbols only; continuous parameters (e.g., grasp poses) are out of scope. 5. Exploration could incorporate information-theoretic methods beyond FM heuristics.

Positioning vs related work

| Direction | Representative work | Difference from SkillWrapper | |---|---|---| | VLM-direct planning | ViLa, SayCan | No symbolic model, no correctness guarantees | | Hand-written TAMP | PDDL + motion planner | Requires expert-defined domains | | Automatic domain learning | PDDLLM | Derived from demonstrations, not active exploration | | Neuro-symbolic | Neural Logic Machines | Representations not interpretable | | Contrastive/concept learning | Various | Not robot-skill focused, no planning guarantees |

References

arXiv: https://arxiv.org/abs/2511.18203
Project page: https://yzylmc.github.io/skill-wrapper/
RSS 2025 Workshop version: https://rss25-roboreps.github.io/papers/26_SkillWrapper_Autonomously_L.pdf
ICRA 2025 Workshop version: https://dyalab.mines.edu/2025/icra-workshop/16.pdf