How AI Assistance Affects Human Skill Development: Evidence from a Controlled Logic-Puzzle Experiment

Forum topic · 小凯 · 2026-08-26

Summary

A new arXiv paper (2508.17621) by Shang Wu, Catarina G. Belem, and Shuyuan Fu examines whether AI assistance that boosts short-term task performance may undermine long-term skill development. The authors ran a controlled logic-puzzle experiment in which participants completed tasks before, during, and after AI assistance was available, and experimentally varied the cost of requesting AI help. Lower-cost assistance induced more frequent AI use. Participants who requested AI help during the AI-access phase performed worse after assistance was removed, and their later unassisted performance was overestimated when predicted from earlier AI-assisted performance. Using a Bayesian latent ability model that separates initial ability, post-AI ability, and participant-specific skill change, the authors find that greater independent problem-solving effort is associated with larger latent ability gains, consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning. The findings carry implications for education, training, and the design of AI assistance pricing and availability.

Paper Overview

Field: Machine Learning
Authors: Shang Wu, Catarina G Belem, Shuyuan Fu
Published: 2025-08-26
arXiv: 2508.17621

Abstract

While AI assistance can improve human task performance in the short term, it may also undermine the development of skills in the longer term. The authors examine this tension in a controlled logic-puzzle experiment involving on-demand AI assistance, where participants complete tasks before, during, and after AI is available.

By experimentally varying AI request costs, they find that lower-cost assistance induces more frequent AI use. They also find that participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed, and their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance.

The authors use a Bayesian latent ability model to separate initial ability, post-AI ability, and participant-specific skill change, while estimating the relationship between independent reasoning and skill development during the AI-access phase. Results show that greater independent problem-solving effort is associated with larger latent ability gains — consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning.

Key Takeaways

Lower request costs lead to more frequent AI use.
Reliance on AI assistance during access predicts worse performance once assistance is removed.
AI-assisted performance is an inflated predictor of later unassisted performance.
Independent problem-solving effort correlates with greater latent skill gains.