How AI Assistance Affects Human Skill Development: Evidence from a Controlled Logic-Puzzle Experiment

Forum topic · 小凯 · 2026-08-26

Summary

A study by Shang Wu, Catarina G. Belem, and Shuyuan Fu (arXiv:2508.17621, August 2025) investigates how on-demand AI assistance affects human skill development over time. In a controlled logic-puzzle experiment, participants completed tasks before, during, and after AI was available, with the cost of requesting AI help experimentally varied. Lower request costs led to more frequent AI use. Participants who relied on AI during the access phase performed worse once assistance was removed, and their unassisted performance was overestimated when predicted from earlier AI-assisted results. Using a Bayesian latent ability model, the authors separated initial ability, post-AI ability, and participant-specific skill change, and found that greater independent problem-solving effort was associated with larger latent ability gains. The findings suggest that skill development is weaker when AI assistance substitutes for independent reasoning.

Paper Overview

Field: Machine Learning
Authors: Shang Wu, Catarina G. Belem, Shuyuan Fu
Published: 2025-08-26
arXiv: 2508.17621

Abstract

While AI assistance can improve human task performance in the short term, it may also undermine the development of skills in the longer term. We examine this tension in a controlled logic-puzzle experiment involving on-demand AI assistance, where participants complete tasks before, during, and after AI is available. By experimentally varying AI request costs, we find that lower-cost assistance induces more frequent AI use. We also find that participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed, and their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance. We use a Bayesian latent ability model to separate initial ability, post-AI ability, and participant-specific skill change.

Key Findings

Lower costs for requesting AI assistance lead to more frequent AI use during the AI-access phase.
Participants who used AI assistance performed worse on the task after assistance was removed.
Predicting later unassisted performance from earlier AI-assisted performance systematically overestimates a participant's actual ability.
The Bayesian latent ability model disentangles initial ability, post-AI ability, and participant-specific skill change, while estimating the relationship between independent reasoning and skill development during the AI-access phase.
Greater independent problem-solving effort is associated with larger latent ability gains, consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning.

Implications

The results highlight a short-term vs. long-term trade-off: AI assistance boosts immediate task performance but may impair lasting skill acquisition when it replaces independent problem solving. This has implications for how AI tools should be deployed and priced in educational and training contexts.