A Dual-Dimensional LLM Framework for Automated Item Similarity Analysis in Large-Scale Assessments
Research area: AI Authors: Jing Huang, Jihong Zhang, Hua-Hua Chang Published: 2026-08-25 arXiv: 2608.24825
Overview
The rapid expansion of large-scale assessments and the growing adoption of automatic item generation have intensified concerns about incidental content redundancy, where construct-irrelevant elements such as wording or contextual framing become unintentionally repetitive across items. Traditional similarity metrics like BLEU or cosine similarity often fail to capture the nuanced structural and semantic layers that drive perceived redundancy simultaneously.
This study proposes a dual-dimensional framework for Automated Item Similarity Analysis (AISA) powered by Large Language Models (LLMs), operationalizing similarity through Structured Decomposition and Semantic Relatedness.
Key Findings
- Psychometric validation indicates that LLM-derived metrics align more closely with indicators of construct-irrelevant local dependence and produce more coherent item parameter groupings than traditional text-based similarity measures.
- The framework was further evaluated in the context of computerized adaptive testing (CAT).
- Simulations show that incorporating LLM-based similarity constraints into item selection improves estimation stability and reduces bias, with minimal efficiency trade-offs, outperforming constraints based on traditional metrics.
- These findings highlight the potential of LLM-driven AISA to support scalable item bank curation, content-aware test assembly, and experience-sensitive adaptive testing across diverse assessment contexts.