Introduction
Research area: NLP Authors: Rawan El Ghali, Umm Kulsoom, Anas Madkoor, Dima Faris Alsaudi, Roaa Abdelmagid, Roaa Ibrahim, Raghad Mousa, Hamza Aljaji, Abdullah Khanafer, Abdallah Alkanani, Salah Feras Alali, Rawan Khaled Mohamed, Ehsaneddin Asgari Published: 2026-09-18 arXiv: 2609.22038
Abstract
We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quranic benchmarks center on general question answering and semantic retrieval, without probing specific linguistic competencies or stratifying by cognitive demand and verse difficulty.
We construct a five-pillar Quranic taxonomy spanning Phonology, Morphology, Syntax, Semantics, and Pragmatics, with 31 leaves covering phenomena from tajwīd and root-and-pattern morphology to occasions of revelation and inter-surah coherence. For each leaf we generate questions stratified by Bloom's cognitive level and verse perplexity, then have LLM as a judge to independently answer and score every item and route the annotations to manual review.
The resulting dataset comprises 980 human-audited questions, each presented in both open-ended and multiple-choice formats. Benchmarking 12 systems shows that Islamic-domain-specific models lead, but all systems achieve higher multiple-choice accuracy (average 84%) than open-ended answer quality (average 60%). The two rankings are highly consistent (Kendall τ=0.73), yet multiple-choice scores conceal failures that become exposed once answer options are removed.
QuranicMMLU therefore provides a rigorous, linguistically grounded framework for evaluating Arabic NLP in the Quranic domain.