English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI on Quranic Arabic

Forum topic · 小凯 · 2026-09-22

Summary

QuranicMMLU is a new benchmark (arXiv 2609.22038) for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Unlike existing Quranic benchmarks focused on general question answering and semantic retrieval, it probes specific linguistic competencies via a five-pillar taxonomy covering Phonology, Morphology, Syntax, Semantics, and Pragmatics, with 31 leaf categories ranging from tajwid recitation rules and root-and-pattern morphology to occasions of revelation and inter-surah coherence. Questions are stratified by Bloom's cognitive level and verse perplexity, scored by an LLM-as-judge pipeline with human review, yielding 980 human-audited items presented in both open-ended and multiple-choice formats. Evaluation of 12 systems shows Islamic-domain-specific models lead, but multiple-choice accuracy (avg 84%) consistently exceeds open-ended answer quality (avg 60%); the two rankings correlate strongly (Kendall tau=0.73), yet multiple-choice scores mask failures revealed only without answer options. The benchmark provides a rigorous, linguistically grounded framework for Arabic NLP evaluation in the Quranic domain.

Introduction

Research area: NLP Authors: Rawan El Ghali, Umm Kulsoom, Anas Madkoor, Dima Faris Alsaudi, Roaa Abdelmagid, Roaa Ibrahim, Raghad Mousa, Hamza Aljaji, Abdullah Khanafer, Abdallah Alkanani, Salah Feras Alali, Rawan Khaled Mohamed, Ehsaneddin Asgari Published: 2026-09-18 arXiv: 2609.22038

Abstract

We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quranic benchmarks center on general question answering and semantic retrieval, without probing specific linguistic competencies or stratifying by cognitive demand and verse difficulty.

We construct a five-pillar Quranic taxonomy spanning Phonology, Morphology, Syntax, Semantics, and Pragmatics, with 31 leaves covering phenomena from tajwīd and root-and-pattern morphology to occasions of revelation and inter-surah coherence. For each leaf we generate questions stratified by Bloom's cognitive level and verse perplexity, then have LLM as a judge to independently answer and score every item and route the annotations to manual review.

The resulting dataset comprises 980 human-audited questions, each presented in both open-ended and multiple-choice formats. Benchmarking 12 systems shows that Islamic-domain-specific models lead, but all systems achieve higher multiple-choice accuracy (average 84%) than open-ended answer quality (average 60%). The two rankings are highly consistent (Kendall τ=0.73), yet multiple-choice scores conceal failures that become exposed once answer options are removed.

QuranicMMLU therefore provides a rigorous, linguistically grounded framework for evaluating Arabic NLP in the Quranic domain.

Tags

#nlp#benchmark#arabic#quranic-arabic#generative-ai#llm-evaluation#linguistic-taxonomy#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178635078