RAGalyst: When AI Evaluation Frameworks Become Gatekeepers for Professional Domains

Opening: When AI Starts "Hallucinating"

Imagine standing at the inspection site of a century-old bridge, holding an AI-generated inspection report. The report confidently states: "Based on historical data, the bridge's main girder structure is intact and requires no maintenance." But the original inspection records show the exact opposite—the main girder has developed fine cracks and urgent repairs are needed. This phenomenon of "confidently making things up" has a professional term in AI: hallucination.

This is not science fiction but a serious reality facing large language models (LLMs). These models are like erudite scholars who occasionally speak carelessly—fluent on any topic, yet capable of packaging completely wrong information as fact. In casual conversation this may be harmless, but in safety-critical domains like military operations, cybersecurity, and bridge engineering, one wrong answer can mean catastrophe.

Retrieval-Augmented Generation (RAG) emerged as a solution—like giving AI a "living dictionary" to consult in real time. But is that enough? If the dictionary is full of jargon and special formats, if the AI doesn't know how to "look things up," or takes things out of context, the results can still be disastrous. And how do we know whether the AI truly understands specialized material?

This is the background of RAGalyst, an automated evaluation system developed by a University of Houston research team—like a strict "examiner" that tests not only the AI's answers but also its "study methods." This article explores how the framework drives an evaluation revolution across three very different professional domains: military operations, cybersecurity, and bridge engineering.

Chapter 1: RAG's "Triple Dilemma"

Dilemma 1: The "Tower of Babel" of Expert Knowledge

Military documents are full of abbreviations like "R&S team" (Reconnaissance & Surveillance), "LOA" (Line of Advance), and "ORP" (Objective Rally Point); cybersecurity logs contain dense IP addresses, ports, and protocol types; bridge engineering reports include structural mechanics parameters and material performance indicators. This out-of-distribution content never appeared in LLM training corpora.

Document structure also differs: bridge inspection records require understanding degradation trends across multiple reports—like reading a long novel where earlier chapters matter—while cybersecurity logs are like short detective stories where a few lines of packet captures reveal an attack. This means no one-size-fits-all "best practice" exists.

Dilemma 2: Evaluation as "Blind Men Touching an Elephant"

Early methods relied on heuristic metrics like BLEU and ROUGE, which only check whether words were "copied correctly"—a correct answer rephrased scores zero, while keyword-stuffed nonsense may score high. The later LLM-as-a-Judge approach seems smarter but has a fatal flaw: the judge's rating criteria were never calibrated against human experts. RAGAS enabled automation, but its generated QA datasets were of questionable quality.

Dilemma 3: The "Chicken-and-Egg" Dataset Paradox

Evaluating RAG requires high-quality QA datasets, but these rarely exist in specialized domains: military documents are classified, cybersecurity data is sensitive, and bridge engineering materials are scattered with high annotation costs. Manual annotation is expensive and constrained by confidentiality; fully automated pipelines like RAGAS lack strict quality filtering, producing noisy datasets.

Chapter 2: RAGalyst's Three Tools

1. Meticulous Document Preprocessing

RAGalyst uses LangChain tools to parse PDF, Markdown, and plain text. The key challenge is chunking—splitting long documents into retrieval-friendly pieces. Too small loses context; too large dilutes relevance. The team found optimal chunk sizes differ by more than 20% across domains. They adopted OpenAI file-search defaults: 800 tokens per chunk with 400-token overlap, then vectorized and stored chunks in a vector database.

> Note: A token is the basic unit of text—a word, character, or part of a word. In AI, tokens are the model's "building blocks" of text.

2. Agentic QA Generation: A "Two-Person Act"

RAGalyst's most elegant design: one LLM plays the user, another the expert, jointly generating high-quality QA pairs in three steps:

1. Context sampling: Randomly sample document chunks as the "script" for QA trios (QAC). 2. Role-play: The user agent poses a specific, unambiguous question; the expert agent generates a ground-truth answer grounded in the context. 3. Quality filtering: Generated pairs must pass three checkpoints with thresholds—Answerability (the question is fully answerable from context), Faithfulness (the answer doesn't deviate from the source), and Answer Relevance (the answer is on-topic).

The team used GPT-4o-mini to generate optimized prompts via the DSPy framework's COPRO and MIPROv2 optimizers; MIPROv2 incorporates Bayesian optimization to continually refine evaluation criteria.

> Note: DSPy is a declarative framework for automatically optimizing LLM prompts. COPRO refines instructions word by word; MIPROv2 optimizes instructions and selects the best examples; LabeledFewShot picks the most representative examples from a dataset.

3. Human-Aligned LLM-as-a-Judge

RAGalyst optimizes two core metrics:

• Answer Correctness: Continuous scoring (0.0–1.0) that rewards semantic equivalence rather than verbatim copying. On the STS-B semantic similarity benchmark, optimized prompts brought GPT-4o-mini's Spearman correlation with human labels to 0.894, far exceeding RAGAS's 0.843 and cosine similarity's 0.622.
• Answerability: Validated on SQuAD 2.0, Gemini 2.5 Pro achieved a correlation of 0.752 with human judgment, ensuring generated questions don't rely on external knowledge.

This human-alignment mechanism gives the AI judge a pair of "human glasses" so its scoring criteria match those of human experts.

Chapter 3: Domain "Wars" in the Lab

Military Operations: A Maze of Abbreviations and Protocols

Using the TC 3-21.76 manual as an example, generated QA pairs captured the over-specific style of military writing: "What should an R&S team do upon enemy contact during reconnaissance?" Answer: "Must return to the RP." This over-specificity becomes an advantage—orders must be clear and unambiguous. Gemini-2.5-flash achieved faithfulness of 0.95, strictly adhering to retrieved context.

Cybersecurity: The Eye of an Information Storm

A few lines of packet captures can contain all the evidence of an attack. Most embedding models performed weakly here, but the text-embedding-3 family bucked the trend—possibly due to heavy exposure to cybersecurity data during training. GPT-4.1-nano scored best on answer relevance at 0.951; verbosity, which is usually penalized, became an advantage here.

Bridge Engineering: An Epic of Time Series

Bridge inspection records span decades, demanding long-context dependency handling. Qwen3-Embedding-8B achieved top recall in this domain. Interestingly, the smaller Qwen3-4B could match or beat the 8B model—a counterintuitive truth: bigger isn't always better; what matters is whether the model has learned the domain's "dialect."

Chapter 4: When Answers Fall Short

The team used GPT-5 to dissect 1,500 QA pairs and identify RAG's "Achilles' heel":

• Over-specificity (71.3% of failures): QA generation sees one chunk and produces a concise answer; in RAG, the model sees 10 chunks and stuffs everything in, over-elaborating. Ask "What is an apple?" and the expected answer is "a fruit," but the RAG system adds color, family, vitamin C, origins, and global cultivation—correct, but mismatched with the ground truth's conciseness.
• Information not retrieved (13%): Relevant information was lost at the retrieval stage; heatmaps show "not retrieved" issues spike when answer correctness drops below 0.75.
• Contextual inconsistency (8.3%): The model retrieved correct information but over-interpreted, adding meaning absent from the source—revealing RAG's deep contradiction: retrieval provides facts, but generation can still "improvise."

Chapter 5: No Silver Bullets

• Embedding models have "domain personalities": MTEB leaderboard star gemini-embedding-001 performed mediocrely on military and bridge domains, outclassed by open-source Qwen3. Even within one family, small models beat larger ones (Qwen3-4B matched the 8B on cybersecurity). Parameter scale is not a panacea; domain adaptation is key.
• LLM "domain biases": Gemini leads on answer correctness and faithfulness; GPT on answer relevance. Notably, closed-source models have no absolute advantage—open-source Qwen3 and Gemma3 can rival GPT-4 and Gemini in specific domains.
• Hyperparameter "Schrödinger optimum": 3–5 retrieved chunks maximize answer correctness—too few misses information, too many dilutes focus—and the optimum varies by domain.

Chapter 6: Do AI Judges Show Self-Preference?

Recent research suggests LLMs favor their own outputs. The team cross-evaluated datasets generated by GPT-4o-mini, Gemini-2.5-flash, and Qwen3-30B in the military domain. Result: almost no evidence of self-preference. The only exceptions were marginal—Qwen3-30B rated its own datasets slightly higher on answer correctness, and Gemini-2.5-flash showed slight faithfulness preference for its own data; GPT-4o-mini was entirely impartial. This "de-biased" property is crucial for evaluation objectivity.

Chapter 7: Takeaways

1. Evaluation as a design guide: RAGalyst shifts evaluation from post-hoc checking to design-stage simulation—like BIM in construction, revealing design flaws before breaking ground. 2. Automation with "human warmth": Via DSPy optimizers, future AI evaluation doesn't replace humans—it learns from them. 3. Domain specificity wins: There is no universal solution; each domain needs its own "dialect dictionary" and scoring criteria. 4. The power of open source: Well-tuned open-source models can build professional-grade RAG systems without huge budgets.

Chapter 8: The Future "Evaluation Universe"

• Multimodal evaluation: Assessing images, video, and sensor data alongside text—e.g., a bridge RAG system understanding both reports and crack photos.
• Dynamic evaluation: Continuous re-evaluation on every data update, like CI systems, preventing performance drift.
• Causal evaluation: Building causal graphs to pinpoint whether retrieval, ranking, or generation failed.
• Adversarial evaluation: Red-team models generating "trap questions" to test robustness.

Epilogue: Teaching AI to Speak Responsibly

With a RAGalyst-evaluated system, the bridge report reads: "According to inspection reports 2019–2023 (documents BE-2019-001 to BE-2023-087), the main girder shows 0.3mm cracks at L/4 span, expanding at 0.05mm/year. Per Highway Bridge Maintenance Code Section 5.2.1, immediate repair procedures are recommended."

Every claim is traceable and verifiable. Imperfect, but honest. As the paper's authors state: "These findings underscore the necessity of systematic evaluation frameworks that enable practitioners to uncover domain-specific trade-offs and make informed design choices for building reliable RAG systems."

In the evolution from "making things up" to "telling the truth," RAGalyst may be a small milestone—but it points toward a larger future: one where AI engages in rigorous dialogue with human knowledge.

---

References

1. Gao, J., Pham, Q. H., Varghese, S., Saurav, S., & Hoskere, V. (2025). *RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAG*. arXiv:2511.04502v1 [cs.CL]. 2. Borgeaud, S., et al. (2022). *Improving language models by retrieving from trillions of tokens*. International Conference on Machine Learning. 3. Lewis, P., et al. (2020). *Retrieval-augmented generation for knowledge-intensive NLP tasks*. Advances in Neural Information Processing Systems. 4. Es, S., et al. (2023). *RAGAS: Automated Evaluation of Retrieval Augmented Generation*. arXiv preprint arXiv:2309.15217. 5. Rajpurkar, P., et al. (2018). *Know what you don't know: Unanswerable questions for SQuAD*. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.