WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Benchmarks

• arXiv: 2609.12171
• Authors: Amey Varhade, Ananya Sutradhar, Ravishankar Krishnaswamy, Navin Goyal
• Published: 2026-09-15

Abstract

Enterprise settings provide a challenging environment for question-answering agents, which often rely on Retrieval-Augmented Generation, Deep Research (DR), and related techniques. Much of this challenge comes from the complexity of enterprise data: information is often spread across evolving and potentially conflicting emails, chat messages, documents, and other artifacts. Existing benchmarks typically have limited real-world complexity, short-form responses, and unnatural queries, so they often fail to capture the challenges of enterprise settings.

Key points

• Introduces an automated pipeline for generating synthetic email datasets reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data.
• Simulates enterprise projects spanning months, involving up to 25 employees in distinct roles interacting over time.
• Dataset design emphasizes ambiguity, information dispersion, and naturally arising queries, better mirroring real enterprise environments than prior benchmarks.
• Evaluation with frontier models on standard agent baselines found average total scores below 80% across all queries on every dataset.
• Findings highlight that enterprise deployment remains an open challenge and underscore the importance of realistic, high-complexity evaluation data for building stronger enterprise deep-research systems.

*Originally posted on zhichai.net; auto-collected 2026-09-15.*