Paper Overview
- Field: AI
- Authors: Gaetano Rossiello, Dharmashankar Subramanian
- Published: 2026-05-28
- arXiv: 2605.27571
- Problem: Reactive, query-driven analytics cannot scale to the vast insight space of real-time streaming data.
- Approach: A continuous discovery loop where LLM-powered agents hypothesize, compile analytics, validate outputs, and generate visualizations/deployable apps.
- Stack: Apache Kafka (event-driven coordination), Apache Flink (stream processing), LLMs (specialized agents).
- Key contribution: Contract-driven design using typed intermediate artifacts, supporting modularity, observability, lineage, and safe execution.
- Evaluation: Demonstrated on retail, finance, and public data use cases.
Abstract
Modern analytics systems are fundamentally reactive, requiring users to define queries over increasingly complex and continuously evolving data. In real-time streaming environments, this paradigm breaks down, as the space of potential insights becomes too large to enumerate manually. We present a multi-agent architecture for autonomous insight discovery over real-time data streams. The system implements a continuous discovery loop in which agents generate hypotheses, compile them into executable analytics, validate generated artifacts, and produce visualizations and deployable applications. The architecture leverages Apache Kafka for event-driven coordination, Apache Flink for stream processing, and large language models to implement specialized agents. A key contribution is a contract-driven design based on typed intermediate artifacts, enabling modularity, observability, lineage, and safe execution of dynamically generated analytics. Use cases across retail, finance, and public data demonstrate a paradigm shift from query-driven analytics to proactive, discovery-driven systems.
Key Points
*Auto-collected on 2026-05-29.*