Paper Overview
Field: AI Authors: Gaetano Rossiello, Dharmashankar Subramanian Published: 2026-05-28 arXiv: 2605.27571
Abstract
Modern analytics systems are fundamentally reactive, requiring users to define queries over increasingly complex and continuously evolving data. In real-time streaming environments, this paradigm breaks down, as the space of potential insights becomes too large to enumerate manually. We present a multi-agent architecture for autonomous insight discovery over real-time data streams. The system implements a continuous discovery loop in which agents generate hypotheses, compile them into executable analytics, validate generated artifacts, and produce visualizations and deployable applications. The architecture leverages Apache Kafka for event-driven coordination, Apache Flink for stream processing, and large language models to implement specialized agents. A key contribution is a contract-driven design based on typed intermediate artifacts, enabling modularity, observability, lineage, and safe execution of dynamically generated analytics. We demonstrate the approach through use cases in retail, financial, and public data domains, illustrating a paradigm shift from query-driven analytics to proactive, discovery-driven systems.
Key Takeaways
- From reactive to proactive: The paper argues that manual query definition cannot scale to real-time streams, motivating autonomous insight discovery.
- Continuous discovery loop: Agents generate hypotheses, compile them into executable analytics, validate outputs, and produce visualizations and deployable applications.
- Infrastructure: Apache Kafka handles event-driven agent coordination; Apache Flink performs stream processing; LLMs power the specialized agents.
- Contract-driven design: Typed intermediate artifacts provide modularity, observability, lineage tracking, and safe execution of dynamically generated analytics.
- Evaluation domains: Retail, financial, and public data use cases demonstrate the shift toward discovery-driven analytics.