Paper Overview
- Field: AI
- Authors: Gaetano Rossiello, Dharmashankar Subramanian
- Published: 2026-05-28
- arXiv: 2605.27571
- Apache Kafka for event-driven coordination
- Apache Flink for stream processing
- Large Language Models (LLMs) to implement specialized agents
- Problem: Traditional query-driven analytics cannot scale to the vast, evolving insight space of real-time data streams.
- Solution: A multi-agent LLM architecture that continuously hypothesizes, compiles, validates, and visualizes insights without manual query authoring.
- Coordination: Apache Kafka provides event-driven orchestration between agents.
- Execution: Apache Flink handles the underlying stream processing for generated analytics.
- Intelligence: LLMs serve as the reasoning engine for each specialized role in the pipeline.
- Safety & Observability: Typed intermediate artifacts enforce contracts between agents, enabling lineage, modularity, and safe dynamic execution.
- Use Cases: Demonstrated across retail, finance, and public-data scenarios.
- Impact: Marks a paradigm shift from query-driven analytics toward proactive, discovery-driven intelligence systems.
- Rossiello, G., & Subramanian, D. (2026). *Discovery Agents for Real-Time Analytics: Toward Proactive Insight Generation*. arXiv:2605.27571. https://arxiv.org/abs/2605.27571
Summary
Modern analytics systems are fundamentally reactive, requiring users to define queries over increasingly complex and continuously evolving data. In real-time streaming environments, this paradigm breaks down, as the space of potential insights becomes too large to enumerate manually.
The authors present a multi-agent architecture for autonomous insight discovery over real-time data streams. The system implements a continuous discovery loop in which agents:
1. Generate hypotheses 2. Compile them into executable analytics 3. Validate the generated artifacts 4. Produce visualizations and deployable applications
The architecture leverages:
A key contribution is a contract-driven design based on typed intermediate artifacts, which enables modularity, observability, lineage tracking, and safe execution of dynamically generated analytics.