Don't Use a PhD as a Security Guard: Proactive Agents Don't Need an LLM to Decide When to Wake Up - Small Model Is 83x Faster and More Accurate

Forum topic · ✨步子哥 · 2026-05-29

Summary

This article critiques the prevailing design of proactive AI agents, in which every user event is serialized into text and fed to a large language model (LLM) that decides whether to take action - a call that results in "do nothing" 99% of the time. The author argues this flattens naturally structured OS event data (actor, verb, object, timestamp) into text, losing information and wasting compute. The discussed paper proposes a two-tier architecture: a lightweight temporal graph learning (TGL) model acts as a gatekeeper, computing a trigger probability and a routing score in ~11ms per event, while the LLM is only invoked when triggered to generate user-facing natural language responses. Across 14 backbone models, TGL improved F1 by an average of +16.7 (up to +46.0). On GPU servers it runs 4-7x faster than the fastest LLM-as-trigger setup; on consumer laptops, 12-83x faster. At roughly 220 MiB (BF16), the model fits on-device alongside privacy-sensitive event streams, removing the need to upload user data. The broader lesson: small specialized models should handle perception and filtering, while LLMs handle reasoning and generation. Limitations include schema-specific training and simulation-based evaluation.

Your phone's AI assistant does something remarkably wasteful every moment of every day: it translates your life into text, then reads that text back to itself.

You open an app - it writes "user opened WeChat." You receive a message - it writes "user received a message from Zhang San." You switch songs - it writes "user switched the playlist." Then it stuffs all this text into a large language model and asks: "Should I proactively do something?"

99% of the time, the LLM says: "No."

This is how today's "proactive agents" work: every single event triggers an LLM call so the LLM can decide whether to act. It sounds intelligent, but it's absurd - like hiring a PhD to be on 24-hour standby just so they can glance at "user scrolled the screen again" every second and say "nothing to see here."

The paper asks a sharp question: do you really need an LLM to make this decision?

The Core Problem: Structure Gets Flattened

The key insight: user activity is naturally structured data, not text.

Operating systems record user behavior as (actor, verb, object, timestamp) tuples - e.g., (user, open, WeChat, 14:30:02). These tuples are already organized as graph structures inside the OS: who did what to what, when, and how events relate to each other.

But what do current proactive agents do? They flatten this graph into text, then make the LLM reconstruct the structure from the text. It's like tearing a map into pieces, arranging the pieces into a sentence, and asking someone to redraw the map from that sentence - redundant, and information is inevitably lost.

The paper's core claim: treat always-on signals as graph updates, not text.

The Solution: A Small Model as Gatekeeper, a Large Model as Writer

The proposed architecture is remarkably simple:

Tier 1: A temporal graph learning (TGL) model as gatekeeper. A tiny graph neural network that consumes the structured event stream and outputs two things:

Trigger probability: is this event worth waking the LLM?
Routing score: if yes, which downstream agent should handle it?

One forward pass, done in 11 milliseconds.

Tier 2: The LLM is only called when triggered. It receives not a raw text stream, but a compact, structured "handoff package" containing only relevant context. The LLM's sole job is turning structured information into fluent, user-facing natural language.

An analogy: TGL is the building's security guard; the LLM is the front-desk receptionist. The guard watches the monitors (event stream) 24/7, decides who needs reception (triggering), and only brings those people to the front desk (routing). The receptionist doesn't need to watch every camera feed - only to talk with the people brought over.

Results: The Small Model Crushes the Large Model

The results are striking:

Across 14 backbone models, TGL improved F1 in every single case, by an average of +16.7 and up to +46.0. All of them, no exceptions.

The speed comparison is even more dramatic:

On a GPU server: TGL takes 11.13 ms per event, 4-7x faster than the fastest LLM-as-trigger configuration
On a consumer laptop: TGL takes 13.99 ms per event, 12-83x faster than the fastest LLM-as-trigger configuration

Memory footprint: the TGL model is only ~220 MiB (BF16 precision) and can be deployed on-device, co-located with the privacy-sensitive activity streams it consumes. User data doesn't need to be uploaded to the cloud.

Trigger stability: a single TGL checkpoint delivered the strongest trigger AUC and most stable deployment thresholds across all tested scenarios. By contrast, LLM-as-trigger performance fluctuated wildly across scenarios.

Why Is the Small Model Better?

This counterintuitive result has deep causes:

1. Structured data is naturally suited to graph models. The (actor, verb, object, timestamp) structure of event streams maps directly onto graph nodes and edges. Graph neural networks natively handle this; LLMs must first "understand" the implicit structure buried in text.

2. Trigger decisions are pattern matching, not reasoning. Deciding whether "user opened Alipay 30 seconds after opening WeChat" is worth triggering is fundamentally sub-pattern matching on a graph. It doesn't need LLM reasoning - just efficient structured pattern recognition.

3. LLM "understanding" is wasted here. LLMs excel at generating fluent text and complex reasoning, but triggering needs neither. Using an LLM for this is like using a sports car for deliveries - it works, but it's absurdly expensive.

4. Latency is fatal. Proactive agents must respond in real time. LLM inference latency (even a single forward pass) becomes a bottleneck during dense event activity. TGL's 11 ms vs. an LLM's 50-900 ms is an order-of-magnitude gap.

The Bigger Lesson: Not Every AI Problem Needs an LLM

The implications go far beyond proactive agents:

LLMs are general-purpose reasoning engines, not omnipotent ones. Many AI applications need not "natural language understanding" but "efficient processing of structured data." Flattening structured data into text for an LLM is an architectural anti-pattern.

The right division of labor: small models for perception and filtering, large models for reasoning and generation. This isn't new - classic two-stage detectors (like Faster R-CNN) were designed exactly this way: a lightweight network proposes candidates, a heavyweight network classifies them. The idea was forgotten in the LLM era because everyone fell in love with the elegance of "one end-to-end giant model."

End-to-end doesn't mean one model does everything. True end-to-end is end-to-end optimization of the whole system, not cramming every function into a single model. When your system has clear "perception - decision - execution" layers, using models of different sizes and types for different layers is the right engineering choice.

An Honest Assessment

The paper's limitations are clear: the TGL model must be trained for specific event stream formats and doesn't generalize zero-shot the way an LLM does. If the event schema changes, TGL may need retraining. Moreover, the experiments were primarily conducted in simulated environments; real-world deployment remains unvalidated.

But the direction is right. While the entire AI community races toward "bigger and more general," this paper reminds us: sometimes small and specialized is the correct answer. Not every problem needs GPT-4 - sometimes a 220MB model is enough, and it does the job better.

---

Paper link: https://arxiv.org/abs/2605.30152

Authors: Xiaoze Liu, Ruowang Zhang, Amir H. Abdi, Michel Galley, Zhikai Chen, Siheng Xiong, Xiaoqian Wang, Jing Gao