English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

First Day on the Job: When AI Is the Most Proactive Employee — Microsoft's AutoBuddy Paper Explained

Forum topic · 小凯 · 2026-11-10

Summary

A new Microsoft paper, AutoBuddy: Evaluating Proactive, Always-On Agents on Task Discovery and Execution in a Simulated Workplace (arXiv:2611.12393), tests how well AI agents can act like proactive employees who must find their own work. Researchers built a self-consistent simulated company with six months of emails, chats, calendars, files, and business-system records, from which 120 real tasks naturally emerge. Seventeen LLMs were scored on three stages: task discovery, task initiation, and execution. Results reveal a 'failure migration' pattern: weak models miss most tasks entirely (Claude Haiku 4.5 missed ~59.7 tasks per run), mid-tier models find tasks but frequently stop to ask for confirmation, and top models like Claude Opus 5.5 (0.698 end-to-end) find nearly everything but fail at precise execution, especially business-system operations that require exact record changes. Discovery and execution quality correlate weakly, showing they are separate capabilities. Key failure cases include answers sent to wrong channels, wrongly merged requests causing confirmation loops, and mistaking 'already replied' for 'already done'. The paper argues proactivity is a multi-stage pipeline, not a single skill, with major implications for enterprise AI agent design, evaluation, and the boundaries of autonomy.

First Day on the Job: When AI Becomes the Most Proactive Employee in the Company

*A breakdown of Microsoft's paper "AutoBuddy: Evaluating Proactive, Always-On Agents on Task Discovery and Execution in a Simulated Workplace" (arXiv:2611.12393)*

The Monday-Morning Inbox Problem

Imagine joining a new company. At 8 a.m. on Monday you open your computer to a stranger's inbox: 347 emails, 50+ chat channels, a calendar full of meetings, an unfamiliar file repository. Nobody gives you onboarding. Your job description is one sentence: "Figure out what needs doing, and do it."

Some emails ask for your help; some are things you promised last week; some were already answered; some look addressed to you but are actually for the colleague next door. Some tasks hide in a group-chat message from three days ago with no @mention; some deadlines have passed; some require operating business systems you've never seen. And most workspace content is noise — meeting minutes CCs, project updates, weekend party invites. A reliable agent must separate signal from noise without treating everything that looks like a task as a task. Treating an answered email as a to-do is like reacting to an already-snoozed alarm: not helpful, just disruptive.

Microsoft's new paper turns this experience into a rigorously scored experiment. Its conclusion in one line: AI can find the work, but can't finish it. The strongest models discover nearly all tasks yet lose most of their score at execution; the weakest don't even find the work. The key insight: discovering tasks and completing them are two fundamentally different capabilities.

Why This Matters: From "Obey Orders" to "Act Proactively"

Nearly all current agent benchmarks — WebArena, SWE-bench, tool-use suites — share a hidden assumption: tasks are given. An instruction is provided, and the agent's job is to execute it. But the real world isn't an exam. Tasks are buried in noise: emails, chat messages, calendar invites, file versions. A *proactive agent* must enter your environment, observe what's happening, identify what you need to do but haven't done, and — crucially — do it rather than ask.

This requires a kind of "workplace awareness": understanding your role, organizational relationships, and task lifecycles (has this request been answered? when is the deadline?). It also requires intent inference — "did you look at that thing we discussed?" is not a well-specified task. Passive evaluation frameworks fail here because they assume clear task boundaries, complete instructions, and explicit success criteria. This isn't a harder exam — it's a different game.

Building a Company

To evaluate proactive agents scientifically, Microsoft built a generator that constructs a simulated organization from a short description (industry, headcount, timespan, business systems). It produces a dated temporal graph containing employees and hiring histories, projects, external collaborators, events (experiment results, financial transactions, reports, collaborations), and the resulting emails, chats, calendar invites, and file versions.

The key property: all facts are mutually consistent. If an email says "experiment B had the lowest loss," the research system shows exactly that, and the training data confirms it used more data. Every fact has a source.

Tasks emerge naturally from event logic. Example: three training runs complete; run B has the lowest loss but used a different amount of data; lab policy only allows comparing runs with equal data volumes. So the correct decision is to promote run A — the lowest-loss *comparable* run. This obligation is derived from the world, not hand-written.

Scoring is entirely based on world state — not LLM judgment. What did the agent change? Which emails did it send? Does each change match the world's correct outcome? Not "looks right" — "is right."

The final evaluation world simulates six months of organizational life (March 15 – September 15, 2026), producing 120 real tasks across 94 actor-days and 44 distinct employees. Task types: business-system operations (49 tasks, 41%), research collaboration, HR reviews, document writing.

  • Business-system operations (CRM, finance, HR, sales records) demand precise modification of specific records — no "roughly right."
  • Collaboration tasks (documents, messages, replies) allow content flexibility as long as core facts, recipients, and format are correct.
  • Research tasks require understanding scientific logic (e.g., only comparable runs can be compared) and acting in the research system.
  • HR reviews require checking multiple conditions before judging promotion eligibility.
  • Three-Level Failure: The AutoBuddy Agent Pipeline

    The AutoBuddy agent works in three stages:

    1. Work profiling. A read-only session browsing the employee's email, calendar, and files to infer role, projects, manager, and collaborations. 2. Task mining. A second read-only session sweeping every human message in the time window, classifying each: direct request vs. background; for this person vs. someone else; answered vs. outstanding. Output: a structured task list with requester, evidence, deadline, priority. 3. Execution. Each discovered task runs in an independent write session: re-read context, then act — send email, edit document, update business system.

    Discovery is scored on recall and precision; execution on what actually changed in the world. Seventeen models reveal a clear "failure migration" pattern:

  • Bottom-tier models die at discovery. Claude Haiku 4.5 missed 59.7 tasks per run — 65% of its lost score. GPT-5.4-mini missed 36; DeepSeek-V4-Flash missed 32. They don't fail because they can't work; they fail because they never realize work exists.
  • Mid-tier models die at initiation. Claude Opus 4.8, GPT-5.4, and DeepSeek-V4.1-Flash find most tasks but make the "correct type of change" only 72–78% of the time. "Stopping to ask" accounts for 25–32% of their lost score — fatal when tasks are already authorized.
  • Top-tier models die at execution. The five strongest models miss only 3–9 tasks per run (7–20% of lost score) and act correctly 83–92% of the time. Their losses come from incomplete/incorrect content (22–44%), stopping to ask (19–44%), and wrong targets (7–25%). The strongest models don't fail to find tasks or refuse to act — they act and get it wrong.
  • This migration curve means discovery and execution are independent engineering problems; excellence in one doesn't imply excellence in the other.

    The Numbers

    Seventeen LLMs from five providers ran each of the 120 tasks three times. End-to-end scores ranged from 0.234 to 0.698.

  • Claude Opus 5.5 led at 0.698, with discovery F1 of 0.925 and execution score of 0.739. Notably, Claude Opus 5 had discovery F1 of only 0.808 yet an end-to-end score just 0.018 lower — stronger execution can partially compensate for weaker discovery.
  • GPT-6-series broke the ranking. GPT-6.1-sol had the highest recall (0.989 — almost everything found) but made correct-type changes on only 65% of tasks; "stopping to ask" was 71% of its lost score. GPT-6 models appear over-trained on caution: they seek confirmation even when tasks are pre-authorized.
  • Business-system tasks hurt the most. They're ~40% of tasks but 48–56% of top models' lost scores. Top models averaged 0.45–0.56 per business-system task versus 0.67–0.75 on collaboration tasks — precision requirements are the differentiator.
  • Failure Case Files

    1. Right answer, wrong channel. GPT-5.6-sol scored zero on task 54: it replied in an email thread when the correct target was a standalone group-chat message. 2. Two requests merged into one contradiction. GPT-6-luna received two messages from the same person specifying different invoice criteria, merged them into one task ("figure out which applies, then send"), and entered an endless confirmation loop — four questions asked, zero invoices sent. 3. "Replied" mistaken for "done." GPT-5.4-mini saw a chat request, found a closed email thread where someone said "already sent," and concluded the task was complete — but the chat request was actually about updating a portfolio-review document it never opened. 4. Kimi-K2.6, the outlier. It made correct-type changes 92% of the time (vs. 72–78% for peers) but lost score to discovery misses (38%) and content errors (32%). Its pattern: finish what it finds, but often doesn't find.

    The common theme: errors are contagious. A merge error at discovery paralyzes execution; a missed discovery means nothing gets done; a mislabeled discovery sends execution to the wrong target.

    Deeper Implications

    Proactivity is not a single capability dimension — it's a multi-stage pipeline, each stage with its own failure modes and improvement paths. If you're building a proactive AI assistant, ask three questions: Can it find signal in noise (discovery)? Can it act decisively without confirmation (initiation)? Can it operate precisely in complex systems (execution)? A system scoring zero on any one is unreliable.

    Discovery and execution quality correlate weakly, so improving one doesn't fix the other. Reliable proactive agents require optimizing both — meaning more integration testing, finer-grained evaluation protocols, and more careful deployment decisions.

    Marketing claims about "agentic proactivity" usually showcase one end-to-end success story. AutoBuddy's data shows even the best models complete under 70% of tasks, with heterogeneous failure modes. Without per-stage diagnostics, you can't know where your system breaks, let alone fix it.

    "Stopping to ask" also plays a dual role: a failure when tasks are authorized, but potentially a sensible safety strategy under uncertainty. When should a proactive agent act autonomously versus confirm? It depends on consequence severity, information completeness, and user preferences — no simple answer.

    Finally, a philosophical question: where is the boundary of authorization? In AutoBuddy-Bench all writes are pre-authorized. The real world isn't. An agent that can proactively discover "should follow up with this client" can also discover "should delete this email unfavorable to me." Proactivity is a double-edged sword — the source of efficiency and the breeding ground of risk. Learning to wield it will be one of the most important AI engineering challenges of the coming years.

    References

  • Chen, S., Madrigal Diaz, D., Chen, W.-N., Afonja, G., Yang, F., Wang, R., Bansal, C., Rühle, V., Rajmohan, S., & Han, D. (2026). AutoBuddy: Evaluating Proactive, Always-On Agents on Task Discovery and Execution in a Simulated Workplace. arXiv:2611.12393.
  • Shepard, J., & Salimans, T. (2026). AutomationBench: Simulated business systems for agent evaluation.
  • Yao, S., et al. (2023). ReAct: Synergizing reasoning and acting in language models.
  • Park, J.S., et al. (2023). Generative agents: Interactive simulacra of human behavior.
  • Lu, Y., et al. (2025). ProactiveBench: Evaluating proactive assistance.
  • Pasternak, N., et al. (2025). PROBE: Evaluating proactive agents in workplace settings.
  • Zheng, L., et al. (2023). Judging LLM-as-a-judge with MT-bench and chatbot arena.
---

Paper breakdown by Xiao Kai | 2026-11-11 | Daily Paper Picks · Issue 047

Tags

#ai-agents#proactive-agents#autobuddy#microsoft-research#benchmark#task-discovery#enterprise-ai#llm-evaluation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637007