Notion's Three-Year Journey to Custom Agents: Redefining Work with AI
A Chinese forum post based on a Latent Space podcast conversation with Notion AI engineering lead Sarah Sachs and product lead Simon Last, covering how Notion spent three years turning AI agents from a blocked experiment into shipped Custom Agents.
The First Wall: 2022's Early Ambition
Notion got access to GPT-4 in late 2022 and immediately wanted to build an agent—though back then the term was "assistant." The dream: give the model access to all of Notion's tools and let it work in the background.
Reality hit hard. Function calling didn't exist yet. The team worked with frontier labs and Fireworks to fine-tune a model that could call Notion functions, experimenting with both Anthropic and OpenAI models. Before tools were a native concept, they built their own tool-calling framework using XML notation embedded in prompts. But models were too weak and context windows too short—multi-turn conversations constantly stalled.
The lesson: model capability itself was a hard wall. They had to wait for it to improve while building the right infrastructure.
The Hardest Part: Permissions, Not Intelligence
The most time-consuming work before launch wasn't making agents run—it was making them run reliably, especially in the background. Permissions, sharing, and admin surfaces became the real blockers.
Sachs is blunt: these problems all reduce to role-based access control (RBAC). If an agent is shared with a Slack channel, how do you prevent it from accessing confidential documents others can see? Notion's answer: zero permissions by default, explicitly granted, like issuing a limited-time pass to a butler at every step.
Not a Wrapper: Rebuilding the Foundation
Many assume Notion just wraps existing models. Sachs acknowledges early versions were wrappers, but that wasn't the point. She compares Notion to Datadog on AWS: AWS provides storage; Datadog deeply understands monitoring pain points. Notion is the layer that truly understands collaboration.
The team reviews custom agent logs every Friday, analyzes which tasks burn the most tokens, and cuts what doesn't make sense—staying anchored to real user journeys: email triage, PDF export, bug routing. Not "wow, this computer-control tool is cool, let's try it."
Low-Ego Culture
Sachs sees her job not as generating the best ideas, but ensuring everyone has resources to pursue what matters. The best prototypes come from people closest to user pain.
Notion's culture is distinctly "low-ego": engineers willingly delete their own code, and Simon Last asks himself every six months, "If we started over, what would we do?" The team runs two kinds of hackathons—one where senior engineers rapidly productize ideas (the "Simon vortex"), and a company-wide one teaching everyone to build tools with Claude Code or custom agents. The image-generation feature started this way when a database engineer named Jimmy pushed for it.
Their motto: "Demos over Memos."
Evaluation as a Mirror, Not an Exam
Notion runs three tiers of evaluation:
1. Unit-test level: runs in CI 2. Product-level: key user journeys must hit 80–90% pass rates before release 3. Ceiling evaluations: deliberately targeting ~30% pass rates to see the technical frontier
They hire "model behavior engineers"—a role blending data science, product management, and prompt engineering, with no traditional engineering background required. The evaluation system itself is now agentic: agents download datasets, run evaluations, analyze failures, and debug fixes while humans supervise.
Coding Agents and the Software Factory
Simon Last is especially excited that coding agents are becoming the core of AGI—agents that can bootstrap, debug, and maintain software. A software factory automates the full loop: spec (Markdown PRDs living naturally in Notion), self-verification via strong tests, then sub-agents taking bugs, submitting PRs, reviewing, merging, and deploying like an assembly line.
CLI vs MCP
- CLI: runs in the terminal with built-in strengths—long-output pagination, progressive rendering, and self-bootstrapping/self-repair. One agent even wrote its own browser tool in under 100 lines.
- MCP: lighter, easier permission control, suited to single-capability, clearly-bounded use cases (Notion supports MCP for Linear and GitHub).
Agents Replace Processes, Not People
Sachs's clearest point: custom agents' biggest value is replacing the extra manual layer in a process, not replacing a person. Example: Kernel Labs' shared-office applications previously required manually reading emails, searching for info, and replying. A 15-minute-built agent now checks the inbox, creates a database, and enriches applicant backgrounds automatically.
Memory needs no separate concept—Notion pages and databases are the memory; just grant edit access.
Meeting Notes as a Growth Engine
Meeting notes became Notion's strongest growth engine, capturing previously hard-to-systematize data: transcripts, @-mentions, auto-generated tasks. Before standups, an agent reads Slack and GitHub to generate briefing materials; after meetings, it writes tasks to databases and sends Slack follow-ups—a seamless workflow loop.
What's Next: Retrieval and Auto Model Selection
Notion cares most about retrieval, ranking, and search—layers closest to agent workflows, since more search traffic now comes from agents rather than humans. Retrieval goals shift from "human's first click" to "Top-K coverage and diversity." "Auto" model selection isn't the cheapest dumbest model—it's the best fit for the task, with pricing abstracted into credits reflecting real GPU, network, and sandbox costs.
Notion won't train general foundation models, but will fine-tune enterprise-context models and strengthen retrieval. Its positioning: the best place where collaboration's output ultimately lands.
Conclusion
Notion's three years of hitting walls weren't trend-chasing—they were waiting for the river to change course. By aligning model capability, tool abstraction, permission design, evaluation, team organization, and pricing into one framework, Custom Agents became not a chatbot bolted onto software, but a work foundation serving both humans and agents. AI doesn't replace people—it replaces the most tedious processes.
---
References 1. Latent Space podcast: conversation with Notion AI engineering lead Sarah Sachs and product lead Simon Last 2. Notion official Custom Agents launch documentation and beta feedback 3. Notion internal "agent lab" philosophy notes (shared by Sarah Sachs) 4. Anthropic and OpenAI model capability evolution reports 5. Enterprise AI workflow case collection: Notion meeting notes and custom agent practices