Notion's Three-Year Journey to Custom AI Agents: Redefining How Work Gets Done

Based on a conversation between Sarah Sachs (Notion AI Engineering lead) and Simon Last (Product lead), originally featured on the Latent Space podcast, this post chronicles how Notion spent three years turning "Custom Agents" from an impossible dream into a core product capability.

The First Wall: 2022's Ambition Meets Reality

Notion's agent ambitions ignited when the team got access to GPT-4 in late 2022. Back then, nobody called it an "Agent" — the preferred term was "assistant." The vision was simple: give it full access to Notion's tools and let it work in the background.

Reality hit hard. Function calling didn't exist yet. Notion worked with frontier labs and Fireworks to fine-tune models that could call Notion functions, trying both Anthropic and OpenAI. Early on, the concept of "tools" didn't even exist — the team designed their own tool-calling framework using XML notation embedded in prompts. The result? Models were too weak, context windows too short, and multi-turn conversations frequently stalled. They occasionally glimpsed potential, but never reached "genuinely usable and impressive."

The lesson: model capability itself was a hard wall. Notion had to wait for models to improve while building the right infrastructure.

The Hardest Problem: Permission Design

Before Custom Agents launched, the biggest time investment wasn't making agents "run" — it was making them run reliably, especially in the background. Permissions, sharing, and admin interfaces — invisible in demos — became the real product blockers.

Sarah Sachs is blunt: these problems all reduce to Role-Based Access Control (RBAC). If an agent is shared to a Slack channel with one group of people, but can access confidential documents meant for another — where's the boundary? How do admins verify it at a glance?

Notion's answer: default zero permissions, explicit grants required — every step works like issuing the agent a limited-time pass.

> Note: RBAC (Role-Based Access Control) is a permission management mechanism, like corporate door cards: regular employees can enter offices, finance can enter the vault. AI agents follow the same principle — blanket access invites security incidents. This design makes agents powerful yet controllable.

This rigor is what lets Custom Agents enter enterprises rather than remain demos.

Not a Wrapper: Notion's "Agent Lab" Philosophy

Many assume Notion is just wrapping existing models. Sarah Sachs acknowledges early encapsulation existed, but that wasn't the core. Notion's real expertise is understanding "how people want to collaborate." She likens it to AWS and Datadog: AWS provides cloud storage; Datadog deeply understands monitoring pain points. Notion is the collaboration-savvy layer on top of the model foundation.

With users spanning individuals to large enterprises, the team balances "listening broadly" with "elegant primitives." Every Friday they review Custom Agent logs, analyzing which tasks burn the most tokens, cutting unreasonable parts, and staying anchored to real user journeys: email triage, PDF export, bug triage — not "ooh, this computer-control tool looks cool, let's try it."

Notion isn't dressing the model in pretty clothes; it's redesigning the whole house so humans and agents can both live comfortably. Hence the "Agent Lab" theory: Notion isn't an OpenAI-style lab, but the place where AI capabilities land in everyday work.

Low-Ego Magic: How Creativity Grows Like Weeds

Sarah Sachs describes her job not as generating the best ideas, but ensuring everyone has resources and channels to pursue what they believe matters. In AI work, the best prototypes often come from people closest to user pain.

Notion's culture is classic "low-ego": people willingly delete their own code and don't care whose idea came first. Simon Last asks himself every six months: "If we started over, what would we do?" Organizational boundaries are fluid — you might report to A today and work for B tomorrow.

They run two kinds of hackathons: one where senior engineers随时 join the "Simon whirlpool" for rapid productization, and a company-wide hackathon teaching everyone to build tools with Claude Code or Custom Agents. Image generation started this way — Jimmy from the database team wanted it badly, got resources, tokens, and support, and it became an official project.

> Note: Low-ego culture is like a soccer team where nobody fights to be the star — everyone passes and covers. If the ball goes in, it doesn't matter who scored. This minimizes internal friction during AI iteration.

"Demos over Memos" became the team creed, with prototype culture flourishing everywhere.

Evaluation: A Mirror Into the Future, Not an Exam

As the whole company builds for agents, evaluation became more than testing. Notion runs three tiers:

1. Unit-test style evaluations running in CI 2. Product-level evaluations — key user journeys must hit 80–90% pass rates before release 3. Ceiling evaluations — deliberately targeting ~30% pass rates to reveal the technical frontier

They even hire "model behavior engineers" — a role blending data scientist, product manager, and prompt engineer skills, without requiring a traditional engineering background.

Now they're turning the evaluation system itself into an agent framework: agents automatically download datasets, run evals, analyze failures, and debug fixes. Humans just watch from outside.

Evaluation is like a magic mirror: it tells you not only how the agent looks today, but what it can become next year.

Coding Agents + Software Factory: The Real Core of AGI

Simon Last is particularly excited: coding agents are becoming the core of AGI — everything can be a coding agent. Agents can bootstrap software themselves, debug, and maintain it. A software factory automates the entire chain — development, debugging, merging, review, maintenance — with agent swarms collaborating.

The spec layer uses Markdown PRDs (naturally living in Notion), self-verification relies on strong test suites, and workflow design resembles an issue-tracking system. When a bug comes in: a sub-agent takes over, submits a PR, reviews, merges, deploys — all pipeline-like while preserving critical invariants.

Not sci-fi — this is Notion's active direction. Imagine your company's codebase as a self-growing forest: agents are gardeners, humans are the chief designers who occasionally trim the direction.

CLI vs MCP: Two Keys, Each With Merits

On integrations, Simon Last is personally bullish on CLI: it runs in the terminal with inherent power — long-output pagination, progressive display, and self-bootstrapping/self-repair. When an agent breaks, it can debug and fix itself in its own environment. Someone even had an agent write its own browser tool in under 100 lines of code!

MCP is lighter and easier to permission, suited for single-capability, clearly bounded scenarios. Notion supports MCP (Linear, GitHub) while building its own tools (Slack, email, calendar) because core entry points need deeper control.

On cost: CLI may execute in one shot, while MCP can repeatedly burn tokens. Notion prices by usage to align cost with capability, without wasting user money.

> Note: CLI is like a Swiss Army knife that can repair itself; MCP is a dedicated key — safe but single-purpose. Choose based on whether you want your agent "wilderness survival" or "greenhouse growth."

What Custom Agents Actually Replace: The Most Annoying Layer of Process

Sarah Sachs puts it best: the biggest value of Custom Agents isn't replacing a person — it's replacing the extra manual intervention layer in a process.

Example: Kernel Labs' shared office applications previously required manually reading emails, searching for info, and replying. Now a 15-minute-built agent checks the inbox, creates a database, and fills in applicant backgrounds automatically.

Bug triage, agent managers, lease fillers... even internal issue trackers where agents message each other, monitor, and fix. Memory? No separate concept needed — Notion's pages and databases *are* the memory; just grant edit access.

It's like handing a company's most tedious grunt work to a swarm of assistants who never complain, freeing humans to focus on what matters.

Meeting Notes: From Records to Workflow Loop

Meeting notes became Notion's strongest growth engine, capturing data that was previously hard to systematize: transcripts, @mentions, auto-generated tasks.

Before standups, agents read Slack and GitHub to generate briefing materials; during meetings, they take notes; after, they sync with calendars, write items into task databases, and send Slack follow-ups automatically. The entire workflow closes the loop — hands off the keyboard, attention fully on the problem.

> Note: This isn't simple speech-to-text — it's turning meetings into "living data" that agents can process downstream. What looks like time savings is actually explosive data growth feeding AI rich context.

Future Currents: Retrieval Optimization and Auto Model Selection

Notion increasingly cares about retrieval, ranking, and search — the foundation layers closest to agent workflows. A growing share of search traffic comes from agents, not humans. Retrieval goals shift from "human first-click" to "Top-K coverage and diversity."

Auto isn't "cheapest and dumbest" — it's a smart advisor picking the best fit for the current task. Pricing uses a credits abstraction layer accounting for real costs (GPU, network, sandbox), letting users decide what's worth paying for.

Notion won't train general foundation models, but will fine-tune enterprise-context models and strengthen retrieval. Their positioning is constant: to be the best place where all post-collaboration sediment ultimately lands.

Conclusion: AI Isn't Here to Steal Jobs — It's Here to Help Serve the Meal

Notion spent three years hitting walls not to chase hype, but to wait for the river to truly change course. They aligned model capability, tool abstraction, permission design, evaluation systems, team organization, and even pricing into one coherent framework. Custom Agents aren't chat assistants bolted onto software — they turn enterprise software itself into a work substrate for both humans and agents.

Imagine a future office where humans and agents collaborate like old friends: agents handle the trivial, humans focus on creating. What AI truly replaces was never people — it's the repetitive, tedious, headache-inducing processes. Notion's three-year journey proves: with the foundation laid properly, agents go from "dreams hitting walls" to "reality changing daily work."

---

References

1. Latent Space podcast: conversation with Notion AI Engineering lead Sarah Sachs and Product lead Simon Last (2025 edition). 2. Notion official Custom Agents release documentation and beta feedback summary. 3. Notion internal "Agent Lab" theory whitepaper (shared by Sarah Sachs). 4. Anthropic and OpenAI model capability evolution reports (around Claude 3 Sonnet comparisons). 5. Enterprise AI workflow optimization case collection: Notion meeting notes and Custom Agents in practice (expanded from podcast examples).