Context Engineering for AI Agents: Lessons from Building Manus

> "The agentic future will be built one context at a time. Engineer them well."

This forum post discusses the Manus team's practical lessons on context engineering — building, managing, and optimizing the input context fed to frontier LLMs — as the preferred path to production-grade AI agents, rather than traditional fine-tuning. The author jokingly calls this iterative trial-and-error craft "Stochastic Graduate Descent," a pun on gradient descent.

Key points

1. Why context engineering over fine-tuning?

• Fine-tuning pipelines have slow (week-scale) feedback loops and are invalidated whenever a new base model generation ships.
• In-context learning leverages frontier models directly; agent tasks are long-horizon, multi-step, and uncertain, making context the true lifeline.

2. KV-Cache is the invisible lifeline

Agent workloads have extremely asymmetric token ratios (~100:1 input:output), so inference cost and TTFT are almost entirely determined by cache hit rate:
\[Cost_{inference} \propto (1 - \text{CacheHitRate}) \times N_{input} + N_{output}\]

Three key practices: 1. Prefix stability — keep the head of the context static; never put a per-second timestamp at the start of the system prompt (a common anti-pattern that invalidates all downstream KV cache). 2. Append-only context — never modify past actions or observations; keep serialization deterministic (even JSON key order must be fixed). 3. Explicit cache breakpoints — manually insert markers in inference frameworks without automatic incremental caching.

These details cut per-task cost from dollars (demo level) to cents (production level).

3. Taming tool explosion with logit masking

Instead of dynamically adding/removing tool definitions in the prompt (which confuses the model when histories reference vanished tools), Manus masks logits at decoding time (constrained decoding):
\[P(token|context) = \text{Softmax}(Logits + Mask)\]

Implementation details:

• A finite state machine tracks the currently allowed tool set; disallowed tokens get −∞ probability.
• Prefill with special tokens (e.g., im_start) to quickly enter tool-call mode.
• Normalize tool names by prefix (browser_, shell_) for prefix masking.
• Support Auto / Required / Specified invocation modes.

This moves control logic from the expensive context to the nearly free decode stage.

4. The file system as unlimited external memory

• Store only URLs or file paths in context instead of full webpages/PDFs/repos.
• The model calls read_file(path) to load content on demand, then optionally compresses/summarizes it before appending.
• This sidesteps the "Lost-in-the-middle" attention problem (Liu et al., arXiv:2307.03172).
• The author likens this to a Neural Turing Machine and speculates that State Space Models (e.g., Mamba, arXiv:2312.00752) with native file I/O could become a more natural agent architecture.

5. Recitation against attention decay

In 50+ step tasks, the original goal drifts out of attention due to recency bias:
\[Attention(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V\]

Manus has the agent maintain a todo.md and recite current progress and remaining goals at the end of the context each step, forcing the global plan back into the strongest attention positions.

6. Keep error trajectories

Rather than wiping the context after failures, Manus deliberately retains erroneous actions and their full error observations. Repeated "Action A → Error X" patterns act as negative samples, letting the model naturally lower the probability of repeating the mistake — recovery from errors is core agent intelligence.

7. Avoid the few-shot trap with structured noise

When many similar action-observation pairs accumulate (e.g., processing 20 resumes), models fall into mode repetition, loops, or hallucination. Countermeasures:
• Slightly randomize serialization templates (wording, field order).
• Vary the phrasing of observation summaries.
• Tweak tool-call formats.

Philosophy: from Demo to Production

The post condenses Manus's lessons into four maxims:

1. KV-Cache is the lifeline — all design must accommodate caching. 2. Decode-time intervention (logit masking) beats prompt engineering — more reliable and token-efficient. 3. Bigger context isn't better — use external storage and on-demand loading to maintain high signal-to-noise ratio. 4. True robustness comes from embracing errors, not from pursuing one perfect plan.

References

1. Manus Team. Context Engineering for AI Agents: Lessons from Building Manus. https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus 2. Liu, J. et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv preprint arXiv:2307.03172. 3. Vaswani, A. et al. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (2017). 4. Gu, A. et al. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint arXiv:2312.00752. 5. OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.