HarnessX Architecture Deep Dive: Darwin Agent Team's Harness Evolution Framework

Key points

• Event-driven pipeline model: Agent lifecycle is decomposed into 8 hook points (task_start → step_start → before_model → after_model → before_tool → after_tool → step_end → task_end). Each hook hosts a chain of processor event handlers that yield, modify, intercept, or split events.
• Minimal Processor Protocol: The entire plugin interface reduces to a single async def process(self, event: Event) -> AsyncIterator[Event]. A processor can pass through (yield event), transform (yield modified_event), intercept (no yield), split (yield event; yield new_event), or raise to terminate. MultiHookProcessor provides eight default no-op methods so subclasses override only what they need.
• ProcessorChain composition: Each processor receives the full output of the previous one, enabling event fission. An empty output list short-circuits the rest of the chain (interception semantics).
• Hook Contract system (safety layer): Per-hook invariants prevent misuse:
• step_start: system prompt must stay at position 0; cannot silently rewrite non-last history.
• before_model (strictest): messages cannot be empty; length delta can only be 0 or +1; if last message is user, only that message may change; if not, only one appended user message is allowed across the entire chain.
• Other hooks forbid any message-length mutation.
• Controlled via HARNESSX_CONTRACT_MODE env var: warn (default, log only) or strict (raise ContractViolationError). Recommended two-phase rollout: collect violations in warn mode, then enable strict.
• RunLoop eight-phase orchestration:
• Phase 0 TaskStart: emits TaskStartEvent; system prompt is assembled; detects prompt changes and emits SegmentBoundaryEvent.
• Phase 1 StepStart: captures pre-step snapshot z_t; emits StepStartEvent with raw_messages, messages, tools; runs context assembly processors.
• Phase 2 BeforeModel: emits BeforeModelEvent; runs last-mile processors (e.g., CostGuard, SkillLoader); RunLoop injects system prompt and validates the sequence.
• Phase 3 Call Model: invokes model_provider.complete; honors skip_model synthetic outputs; emits ModelResponseEvent.
• Phase 4 AfterModel: may emit SpawnSubAgentEvent for multi-agent forks; updates token/cost tracking.
• Phase 5 Execute Tools: honors interrupt_on; emits ToolCallEvent / ToolResultEvent; truncates oversized outputs to disk.
• Phase 6 StepEnd: emits StepEndEvent with cost/token/time; appends a TrajectoryStep.
• Phase 7 Loop termination: handles end_turn/stop, thinking-only continuations, length-truncated continuations, empty-first-turn retry, task-done signals, and budget overflow.
• Phase 8 TaskEnd: emits TaskEndEvent with output, exit_reason, stats; evaluation processors run here.
• Interrupt and resume: interrupt_on returns exit_reason='interrupted' with the paused ToolCall. Resume by passing the same state back into the loop.
• Dual-track message invariant: raw_messages is append-only factual history (user/assistant/tool outputs); messages is the effective context including processor-injected guidance. Invariant: len(raw_messages) == len(messages). Historical structure changes trigger SegmentBoundaryEvent, which rotates the journal JSONL so new segments can resume independently.
• State and Trajectory: State holds run_id, raw_messages, messages, step, token/cost accumulators, tool_results, slots, budget caps, spawn_depth, pending_subagents, and last system-prompt hash. StateSlot is a typed dynamic key-value store (slot_type as namespace) with automatic non-serializable handling. TrajectoryStep records snapshot + delta + action + observation + reward, making it directly consumable by RL trainers.
• Seven built-in processor categories:
• Context: SystemPromptProcessor (default/template/null builders), UserWrapperProcessor (XML, CoT), EnvironmentContextInjector.
• Control: LoopDetectionProcessor (SHA256 tool+input fingerprint, 3 warn / 5 abort; name-only 8 warn; compaction-aware reset when message count drops ≥5), CompactionProcessor (token-thresholded summarization preserving tool_use/tool_result pairs), ToolCallCorrectionLayer, ParseRetryProcessor, SelfVerifyProcessor, TodoWriteEnforcer, CostGuardProcessor (70% budget warning), TokenBudgetProcessor, ToolFailureGuard, RepeatedFileEditDetector, BgInstallGuard, SycophancyDetector.
• Evaluation: EvaluationProcessor with LLMJudgeEvaluator, SelfVerifyEvaluator, and PRM variants (TerminalPRM, DiscountedPRM, ToolSuccessPRM, LLMJudgePRM).
• Memory: MemoryExtractionProcessor (default OldestMessagesExtractor(n=20)), MemoryRetrievalProcessor with SlidingWindowMemory, SummarizationMemory, or custom backends.
• Multi-Model: ModelRouterProcessor with ProviderGroup fallback.
• Observability: OTelProcessor, CheckpointProcessor, EpisodeMetricsProcessor.
• Tools: ProgressiveSkillLoader (keyword-matched dynamic SKILL.md injection with caching), ToolFilter/ToolWhitelist, ModelSchemaAdapter.
• HarnessBuilder composition: All add, slot, add_tool calls return new instances (immutability). Merge via builder_a | builder_b detects Slot conflicts, Tool name conflicts, and singleton_group duplicates. Plugins can declare processors, tools, slash commands, MCP servers, lifecycle hooks, and skill dirs. YAML config supports both Hydra-style _target_ and short type schemas.
• veRL integration: HarnessXAgentLoop (registered as harnessx_agent) implements a PENDING → GENERATING → PROCESSING_TOOLS → TERMINATED state machine with semaphore-bounded concurrent tool calls and automatic response truncation. Config knobs: max_assistant_turns: 12, max_parallel_calls: 8, plus a tool allowlist (WebSearch, WebFetch, Browser, Bash, CodeInterpreter, Read). Training flow: veRL PPO/GRPO trainer → HarnessXAgentLoop rollout → custom reward (e.g., 0.8*accuracy + 0.1*format + 0.1*tool_call) → policy gradient update.
• Engineering patterns worth noting:
• AST-based semantic hashing strips docstrings/comments before SHA-256 hashing so processor serialization detects version drift.
• Topological ordering via _order and _after (Kahn algorithm) prevents cycles and orders processors inside each hook.
• Error isolation: harness/contract errors propagate; other exceptions are logged and the event passes through, so a buggy processor cannot crash the whole run.

Conclusion

HarnessX demonstrates production-grade maturity through seven architectural pillars: a minimal Processor Protocol with rich yield semantics; a Hook Contract system with two-phase rollout for safe plugin ecosystems; immutable composition via HarnessBuilder; full lifecycle coverage across eight hooks; RL-ready Trajectory/State-delta data structures; OTel/Checkpoint/Journal observability; and a dynamic skill ecosystem via ProgressiveSkillLoader. The core thesis: agent framework competitiveness comes from simplicity, composability, and safety of the architecture, not from any single feature.

References

• HarnessX source: https://github.com/Darwin-Agent/HarnessX
• veRL: https://github.com/volcengine/verl
• License: MIT