Nemotron 3.5 Lightning In-Depth: A 30B-A3B MoE Built for Local Agent Execution

Key points

• Identity and positioning: NVIDIA's Nemotron 3.5 Lightning (checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B), released 2026-08-11, is targeted at the agent execution layer: tool calling, result validation, sub-agent dispatch, and retries, rather than planning or deep reasoning.
• Architecture: A Hybrid MoE mixing Mamba-2 state-space layers (for long-sequence efficiency), MoE feed-forward layers (for capacity/compute decoupling), and selective attention layers. Total 30B parameters, ~3B active per token; family figures cite 31.6B/3.6B.
• Context window: Theoretical envelope of 1M tokens; ~256K on a single H100; Ollama on DGX Spark defaults to 262,144.
• Checkpoints and licensing: BF16 (~60 GB), NVFP4 (~15 GB), NVFP4-DSpark. Community GGUF (bartowski, LM Studio) followed shortly. License is OpenMDW-1.1, which permits commercial use, distillation, and redistribution, with weights, training data, and recipes released.
• Lineage and ecosystem: Distilled from Nemotron 3 Ultra (550B); base model trained on 20T+ tokens; co-tuned with agent-harness partners including Cline, OpenClaw, OpenCode, Hermes Agent, Kilo Code, LangChain, and OpenHands. The notion of a formal "Nemotron Alliance" in some scene copy is inaccurate; it describes ecosystem co-development.
• Spec-correction notes: The "1M context" is an input envelope, not guaranteed accurate reasoning at that length. The "4x throughput" claim is NVIDIA self-evaluation using "up to 4x" marketing language, with no independent reproduction.
• Speculative decoding toolkit: Three draft mechanisms are provided: MTP (multi-token prediction, embedded at training time, best at medium-to-high concurrency), DSpark (semi-autoregressive parallel draft optimized for DGX Spark and low-concurrency data-center use; SGLang reports it as the best of the three on Spark), and DFlash (lightweight diffusion-based draft model that proposes blocks in parallel).
• NVFP4 quantization: Single checkpoint reusable across Blackwell, Hopper, and Ampere; ~15 GB weight footprint is the key to running a 30B-class MoE on a single consumer GPU.
• Per-request reasoning control: enable_thinking flag and thinking_budget parameter let orchestrators turn chain-of-thought on/off per request.
• Local deployment: ollama run nemotron-3.5-lightning for standard chat, ollama run nemotron-3.5-lightning:30b-mlx for Apple Silicon. The ollama launch <harness> syntax is from the official announcement; if a local Ollama build lacks the subcommand, start the model with ollama run and point the harness at its API.

VRAM sizing (the "3B active" caveat)

All experts must be resident, plus KV cache for long context:

• BF16: ~60 GB weights, plus KV; requires multi-GPU H100/A100.
• NVFP4: ~15 GB weights; ~16–26 GB with KV depending on context. RTX 5090 (32 GB) comfortable; RTX 4090 (24 GB) viable at moderate context; DGX Spark measured at 26 GB including 262K context.
• GGUF Q4: ~18 GB weights, ~18–22 GB total; Q2/Q3 fits 16 GB cards.
• MLX (Apple): Q4 ~18 GB / Q8 ~30 GB; uses unified memory. Suitable on M4 Max (36/48 GB) and M3 Ultra (80/128 GB).

Throughput

Artificial Analysis measured ~670 tok/s output. NVIDIA positions Lightning on the accuracy–speed Pareto frontier versus similar-class models and claims it is about 30% faster than Qwen3.6 35B at comparable accuracy.

Comparison snapshot (local-agent oriented)

• Nemotron 3.5 Lightning — 30B/3B, 1M (practical ~256K), agent execution, OpenMDW-1.1, NVFP4 15 GB local-friendliness: 5/5.
• gpt-oss-120b — 120B/5.1B, 128K, general/agent, Apache 2.0, MXFP4 (large VRAM), 3/5.
• Qwen3-235B-A22B — 235B/22B, 128K+, general flagship MoE, Apache 2.0, 3/5.
• Llama 4 Maverick — 400B/17B, 1M, general MoE, Llama 3.1 Community, 2/5.
• Llama 4 Scout — 109B/17B, 10M, ultra-long-context MoE, Llama 3.1 Community, 2/5.
• DeepSeek V3 / R1 — 671B/37B, 128K, general/reasoning, MIT, multi-GPU only, 1/5.
• Gemma 3 (27B) — 27B dense, 128K, general dense, Gemma T&C, 4/5.

Strengths and weaknesses (honest framing)

• Strengths: 86% on PinchBench (BF16 85.37 / NVFP4 83.43), reportedly ~30% faster than Qwen3.6 35B at similar accuracy; approaches gpt-oss-120b quality at roughly 1/4 the active compute; NVFP4 makes single-consumer-GPU deployment routine; fully open weights, data, recipes, and RL datasets; aligned with mainstream agent harnesses out of the box.
• Weaknesses: Terminal-Bench 2.1 score only ~24.6, so it is not suitable as the primary coding/agent brain; benchmark numbers are NVIDIA-self-evaluated and lack independent reproduction; long-context quality degrades and must be tested with real workloads; OpenMDW-1.1 is permissive but the third-party tooling ecosystem is still maturing.

Decision rule

• Need deep reasoning or complex planning? Use a frontier model (Nemotron 3 Ultra, Opus, DeepSeek R1), routed via Switchyard.
• High-frequency execution (tool calls, result checks, sub-agent dispatch, retries)? Lightning on local NVFP4 is the best fit.
• General dialogue or creative writing? Pick from Qwen3, Gemma 3, or gpt-oss based on license and hardware.

NeMo Switchyard: the bigger story

Lightning is the execution-layer answer; NeMo Switchyard (github.com/NVIDIA-NeMo/Switchyard) is the routing library that wires it into production. Switchyard translates between OpenAI Chat, OpenAI Responses, and Anthropic Messages formats, applies escalation policies (cheap model first, small judge reviews quality, escalate on failure), and was released as an officially supported companion to the model.

Reported cost outcomes:

• Switchyard overall routing: frontier-level completion at roughly 1/3 the cost of running Opus 4.8 alone.
• LangChain: 145 multi-turn DeepAgents tasks; only 7% of calls routed to a frontier model; cost reduced 74%.
• Ramp: SWE-Bench parity with frontier performance; cost down 58%, runtime down 33%.
• Custom tuning in production by CrowdStrike, Harvey, Trajectory, and CodeRabbit for security, legal, and code-review workloads.

Use-case verdicts

• Long-term personal assistant (email/calendar/projects/booking): Viable locally with privacy benefits; pair with a frontier planner via Switchyard.
• Sub-agent coding (running tests, searching the codebase, applying patches): Strong execution fit; weaker on autonomous architecture design given the Terminal-Bench result.
• Security operations (triage, log lookup, metric correlation): Strong fit; precedent from CrowdStrike tuning.
• Parallel local + cloud tiers: Single CLI/API surface; Switchyard handles layering cleanly.
• Custom expert training: OpenMDW-1.1 + NeMo recipes + Unsloth quantization on the same day enables local fine-tuning with a low barrier.

Bottom line

Nemotron 3.5 Lightning is not "a smaller frontier model." It productizes the most token-expensive grunt work in agent workflows: 30B capacity, 3B active compute, NVFP4 weights around 15 GB that fit on a consumer card, agent-harness alignment at the factory, and a fully open license for fine-tuning. Its strongest pairing is a multi-model system where a frontier model plans and Lightning executes, orchestrated by NeMo Switchyard. For local-first, data-on-device, always-on assistants, it is currently the most local-friendly open execution-layer model; for primary-agent responsibilities, manage expectations.

Sources

• NVIDIA official tech blog (developer.nvidia.com, 2026-08-11): architecture, speculative decoding, PinchBench 86%, NeMo Switchyard.
• NVIDIA Local AI blog (blogs.nvidia.com, 2026-08-11): release announcement, hardware matrix, ecosystem partners.
• Hugging Face model card nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-{BF16,NVFP4}.
• SGLang Day-0 support post (lmsys.org, 2026-08-11): MTP/DSpark/DFlash implementation, reasoning control.
• LLM Stats research (llm-stats.com, 2026-08-11): parameters, context, benchmarks with caveats.
• AGI Hunt, IT之家, 钛媒体/网易号 (2026-08-11~12): Chinese-language secondary verification and ecosystem coverage.
• NxCode, Dev.to, AlphaSignal (2026-08): independent analysis and cost case studies (LangChain, Ramp).