Nemotron 3.5 Lightning: NVIDIA’s Fast Agent Model

NVIDIA Nemotron 3.5 Lightning: The Fast Open Model for Always-On AI Agents

NVIDIA is positioning Nemotron 3.5 Lightning as an execution engine for the next generation of AI agents—not as a replacement for every frontier model, but as a faster, more efficient specialist for the repetitive work that keeps long-running agents busy.

Released on August 11, 2026, Nemotron 3.5 Lightning is an open 30-billion-parameter Mixture-of-Experts model with only 3 billion active parameters per token. NVIDIA says it delivers up to 4× faster output than similar-sized models and can complete agentic workloads up to 30% faster at comparable accuracy. The model is available with BF16 and NVFP4 checkpoints, alongside speculative-decoding options and recipes for customization.

The launch also introduces NeMo Switchyard, an open-source model-routing library designed to place each request on the most appropriate model. Together, the two releases reflect a change in how AI systems are being built: one large model plans, while smaller specialized models execute high-volume steps such as tool calls, code review, validation, classification, and monitoring.

What is Nemotron 3.5 Lightning?

Nemotron 3.5 Lightning is a text-only, reasoning-capable model for coding, tool calling, chat, and agentic workflows. It is part of NVIDIA’s Nemotron 3.5 family and is designed for local systems, workstations, data centers, and cloud deployment.

Its headline architecture is a hybrid Latent Mixture-of-Experts design. The model interleaves Mamba-2 and MoE layers with selected attention layers, rather than using a conventional dense Transformer at every layer. A router activates only a subset of the experts for each token. That gives the model a large total capacity while keeping the per-token compute closer to a much smaller model.

The practical numbers are:

• 30 billion total parameters
• 3 billion active parameters per token
• Text-only inference
• Reasoning and coding support
• Native Multi-Token Prediction layers
• BF16 and NVFP4 checkpoints
• Open weights, data, and training recipes under NVIDIA’s Open Model License family
• Support for English, several major European languages, Japanese, and programming languages

This is not a multimodal model. It is intended for text reasoning, code, tool use, and agent execution. Developers who need image, audio, or video understanding should use a separate model in the system.

Why the execution layer matters

Long-running agents spend surprisingly little of their time on headline reasoning. Once a plan exists, the system may need to fetch a file, call an API, inspect a diff, validate a tool result, format structured output, retry a failed request, or delegate a small subtask.

Using a frontier model for every one of those operations creates an expensive and slow pipeline. A better architecture is a system of models:

This pattern is especially relevant to developers building AI agent stacks with MCP and coding tools. The model that chooses a tool does not always need to be the model that designed the entire workflow.

NVIDIA’s performance claims

NVIDIA reports that Nemotron 3.5 Lightning reaches the accuracy-speed Pareto frontier for small open models on the Artificial Analysis Intelligence Index. The company also cites PinchBench results where Lightning reached 86% accuracy and completed 10,000 tasks 30% faster than Qwen3.6 35B at similar accuracy.

The company’s headline claim is up to 4× faster output speed compared with similar models. NVIDIA also reports up to 30% faster agentic task completion, which is a more useful metric than tokens per second because it includes the full workflow rather than only generation speed.

These are vendor-reported or benchmark-specific results. They should not be treated as universal guarantees. Actual performance depends on the serving engine, quantization, prompt length, concurrency, tool latency, GPU, draft model, and agent harness. Teams should reproduce the comparison on their own traces before changing production routing.

Still, the benchmark framing is directionally important. Agent performance is often bottlenecked by end-to-end completion time, not by the raw intelligence of the largest model. A slightly less capable model that finishes routine steps quickly can improve the total system more than a slower model with marginally better single-turn quality.

Speculative decoding is built into the design

Nemotron 3.5 Lightning includes Multi-Token Prediction layers, which provide native support for speculative decoding. Instead of generating and verifying one token at a time, a draft path proposes multiple tokens and the main model verifies them in batches.

NVIDIA is also releasing two draft-model options:

• DSpark, recommended for DGX Spark inference and low-concurrency data-center workloads
• DFlash, which developers can benchmark against DSpark for their own serving patterns

The best choice depends on concurrency. NVIDIA notes that MTP and draft-model behavior can change as concurrency rises, and the optimal draft length may become smaller under heavier parallel load.

For developers, this means the model checkpoint is only part of the performance story. The serving stack, draft model, scheduler, and concurrency profile all need to be measured together. A benchmark that looks excellent at one request may behave differently with eight, 32, or hundreds of concurrent agent calls.

NVFP4 makes local deployment more practical

Lightning ships with an NVFP4 checkpoint as well as BF16. NVIDIA says the specialized NVFP4 kernels are used across Blackwell, Hopper, and Ampere GPUs, allowing the same quantized artifact to serve both desktop-class systems and data-center deployments.

The smaller active-parameter count and quantization make the model attractive for local AI systems such as NVIDIA DGX Spark, GeForce RTX 5090 PCs, DGX Station, and Jetson devices. It is also supported by tools including LM Studio, llama.cpp, Ollama, and Unsloth.

This is where Lightning connects with the broader local-AI movement described in the Unsloth Desktop local AI workspace. The model is not restricted to a hosted NVIDIA endpoint; developers can experiment on local hardware, move to a workstation, or deploy through a managed inference provider without changing the fundamental model family.

The BF16 release is aimed more at customization and post-training than at the most memory-efficient direct inference. For deployment, developers should evaluate NVFP4 first, then compare quality and throughput against BF16 when fine-tuning or debugging numerical behavior.

NeMo Switchyard: routing instead of one-model-for-everything

NeMo Switchyard is NVIDIA’s open-source model-routing library for agent systems. It can route each request to a suitable open, proprietary, or NVIDIA model without requiring developers to rewrite the application interface.

A router can optimize for different objectives:

• Quality for difficult decisions
• Latency for interactive steps
• Cost for high-volume background work
• Privacy for sensitive data
• Local availability when cloud access is restricted

The intended pattern is simple: a frontier model handles planning, while Lightning handles the execution layer. If a tool call fails repeatedly or a task becomes ambiguous, the router can escalate to a stronger model.

NVIDIA cites internal Switchyard benchmarks showing frontier-level accuracy at close to one-third of the cost of using Opus 4.8 alone. Partner results include cost and latency reductions from teams integrating routing into coding, formal verification, developer tools, and multi-turn agents. Because these are internal or partner-reported results, independent evaluation remains important.

Routing itself is not free. A router adds classification logic, observability requirements, and more complicated failure modes. It can also create inconsistent behavior if a task moves between models with different system prompts, context handling, or tool-call formats. Production teams should log the selected model, reason for the route, latency, token count, and final task outcome.

Customization is a central feature

NVIDIA is releasing weights, training data, and recipes so organizations can adapt Lightning to their own domains. The model can be fine-tuned with LoRA or full supervised fine-tuning using NeMo Automodel and NeMo Megatron Bridge. NVIDIA also points to NeMo RL and NeMo Gym for reinforcement learning and environment-based evaluation.

The release includes Nemotron-RL-Agentic-Terminal-Pivot, an agentic reinforcement-learning dataset used for coding-agent capabilities. This provides a starting point for developers who want to improve terminal interaction, repository operations, or tool-use behavior.

A useful real-world example comes from CodeRabbit. In a two-stage experiment, CodeRabbit used supervised fine-tuning and reinforcement learning with verifiable rewards to teach Lightning its code-review routing policy. On a frozen 1,000-task evaluation, exact route agreement improved from 75.8% for the baseline GPT-class model to 80.4% after SFT and 80.7% after SFT plus RLVR. CodeRabbit reported that the tuned model reduced estimated inference cost by about 50% in its test setup.

That result is not proof that every company will see the same gain. It does show why a smaller open model can be valuable: a narrow, high-volume task may respond better to targeted post-training than to a larger general-purpose model used unchanged.

Deployment options and API compatibility

Nemotron 3.5 Lightning is available through Hugging Face, ModelScope, OpenRouter, NVIDIA NIM, build.nvidia.com, and a growing group of hosted inference providers. It can also be served through vLLM, SGLang, TensorRT-LLM, Ollama, llama.cpp, LM Studio, and Unsloth.

NVIDIA’s NIM documentation exposes OpenAI-compatible and Anthropic-compatible APIs for chat completions, responses, Anthropic messages, tool calling, and agentic workflows. That makes it easier to test Lightning inside an existing application without redesigning the client layer.

NIM adds a packaged, model-specific deployment path with tested profiles, quantization choices, GPU-memory settings, parallelism, reasoning parsing, and tool-call parsing. Developers can still override settings and pass raw vLLM arguments when they need more control.

The model-specific NIM documentation calls out several operational details:

• Use the nemotron_v3 reasoning parser when reasoning output needs to be separated correctly.
• Enable the required auto-tool-choice and parser settings for automatic tool calling.
• Leave the KV-cache dtype at the recommended FP8 default; NVIDIA warns against forcing NVFP4 KV cache.
• Validate the selected profile with /v1/metadata, /v1/models, and startup logs.
• Treat BF16 multi-GPU startup issues on specific B200/B300 configurations as a deployment compatibility problem, not automatically as an out-of-memory error.

These details matter because agentic reliability often fails at the serving boundary. A strong model with the wrong parser can emit reasoning tokens in the visible answer or produce tool calls that the harness cannot interpret.

What developers should test before adopting it

Before routing production traffic to Lightning, measure more than tokens per second:

1. Task success: Did the agent complete the actual workflow?
2. Tool-call validity: Were arguments syntactically and semantically correct?
3. Escalation rate: How often did the router need a stronger model?
4. Latency distribution: Compare p50, p95, and p99 rather than averages.
5. Cost per completed task: Include retries, routing, and tool execution.
6. Context behavior: Test long histories, retrieved documents, and multi-turn sessions.
7. Safety boundaries: Verify that tool permissions and sandbox limits remain enforced.
8. Regression quality: Keep a frozen evaluation set for every fine-tuning or router change.

For security-sensitive agent systems, local deployment does not remove the need for isolation. A model that can call tools, execute code, or reach internal services should run with least-privilege credentials and explicit network policy. The same verification-first approach used in AI agent tool authorization threat modeling applies here.

The bigger picture

Nemotron 3.5 Lightning is notable less because it is another 30B model and more because NVIDIA is packaging it around a systems-level thesis: agent applications need specialized execution models, model routing, efficient inference, and domain post-training.

That thesis is likely to shape the next phase of AI engineering. The winning stack may not be one giant model answering every request. It may be a coordinated set of models, with routing policies that send each task to the smallest model capable of doing it reliably.

Lightning gives developers an open, customizable candidate for that execution layer. NeMo Switchyard provides a path to combine it with stronger models. Together, they offer a practical way to trade off quality, latency, cost, privacy, and hardware locality—without forcing every agent step through the most expensive model available.

Frequently asked questions

Is Nemotron 3.5 Lightning open source?

NVIDIA describes it as an open and customizable model, with weights, training data, and recipes released under the applicable Open Model License terms. Always review the current license file and usage conditions before commercial deployment.

How large is the model?

It has 30 billion total parameters but activates approximately 3 billion parameters per token. Memory requirements still depend on the checkpoint precision, runtime, context length, KV cache, and serving configuration.

Is it multimodal?

No. Nemotron 3.5 Lightning is text-only. It is designed for reasoning, coding, tool calling, and agentic workflows.

Can it run locally?

Yes. NVIDIA lists local targets including RTX PCs, DGX Spark, DGX Station, and Jetson, and the model is supported by tools such as Ollama, llama.cpp, LM Studio, and Unsloth.

What is NeMo Switchyard?

NeMo Switchyard is an open-source model-routing library that directs requests to different models based on quality, latency, cost, or other policies.

Where can I try it?

Developers can try Nemotron 3.5 Lightning through build.nvidia.com and OpenRouter, download checkpoints from Hugging Face or ModelScope, or deploy it using NVIDIA NIM and supported inference engines.

Sources

• NVIDIA Technical Blog: Nemotron 3.5 Lightning
• NVIDIA Blog: Nemotron 3.5 Lightning and NeMo Switchyard
• NVIDIA NIM documentation
• NVIDIA Nemotron 3.5 Lightning on Hugging Face
• CodeRabbit post-training case study