Gemini Robotics ER 2: Live API, Tool Orchestration, and Safety Guide

Gemini Robotics ER 2 is Google DeepMind’s high-level reasoning model for robots: it takes text, image, video, or audio, reasons about the physical scene, plans multi-step work, and calls robot or application tools. The practical developer pattern is to keep ER 2 above the motor-control layer. Let it decide what should happen and when a step is complete; let a vision-language-action (VLA) model or robot API perform the movement.

Google announced ER 2 on July 30, 2026. It is available to developers through the Gemini API and Google AI Studio, with a private preview on Gemini Enterprise Agent Platform. The API documentation currently lists a standard preview endpoint and a streaming preview endpoint for low-latency, bidirectional sessions.

The important limitation is just as clear: ER 2 is not a safety-certified robot controller. Google’s model card says not to use the Robotics Models for safety-critical applications such as healthcare or transportation. A production system still needs physical safety controls, authentication, authorization, emergency stops, human supervision, and deterministic checks outside the model.

What Gemini Robotics ER 2 does

ER 2 is a vision-language model based on Gemini 3.5 Flash. Google describes it as an embodied-reasoning layer that can understand an environment, converse with people, plan several steps, orchestrate tools, and hand execution to a lower-level VLA. Its inputs can interleave text, images, video, and audio; the model card lists a context window of up to 128K and up to 64K output tokens.

The API exposes two model IDs:

The standard endpoint is useful for a bounded perception or planning request. The streaming endpoint is the better architectural fit when the robot is continuously observing a changing scene and must avoid long stop-and-think pauses.

Google’s documentation also notes that ER 1.6 is scheduled to shut down at the end of August 2026. Teams still using that model should test the replacement rather than treating the model-name change as a drop-in production upgrade.

The architecture developers should build

The safest mental model is a three-layer loop. ER 2 should not receive unrestricted control of motors, credentials, or arbitrary network access. Instead, it selects from narrow tools whose implementations enforce policy.

Visual: the recommended ER 2 control boundary. The model proposes tool calls; the application, robot controller, and physical safety system decide whether those calls can execute. This is an original architecture diagram based on Google’s ER/VLA separation and the safety constraints in the official documentation and model card.

A useful request flow is:

1. Convert the user’s request into a task with an explicit workspace, time limit, and allowed tools.
2. Send the smallest useful visual context to ER 2.
3. Expose typed tools such as navigate_to, identify_object, or request_gripper_action, not a generic shell.
4. Validate every argument in the tool implementation.
5. Require approval or a deterministic safety gate for actions with meaningful physical consequences.
6. Feed observations and tool results back to the model.
7. Mark completion using sensor evidence or a verified event, not the model’s statement alone.

This separation also supports mixed fleets. A wheeled robot may expose navigation tools while a manipulator exposes grasping tools. ER 2 can coordinate the semantic workflow without pretending that all robots share the same motor interface.

Getting started with the Gemini API

The official robotics overview provides a Python example using the GenAI SDK and the Interactions API. The following reduced pattern sends an image and asks ER 2 to return normalized points. It is appropriate for a prototype perception step; it is not a complete grasp controller.

1from google import genai
2
3client = genai.Client()
4image = client.files.upload(file="workcell.png")
5
6prompt = """
7Point to no more than 10 items in the image.
8Return JSON in the form:
9[{"point": [y, x], "label": "object name"}]
10Use normalized coordinates from 0 to 1000.
11"""
12
13response = client.interactions.create(
14    model="gemini-robotics-er-2-preview",
15    input=[
16        {"type": "image", "uri": image.uri, "mime_type": image.mime_type},
17        {"type": "text", "text": prompt},
18    ],
19    generation_config={"thinking_level": "high"},
20)
21
22print(response.output_text)

Before connecting the output to a robot, parse it with a strict schema, reject coordinates outside the expected range, verify that the object is inside a permitted workspace, and run a second perception or sensor check for high-precision actions. Google’s documentation recommends improving the visual input by cropping or zooming unclear objects, breaking complex tasks into steps, and querying multiple times for high-precision spatial outputs.

The model ID is not a substitute for dependency and API-version pinning. Keep the SDK version, model ID, prompt schema, and response schema under test. Store representative images and expected tool decisions so that a model upgrade can be evaluated against the same cases.

Streaming for real-time orchestration

ER 2 Streaming is designed for the Gemini Live API. The goal is a bidirectional session in which the robot can send continuous audio/video and receive low-latency reasoning and function calls. The model is still a planner and orchestrator; the low-level movement loop should remain deterministic enough to stop quickly when sensors detect danger.

Visual: streaming request and approval flow. The application policy layer remains between model output and actuation. This prevents a prompt injection in a scene, an ambiguous object label, or a stale tool result from becoming an unreviewed physical command.

For latency-sensitive deployments, measure the complete loop rather than only model response time: capture delay, upload time, inference time, tool validation, actuator response, and sensor confirmation. Google says the streaming endpoint is optimized for low-latency robotics, but the end-to-end result depends on the camera pipeline, network, robot SDK, and safety controller.

Progress understanding and moment finding

ER 2’s most useful upgrade is temporal reasoning. Google reports 57.4% accuracy for continuous progress classification and 91.3% accuracy for moment finding, with a 0.96-second mean absolute distance in its reported evaluation. These are Google’s evaluations, not a guarantee for a particular camera, task, robot, or environment; they should be treated as starting points for a local benchmark.

Progress classification divides a task into five ranges, from 0–20% through 80–100%. Moment finding asks the model to identify when a critical event occurred—for example, when pouring should stop. These capabilities can reduce a common failure mode in long workflows: executing the next action because a model assumed the previous action finished.

A robust implementation should combine model judgment with independent evidence:

This is where a robotics application differs from a text agent. A plausible answer is not proof of a safe physical state.

Tools, multi-robot workflows, and security

Google’s release describes ER 2 calling Google Search or user-defined functions and coordinating multiple robots. That flexibility needs a capability model. Each tool should declare its input schema, permitted targets, maximum duration, required identity, and whether human approval is mandatory.

Do not expose these primitives directly to the model:

• unrestricted shell execution;
• arbitrary HTTP requests with production credentials;
• a tool that accepts raw motor commands;
• a database writer without tenant and authorization checks;
• a function that hides side effects behind a vague name such as do_task.

Treat all scene content as untrusted input. Signs, screens, labels, and audio can contain instructions that conflict with the user’s task. Keep system policy and tool authorization outside the visual prompt, and re-check authorization immediately before execution. Log the original observation reference, normalized tool arguments, policy decision, actuator result, and stop reason while redacting personal data and secrets.

For multi-robot collaboration, route messages through an application coordinator. A robot should not automatically trust a peer’s claim that an object was moved or a workspace is clear. Use signed identities where appropriate, explicit task ownership, timeouts, idempotency keys, and a shared state store with conflict handling.

Google’s model card also contains a privacy warning: video and audio may include identifiable people, and operators must handle notice, consent, and data-use obligations under the applicable Gemini API terms. Design the capture pipeline to minimize collection, blur or exclude people where possible, restrict retention, and document where data is processed.

Common implementation mistakes

Treating the model as the safety controller

The model can recommend stopping, but the emergency stop and protective limits must work when the model, network, or application is unavailable. Put hard speed, force, workspace, and proximity limits in the robot-control layer.

Sending too much video

High-resolution frames and high thinking levels can increase processing time and cost. Crop the relevant region, choose an appropriate frame rate, and measure whether extra visual detail changes the decision enough to justify the latency.

Assuming a completion sentence is a postcondition

Require evidence. “The cup is on the shelf” should be checked by a camera, force sensor, RFID tag, or another deterministic signal suited to the task.

Migrating ER 1.6 without regression tests

The official docs specify the replacement model IDs, but model behavior, tool selection, latency, and output shape still require testing. Replay your failure cases before switching a live robot fleet.

Ignoring API-key restrictions

Google’s documentation warns that unrestricted API keys can produce a 403 and should be secured with restrictions. Keep keys server-side, use environment-specific credentials, limit project permissions, and never place a key in robot firmware or a browser bundle.

When ER 2 is a good fit

ER 2 is a strong candidate for prototypes and bounded production workflows that need multimodal scene understanding, natural-language task decomposition, progress tracking, or coordination across heterogeneous robot APIs. It is less suitable as the sole controller for safety-critical work, hard real-time motion, or deployments where connectivity and cloud processing are unacceptable.

The best first project is a non-critical workcell with a small tool set, synthetic or de-identified data, an operator-visible approval UI, and replayable evaluation cases. Start with perception and planning, then add one reversible action at a time.

FAQ

Is Gemini Robotics ER 2 the same as Gemini Robotics 2?

No. ER 2 is the high-level embodied-reasoning VLM. Google’s broader Robotics 2 release also describes VLA and On-Device models that execute actions or run locally; those models have separate availability and hardware considerations.

Which model ID should a new prototype use?

Use gemini-robotics-er-2-preview for ordinary image/video reasoning and tool orchestration. Use gemini-robotics-er-2-streaming-preview with the Live API when continuous, low-latency audio/video interaction is the central requirement.

Can ER 2 safely control a robot by itself?

No. The model can make mistakes, and Google explicitly places responsibility for the safe environment on the developer. Keep authorization, physical limits, emergency stops, and completion checks outside the model.

Does this replace a VLA model?

No. ER 2 is designed to reason and orchestrate. A lower-level VLA, navigation stack, or robot API performs the actual movement.

Conclusion

Gemini Robotics ER 2 is most useful when treated as a multimodal task supervisor rather than a magical robot brain. Its standard and streaming endpoints, progress understanding, moment finding, tool orchestration, and multi-robot coordination create a practical high-level layer for physical AI. The production pattern is straightforward but non-negotiable: narrow tools, external policy checks, independent sensor evidence, privacy controls, and a deterministic safety system between every model decision and actuator.

For adjacent agent architecture, compare this layered design with the Google Managed Agents guide, the Gemini 3.7 Flash developer guide, and the OpenClaw runtime security guide. Those systems solve different problems, but all benefit from the same rule: models may propose actions; software must enforce what is allowed.

Sources and visual credits

• Google DeepMind: Introducing Gemini Robotics ER 2 — release details, architecture, reported evaluations, and safety claims.
• Google AI for Developers: Gemini Robotics ER overview — model IDs, Python example, streaming links, limitations, and privacy guidance.
• Google DeepMind model card: Gemini Robotics ER 2 — model basis, inputs, distribution, limitations, and intended use.
• Google DeepMind: Gemini Robotics 2 — ER/VLA/On-Device model roles and multi-robot context.
• Ars Technica independent coverage — independent reporting and technical context.
• Google Gemini robotics samples — official sample repository.

The Mermaid diagrams are original editorial illustrations based on the official ER 2 architecture and API documentation. The comparison table and request-flow diagram are original synthesis; no product screenshot or benchmark visual has been invented.