Open-RUA/uni-robot-agent

uni-robot-agent is an open-source harness for robot-use agents, with Codex and Claude Code runtimes.

Quick start · Python SDK · Dashboard · Environments · How it works · Contributing · License · Citation

What's New

v0.1.1 fixes fresh RoboDojo setup with isolated simulator dependencies and automatic robot configuration. See pip and uv installation below.

Features

Use one package to run a built-in agent, call robot tools from your own Python strategy, and inspect live or saved episodes. Direct mode uses robot tools; Hybrid mode adds a compatible action model such as Pi0.5. Prompts, skills and the dashboard are included in the package.

The harness follows four design principles:

• Unified: shared tool interfaces and feedback across environments, with native APIs handled by adapters.
• Efficient: overlap agent reasoning with tool execution while preserving order for operations that share robot resources.
• Realistic: support the paper's RGB-only setting, using proprioception, calibration and robot descriptions without simulator depth or task-verifier feedback to the agent.
• Modular: replace the agent runtime, model provider, tools, environment or model services through explicit interfaces.

RoboDojo enables depth by default. Select --no-depth for the paper's RGB-only setting and retain camera calibration. See design principles and observation settings.

Quick start

1. Install and prepare RoboDojo. Use a Python 3.11 virtual environment on a Linux GPU host.

pip install uni-robot-agent  # or: uv pip install uni-robot-agent
uni-robot-agent setup --integration robodojo

2. Run a task. Set your vision model and Responses-compatible endpoint:

export UNI_ROBOT_AGENT_API_KEY='YOUR_API_KEY'
export UNI_ROBOT_AGENT_PROVIDER_URL='https://provider.example/v1'
uni-robot-agent run --integration robodojo --deploy local \
  --task stack_bowls --model YOUR_MODEL_ID \
  --action-profile direct --no-sam3 --output-dir ./runs --monitor

Open the printed dashboard URL; results are saved under ./runs. More options.

Python SDK

Run the agent against an already running environment. This uses the same API key as the CLI; replace the model and provider placeholders with your settings:

from uni_robot_agent import AgentConfig, SessionConfig, run
result = run(SessionConfig(
    integration="robodojo",
    env_url="http://127.0.0.1:18760",
    no_sam3=True,
    agent=AgentConfig(
        model="YOUR_MODEL_ID",
        provider_url="https://provider.example/v1",
    ),
    output_dir="runs",
))
print(result.agent_finished, result.task_success, result.session_dir)

agent_finished records agent completion; task_success is the environment's reported outcome, or None when unavailable. Use Session.tools to implement a custom strategy, RunRecord to read saved runs, and setup() / deploy() to own local services explicitly. See the SDK guide and Python examples.

Dashboard

Add --monitor to a CLI run to view camera streams, tool calls, execution timing and available task outcomes. The monitor prints a local URL; no frontend build or upload is required.

Replay saved episodes or browse all runs under a parent directory:

uni-robot-agent monitor --run-dir ./runs

Select an episode, play or pause the recording, switch cameras and jump between tool calls. See live monitoring and replay for remote-host access and other options.

Environments

Simulation

| Integration | Environments and tasks | Guide | |---|---|---| | RoboDojo | Dual-arm manipulation with Direct tools or Hybrid Pi0.5 control | Setup | | RoboUse | Selected tasks from LIBERO, RoboCasa, MetaWorld, RoboSuite, ManiSkill, Tabletop, Fetch and Kitchen; also Menagerie, composed-robot, Genesis and Factory task adapters | Tasks | | RoboLab | Single-DROID tasks in a separately prepared Isaac Sim / Isaac Lab runtime | Integration |

Real robots

| Platform | Public release status | |---|---| | ARX One | Coming soon | | Franka | Coming soon |

Real-robot integrations and deployment guides are coming soon. The public package does not yet include a supported physical-robot connection workflow.

How it works

The executor schedules tool calls from agents and the Python SDK, and shared robot tools access native backends through services. Architecture guide

Contributing

Follow CONTRIBUTING.md to set up a development environment, or open an issue with a minimal reproduction, package version, platform and relevant logs.

License

This project is licensed under the MIT License.

Upstream code, fonts, and external assets and models retain their respective licenses.

Citation

If you use uni-robot-agent in your research, please cite our paper:

@article{yuan2026democratizing,
  title={Democratizing Robot-Use Agents: A Unified Harness with Open Models},
  author={Yuan, Jiarui and Zuo, Yuxin and Huang, Xiaoyu and Zhang, Wenbo and Park, Kwanyoung and Yang, Junlin and Zhang, Tianxiang and Chen, Ruoqu and Ouyang, Yutao and He, Bingxiang and others},
  journal={arXiv preprint arXiv:2611.01560},
  year={2026}
}