Husarion Robot XL demo

This demo utilizes Open 3D Engine simulation and allows you to work with RAI on a small mobile platform in a nice apartment.

Quick start

LLM model

The demo uses the complex_model LLM configured in config.toml. The model should be a multimodal, tool-calling model. See Vendors.

ROS 2 Sourced

Make sure ROS 2 is sourced. (e.g. source /opt/ros/humble/setup.bash)

1. Follow the RAI setup instructions in the quick setup guide.

2. Download the newest binary release:

   ./scripts/download_demo.sh rosbot
   

3. Install and download required packages

   sudo apt install ros-${ROS_DISTRO}-navigation2 ros-${ROS_DISTRO}-nav2-bringup
   vcs import < demos.repos
   rosdep install --from-paths src --ignore-src -r -y
   poetry install --with perception
   colcon build
   

Alternative: Demo source build

If you would like more freedom to adapt the simulation to your needs, you can make changes using O3DE Editor and build the project yourself. Please refer to rai husarion rosbot xl demo for more details.

Running RAI

1. Running rai nodes and agents, navigation stack and O3DE simulation.

   source setup_shell.sh
   ros2 launch ./examples/rosbot-xl.launch.py game_launcher:=demo_assets/rosbot/RAIROSBotXLDemo/RAIROSBotXLDemo.GameLauncher
   

2. Run streamlit gui:

Choosing the right model

Due to the multimodal requirements of the demo (such as processing camera images), the best results will be achieved with tool-calling LLMs that support vision, such as those from OpenAI, Anthropic, and similar providers. You can use local models (e.g., gpt-oss-20b, qwen3) configured in config.toml, but with these, image-based tasks (like interpreting the robot's camera feed) will not work—the agent will have a reduced scope of capabilities.

source setup_shell.sh
streamlit run examples/rosbot-xl-demo.py

1. Play with the demo, prompting the agent to perform tasks. Here are some examples:

   • Where are you now?
   • What do you see?
   • What is the position of bed?
   • Navigate to the kitchen.
   • Please bring me coffee something from the kitchen (this one should be rejected thanks to robot embodiment module)

Changing camera view

To change camera in the simulation use 1,2,3 keys on your keyboard once it's window is focused.

How it works

The rosbot demo utilizes several components:

1. Vision processing using Grounded SAM 2 and Grounding DINO for object detection and segmentation. See RAI perception.
2. RAI agent to process the request and interact with environment via tool-calling mechanism.
3. Navigation is enabled via nav2 toolkit, which interacts with ROS 2 nav2 asynchronously by calling ros2 actions.
4. Embodiment of the Rosbot is achieved using RAI Whoami module. This makes RAI agent aware of the hardware platform and its capabilities.
5. Key informations can be provided to the agent in RAI Whoami. In this demo coordinates of Kitchen and Living Room are provided as capabilities.

The details of the demo can be checked in examples/rosbot-xl-demo.py.