Embodied AI / ROS 2 Jazzy / Param Rudra HPC, IIT Patna

R2-D2Redux

A robot that reasons before it moves

Say “go to the table and tell me what is on it.” A vision–language model reads the camera, a semantic map places the table in metres, a 20-billion-parameter planner writes a route and a sentence to say, and Nav2 drives the wheels there. Four machines, five services, one closed loop from photons to PWM.

4Machines

5Services

2Foundation models

5.5kLines of code

Fig. 00 — Platform, three-quarter view Differential drive

§ 01

The brief

Why this exists

Most small robots do one of two things well. They move — SLAM, a costmap, a planner, clean odometry — or they talk — a language model wired to a chat box, with no idea where it is. The interesting engineering lives in the seam between them, and the seam is where nothing is standardised.

R2-D2 Redux is that seam, built end to end. Perception runs on an A100 because a 7-billion-parameter vision model will not fit on a Raspberry Pi. Planning runs on the same node because a 20-billion-parameter planner will not either. The robot carries only what must be physically on the robot: sensors, wheels, and the code that streams one to the other. Everything in between is a network problem — WebSockets, HTTP, an SSH tunnel, and a lot of care about what happens when one of them drops.

The claim

A natural-language instruction becomes a metric navigation goal without a human ever touching a coordinate.

PerceptionQwen2-VL-7B-Instruct

PlanningGPT-OSS 20B

Mappingslam_toolbox + custom fusion

MotionNav2 → micro-ROS → PWM

TransportWebSocket · HTTP · autossh

§ 02

The signal path

Sense flows out, command flows back — the loop closes at the wheels

Sensing — robot to reasoning Command — reasoning to wheels Dashed rule — machine boundary

§ 03

Four machines

Split by what each one is physically good for

Bare metal

ESP32 microcontroller

Runs micro-ROS over serial and nothing else. Subscribes to /cmd_vel, converts linear and angular velocity into two 8-bit PWM channels, and blinks an LED at 10 Hz if the ROS agent ever drops.

Firmwareesp32_drive.ino

MessageTwistStamped

PWM5 kHz · 8-bit

Onboard Linux

Raspberry Pi

Carries the sensors and the network. Runs ROS 2 drivers, slam_toolbox and Nav2 locally, and streams every camera frame, laser scan, IMU sample and transform up to the brain as base64 JSON over a WebSocket.

Noder2d2_bridge

SensorsLIDAR · RGB · IMU

UplinkWebSocket, 10 MB frames

HPC

GPU node

The part that thinks. A 7B vision model and a 20B planner share an A100, alongside the hub that fans sensor data out to them and the semantic map that turns their output into coordinates.

Services4

Ports5000 · 8080 · 9001–9004

ClusterParam Rudra, IIT Patna

Bridge

Laptop

Holds an autossh tunnel open to the compute node so the HPC job can be reached at all, and translates the planner's HTTP goals into Nav2 action calls on the robot's ROS 2 graph.

Noder2d2_reciever

APIPOST /navigate · /speak

Keepaliveautossh, 30 s

§ 04

Service sheet

Every long-running process, where it lives, what it listens on

Service	Host	Port	Responsibility
Brain hubsrc/r2d2_brain/brain.py	GPU node	9001–9004	Terminates the robot's WebSocket, buffers every message to disk on writer threads, samples one frame in five to the vision model, and re-broadcasts enriched frames to the map.
Vision modelsrc/r2d2_brain/vlm.py	GPU node	5000	Flask endpoint wrapping Qwen2-VL-7B-Instruct. Takes a base64 JPEG, returns objects with boxes and an importance rank, a scene summary, and navigation advice — strictly as JSON.
Semantic mapsrc/r2d2_brain/sementic_map.py	GPU node	8080	Fuses laser scans into a 400 × 400 occupancy grid and paints vision labels onto it, so “the chair” becomes a coordinate. Serves the map over REST and checkpoints it every minute.
Plannersrc/r2d2_brain/gpt_oss.py	GPU node	7000	The hinge. Assembles world context, prompts GPT-OSS 20B for a JSON plan, validates it, dispatches goals and speech, and records what worked in SQLite for later reuse.
Nav2 bridgesrc/r2d2_comms/…/r2d2_reciever.py	Laptop	8888	A Flask server living inside a ROS 2 node. Converts {x, y} into a PoseStamped goal on the navigate_to_pose action, and text into the TTS topic.
Sensor bridgesrc/r2d2_comms/…/r2d2_bridge.py	Raspberry Pi	—	Subscribes to every sensor topic on the robot, encodes frames, and keeps the uplink alive with heartbeats and reconnects.

§ 05

One sentence, traced

“Go to the table and tell me what is on it” — every hop it takes

The instruction arrives

Typed into the command client, or straight into the planner's own terminal. Anything that isn't a built-in like observe or locate is treated as natural language.

command_client.py → ws :7000

The world is described

The planner pulls the robot's current pose, the object list from the semantic map, and a fresh reading of the camera from the vision model. All of it becomes prose in the prompt — positions in metres, objects grouped by type.

GET :8080/map · POST :5000/process_frame

Memory is consulted

If a similar instruction has succeeded before, its stored plan is injected into the prompt as a worked example. The robot gets better at the routes it has already driven.

navigation_tasks.sqlite

The plan is generated

GPT-OSS 20B returns reasoning, a list of metric waypoints, and the sentences to speak. The output is parsed defensively — models do not always honour “start with a brace”.

GPT-OSS 20B · 1024 tokens

Goals become motion

Each waypoint is POSTed to the laptop, promoted to a PoseStamped with a yaw quaternion, and handed to Nav2. Nav2's controller emits velocity, the ESP32 turns it into PWM, the robot moves — and the camera sees something new.

POST :8888/navigate → navigate_to_pose

Plan contract — planner output strict JSON

// the only shape the planner is allowed to emit
{
  "reasoning": "The table is the object at (2.5, 1.3).
                Approach from the near side, then look.",
  "navigation": [
    {
      "x": 2.5,
      "y": 1.3,
      "description": "Navigate to the table"
    }
  ],
  "speech": [
    "I'll go to the table now.",
    "I've reached the table. I can see
     several objects on it."
  ]
}

Coordinates are in metres in the map frame, not pixels and not vibes. That constraint is what makes the language model's output executable: everything downstream of this JSON is ordinary, deterministic robotics.

§ 06

Seeing, and remembering

A language model's description is useless until it has coordinates attached

Vision contract — vlm.py Qwen2-VL-7B

{
  "objects": [{
    "name":        "chair",
    "description": "wooden chair, facing left",
    "box_2d":      [412, 208, 640, 512],
    "importance":  4   // 1–5, to a navigator
  }],
  "scene_inference":   "A cluttered lab bench ahead.",
  "navigation_advice": "Keep left of the chair."
}

Asking a vision model for a caption gives you a caption. Asking it for a typed record — boxes, a relevance rank, one line of advice — gives you something a planner can act on and a map can index. Identical frames are hashed and cached, because the robot spends a lot of time looking at the same wall.

Three side-by-side grids: an occupancy grid showing scanned free space and walls, an empty semantic grid, and the two blended together, with a legend of nine semantic classes. — Fig. 01 — Occupancy × semantics, blended 20 × 20 m @ 0.05 m

The map is a 400 × 400 grid holding two layers: geometry from the laser, and one of nine semantic classes — chair, table, door, wall, person, obstacle — projected from whatever the vision model named in the frame the robot was looking at when it scanned. Ask for “the table” and the planner reads a coordinate out of this grid.

§ 07

The hard parts

What actually took the time

Async correlation

Matching a slow answer to a fast frame

Vision inference takes seconds; the camera does not wait. Frames go into a dictionary keyed by frame ID under a lock, the vision thread reunites its result with the original frame when it finishes, and the dictionary evicts its oldest entries past a hundred so a stalled GPU can't leak memory forever.

Back-pressure

Deciding what to drop

Bounded queues sit in front of both disk and GPU, and each one has an explicit answer for being full: the writer warns and keeps the newest, the vision queue skips the frame. Nothing blocks the WebSocket reader, because a blocked reader means a robot that stops streaming.

Fragile links

Assuming the network will fail

An HPC job behind a login node is not reachable from a robot. autossh forwards six ports to the compute node and rebuilds the tunnel when it dies; vision calls retry three times with a backoff; every socket sets TCP_NODELAY so small control messages don't wait on Nagle.

Model output

Trusting a language model exactly as far as you can parse it

The planner is told to emit nothing but JSON, and then treated as if it won't. Responses are scraped for the outermost brace pair, validated field by field, and rejected outright if the coordinates aren't numbers — a bad plan should be a logged failure, not a robot driving into a wall.

Memory

Getting better at the same errand

Successful runs — instruction, reasoning, waypoints, speech — are written to SQLite. On a similar instruction the closest match is replayed into the prompt as a worked example, which costs nothing at inference time and sharply steadies the output format.

Observability

Being able to see what it thought

Every message is appended to a JSONL log with its type, frame number and timestamp; the hub prints throughput, queue high-water marks and vision error counts every minute. When the robot does something strange, the transcript of what it saw is still on disk.

§ 08

The build

Laser-cut acrylic, three decks, everything reachable

Front view of the robot showing the LIDAR mounted directly above the webcam. — **Front elevation**Camera and LIDAR share an optical axis, so a bearing from the laser and a bearing from the image mean the same thing without calibration gymnastics.

Three-quarter view showing the Raspberry Pi, wiring loom, battery and XT60 connector. — **Power and compute deck**The Pi sits on the middle deck with the loom routed to the edges; the pack and its XT60 live on the bottom deck, keeping mass low over the axle.

Close-up of the webcam mounted on the top deck of the robot. — **Sensor head**An ordinary autofocus webcam. Every frame it produces is JPEG encoded, base64 wrapped, and sent a few hundred kilometres to be looked at by an A100.

§ 09

Bringing it up

Order matters — each stage waits on the one before it

1 — GPU node4 processes

# vision first: it loads 7B weights
python3 src/r2d2_brain/vlm.py

# then the hub the robot dials into
python3 src/r2d2_brain/brain.py

# the map subscribes to the hub
python3 src/r2d2_brain/sementic_map.py

# the planner needs all three
CUDA_VISIBLE_DEVICES=0 \
  python3 src/r2d2_brain/gpt_oss.py

2 — Laptoptunnel + bridge

# forward 5000, 8080, 9001-9004
bash start_brain_tunnel.sh

# ROS 2 side, serves :8888
ros2 run r2d2_comms r2d2_reciever

3 — Robotlast

# drivers, slam_toolbox, nav2, then
ros2 run r2d2_comms r2d2_bridge

4 — Talk to itany host

python3 src/command_client.py

> what do you see?
> find the chair
> go to the table and tell me
  what is on it

Built-ins sit alongside the free-form commands: observe for a fresh look, locate <object> for a laser range to something named, context for pose and map contents, history for what has worked before.