Exploring Autonomous Driving Through Mixed Reality

An autonomous vehicle shows nothing of what it sees. A visitor can stand right next to the fleet and still have no way to tell what the car recognises, or how sure it is.

BeIntelli AR is a mixed-reality tool that makes that perception readable for someone with no background: the visitor stands next to the fleet and sees what it perceives, one layer at a time.

The app had to work in two demo contexts. Without the car: a 1:1 3D model of vehicle and delivery robot, with sensor data and point clouds on the virtual asset. With the car on site: the same layers overlaid on the real fleet, live. Treating these as one product, not two, was a deliberate choice.

Validated with about ten internal testers, most of them first-time mixed-reality users, on one question: can each layer be read in a few seconds?

The design question was which signals from the perception stack to show at all.

Layers, switchable one by one. The live LiDAR point cloud, the sensor cones, the sensor positions on the car, and what the vehicle currently recognises as an object. The presenter can start with the simplest layer and add detail.

Live, not recorded. With the real vehicle on site, the point clouds and detections come from the live pipeline, so what a visitor sees is what the car perceives at that moment.

Density for a moving viewer. In mixed reality the gaze is unconstrained and the visitor walks around. The dominant signal from any angle is the recognised object and the live point cloud; sensor positions and telemetry appear only when someone steps closer.

As Product Owner I had to keep testing whether a designed interaction survived the constraints of the medium. An overlay that reads well in Figma can be unreadable through a headset at two metres, and the only way to know is to build it and walk around it.