Exploring Autonomous Driving Through Mixed Reality

Within the BeIntelli project I held a dual role as Product Owner and sole UX/UI designer. Product ownership meant negotiating requirements with the BeIntelli autonomous-driving team, prioritising scope, and briefing the mixed-reality engineer who carried out the technical implementation. Design ownership covered the native app, the spatial overlays, the information layer, the interaction model, and the user flow end-to-end.

Early in the project, it became clear that the application had to serve two very different demo contexts. In contexts where the physical vehicle could not be brought along (international trade shows, public outreach events), the application works in a fully virtual mode: a 1:1 3D model of the vehicle and the delivery robot, with sensor and point-cloud information rendered on the virtual asset. In contexts where the real vehicle is on site, the application switches to a combined real-and-virtual mode: visitors and presenters can walk around the actual car, while the spatial layer overlays sensor positions, LiDAR point clouds, and live cloud data directly onto the real fleet. Treating these as one product, not two, was a deliberate choice.

The design iterations were validated through internal user testing with approximately ten participants from the broader DAI-Labor and TU Berlin context, focused on first-time mixed-reality users and on how clearly each layer of sensor information could be read in a few seconds.

BeIntelli AR turns the autonomous fleet's machine-perception into a spatial visualisation that a non-expert visitor can read in seconds. The design challenge was deciding which signals from a complex perception stack should be made visible at all, and how each should be expressed in a shared mixed-reality vocabulary.

The perception stack as a layered visualisation. The MR scene composes several layers: the real-time LiDAR point cloud emitted by the vehicle, the sensor cones and fields of view for each on-board sensor, the sensor positions on the vehicle itself (so a visitor can connect a point on the car to a specific sensor in the visualisation), and the object recognition layer, which highlights what the vehicle currently identifies as an object of interest. Each layer is independently switchable, so the presenter can guide a visitor from the simplest layer to the most detailed one.

Live data integration with the cloud. When the real vehicle is on site, the MR view is connected to the live data pipeline so that the point clouds, sensor cones, and object detections shown are not pre-recorded but reflect what the vehicle is perceiving at that moment. The supporting telemetry from the sensors is rendered alongside the visualisation as contextual information.

Spatial information density. Mixed reality demands different rules than a 2D screen: the user's gaze is unconstrained, the surrounding environment is part of the canvas, and the visitor is physically moving. The information density was tuned so that the most important signal (a recognised object, the live point cloud) is dominant in any viewing angle, and the secondary signals (sensor positions, telemetry) reveal themselves only when the visitor approaches that part of the vehicle. The same product therefore works equally well in the fully virtual demo mode and in the combined-with-real-vehicle mode: the information hierarchy adapts to context, the underlying design system does not change.

BeIntelli AR is in active use as the visual demonstrator for the BeIntelli initiative. The application has been shown at the Consumer Electronics Show (CES) in Las Vegas, and is part of the regular programme of the Lange Nacht der Wissenschaften, the long-running public science outreach event in Berlin.

Beyond these public events, the application is used continuously whenever partners or external visitors come to the lab. The fact that it has remained the standard visualisation tool across these very different audiences, from the international tech-industry context of CES to the academic outreach context of the Lange Nacht and to closed partner demos, is the validation that matters most for a research demonstrator of this kind.