In situations where it is difficult to perform robotic laboratory tasks with physical hardware - such as handling radioactive materials, or collecting imitation learning demonstrations at scale - it is useful to be able to simulate the robotic laboratory environment. Mixed Reality interfaces enable operators to interact with these simulations in an intuitive and immersive format. In order to achieve quality, accurate control from the operator, it is important that the simulated hardware, environment, and interaction paradigms are as good as they can be. In this work, we perform a survey of existing techniques for robot simulation, robot control, 3D scene reconstruction, and VR user interaction, and propose methodology for examining the most effective combination of methods for remote teleoperation. Further, we examine how to address user-perceived differences between the simulation interface and the physical control interface.

PDF

Brynteson, H., Angara, A., Atlas, Y., Hyatt, M., Han, M., Insley, J., Ferrier, N., Kim, Y., Knowles, J., Mateevitsi, V., Papka, M., Rizzi, S., Using Virtual Reality for Remote Dexterous Manipulation: An Exploration of Methods, Society for Laboratory Automation and Screening 2026, Boston, MA, February 7th, 2026.