TeleOperationStation

To explore user interfaces for teleoperation I developed a mixed reality vehicle simulation using a custom-built miniature vehicle. The miniature vehicle is placed in a miniature environment built from paper a carton boxes. The operator can control the miniature vehicle remotely from another room through the environment from A to B. To get a higher situational awareness the teleoperation station was developed as a VR application putting the operator in the center of four live camera feeds.

This research is part of the H2020 project AWARD.

Relevant Publications

Gafert, M., Mirnig, A. G., Fröhlich, P., Kraut, V., Anzur, Z., & Tscheligi, M. (2023). Effective remote automated vehicle operation: a mixed reality contextual comparison study. Personal and Ubiquitous Computing, 27(6), 2321–2338. https://doi.org/10.1007/s00779-023-01782-5

Gafert, M., Mirnig, A. G., Fröhlich, P., & Tscheligi, M. (2022). TeleOperationStation: XR-Exploration of User Interfaces for Remote Automated Vehicle Operation. CHI Conference on Human Factors in Computing Systems Extended Abstracts, 1–4. https://doi.org/10.1145/3491101.3519882

Gafert, M., Fröhlich, P., Ritzinger, U. M., & Baldauf, M. (2021). Challenges for Future Automated Logistics Fleet Interactions. Proceedings of the Workshop on Automation Experience at the Workplace Co-Located with the ACM Conference on Human Factors in Computing Systems (CHI 2021). https://ceur-ws.org/Vol-2905/paper7.pdf

Miniature Vehicle

The 1:16 vehicle is custom 3D designed and printed to enclose a Raspberry PI 4 connected to four Arducam B0261 120° cameras which stream a MJPEG stream to the operator (using a Unity VR application). The drive train consists of a Nema 8 8HS15-0604S stepper motor and a TMC2209 stepper driver. The steering controlled by a servo. Given the power limitations of the stepper a slow acceleration curve was used which mimics the behavior of a truck.

The steering and driving commands are sent over websockets to the Raspberry C++ application and parsed with json by nlohmann. The websocket server is started on the Raspberry using TinyWebsockets. The application uses pigpio to control the relevant actors by hardware PWM. The MJPEG streams of the camera are sent with mjpg-streamer.

3D Model

spin me around

Interfaces and Unity Application

The operator is surrounded by four live camera feeds depicted on large curved monitors. In front of the monitors different user interfaces which assist the operator are depicted. Depending on the scenario different interfaces are shown to allow a comparison between a set of user interface requirements.

The application is developed in Unity and uses the Varjo XR-3 headset to allow the operator to see his or her real hands and steering wheel.

The reactive interfaces are developed in C++ using Dear ImGui rendering textures which are shared with Unity. For input a Fanatec Steering wheel and pedals are used. Their input is serialized to JSON and sent over websockets to the miniature vehicle using NativeWebSocket. The MJPEG stream is parsed in C# as I did not find a suitable alternative. This creates the biggest bottleneck and limits the resolution of the four streams to 640x480.