Mobile Multi Sensory Platform

The Mobile Multi Sensory Platform (MMSP) was the first multi sensory system by AIT which can be used in VR scenarios to simulate wind, heat, rain and pain. A later version also includes scent. It was built as part of the H2020 funded SHOTPROS project.

Supplementary video of the publication below showcasing the features of the MMSP.

Publications

Uhl, J. C., Regal, G., Gafert, M., Murtinger, M., & Tscheligi, M. (2023). Stress Embodied: Developing Multi-sensory Experiences for VR Police Training. In Lecture Notes in Computer Science (pp. 573–583). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42280-5_36

Wind, Heat, Rain

For wind, heat and rain a blower, heat blower and a mist system are mounted to a steel frame. Each component is controlled by a central Raspberry PI which uses sliders and buttons as input for haptic intensitiy control. The input system used is a Monogram Palette controlled by Ambit which sends commands via a unix pipe to the custom C++ program on the Raspberry PI.

To control the wind the blower’s different 230V intensity circuits are connected by multiple 5V relays. The mist and heat components are controlled by a hardware PWM signal created with pigpio boosted to 12/24V by some mosfets which are powered by two 12/24V power supplies. For the mist the 12V signal is fed straight into the pump. For the heat the PWM signal is fed into a PWM gas valve which controls the flow of gas to the blower. For safety reasons the heat blower’s main gas valve and ignition have to be turned on manually.

The steel frame of the MMSP with wheel and flame resitent boards.

Pain

To simulate pain four Pavlok 2 shock bracelets are used. Altough, there is no open API the BLE GATT charactersitics are not protected and any BLE device can connect with the bracelets and send commands. The SimpleBLE library was used to connect the Pavlok BLE devices.

From left to right: First test of BLE communication with the Pavlok, testing the shock feature, controlling the Pavlok with the MMSP haptic interface.

Scent

To simulate scent an additonal system was built which extended the MMSP to control OVR ION 2 scent devices. Altough, the OVR devices can also be controlled over BLE we created a MQTT to BLE bridge to control the OVR devices over MQTT to cover a larger area than bluetooth. To accomplish this each participant wearing the OVR device wore an additonal ESP32 based board with an attached battery. The ESP32 is capable of BLE and Wifi. esp32-ble2mqtt allowed us the control the BLE devices with MQTT messages sent from the Raspberry PI. In smaller version the program is also ported to Windows and can be used without the other components of the MMSP.

Completed MMSP with heat, wind, rain, shock and haptic interface.