Towards Device-Free Gaming with mmWave Radar

Given the significant amount of time that people spend indoors, public spaces are reinventing themselves to deliver immersive experiences. One approach is to provide interactive gaming, such as the spaces created in big international airports like Singapore. The problem is that these gaming areas rely on either (i) wearable devices, which can easily spread germs, particularly in highly transited areas, such as those in airports, or (ii) cameras, which are increasingly raising privacy concerns and are being forbidden in some public areas. To provide an immersive gaming experience that is device-free and privacy-aware, we propose a system based on mmWave radar. In particular, our work provides three contributions. (1) For the first time, we compare various tracking and human pose estimation (HPE) models in the SoA under a unified framework. This approach allows us to identify the best methods in terms of position accuracy, latency, and smoothness-which are critical gaming metrics. (2) We build an integrated system with tracking and HPE capabilities. Our system includes the design of four games with different levels of complexity and a dataset collected specifically to train models for gaming applications. (3) Our evaluation, which includes a subjective users' survey and an objective comparison with a Kinect console, shows that today's mmWave is suitable for games where only the user's location is required or when coarse-grained gestures are needed. There are two key areas where mmWave research needs to improve to match camera-based consoles: sensors need a higher sampling rate, since a high frame rate is critical for seamless gaming, and HPE models need to master fine-grained gestures, in particular when arms end up in front of the body. All the code and datasets will be made available as a stepping stone for future work.