Sonar on stock smartwatches leads to successful wrist tracking | Cornell Chronicle

Imagine double-clicking your thumb and index finger to skip to the next track or click on your laptop or desktop computer without a mouse, using smart fingers.

A new first-of-its-kind wearable technology from researchers at Cornell and KAIST, South Korea, brings that vision closer to reality. The system, called WatchHand, equips off-the-shelf smartwatches with a tiny AI-powered sonar capable of tracking hand movements.

The technology has the potential to change the way we interact with our devices by continuing to track hand gestures in real time using smartwatches and built-in speakers and microphones, the researchers said. It is the first time that AI-powered acoustic sensing for wrist tracking has been implemented in off-the-shelf smartwatches without the need for additional hardware.

“In the future, with this kind of hand tracking technology, we may be able to know how to type with our intelligence,” said Chi-Jung Lee, a doctoral student in the department of information science at the Cornell Ann S. Bower College of Computing and Information Science, and co-author of “WatchHand: Simplifying Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches,” will be presented at the Association for Computing Machinery (ACM) CHI conference on Human Factors in Computing Systems starting April 13 in Barcelona.

“Our hands can act as a computer input device,” he said.

Along with the interaction of gesture-based devices, direct use of the WatchHand can support assistive technologies for users with limited mobility or speech and be used as a controller in augmented reality and virtual reality situations, the researchers said.

This device represents a major breakthrough for human-computer interaction, said co-author Jiwan Kim, a doctoral student in electrical engineering at KAIST and a visiting researcher at SciFi Lab last year.

“WatchHand significantly lowers the barriers to wrist tracking,” said Kim. If any device has a single speaker and microphone, our method works.

Existing wrist-tracking prototypes require a lot of equipment, making them impractical for everyday use, the researchers said, but the WatchHand uses the microphone and speaker found in standard smartwatches. Featuring the WatchHand, the smartwatch’s speaker emits inaudible sound waves that bounce off the wrist and bounce back into the watch’s microphone, creating an echo profile. The WatchHand machine learning system, running on a smartwatch, reads this echo profile and estimates the appearance of the hand in 3D and in real time.

All hand data and processing will take place locally, meaning personal data will not be shared, the researchers said.

WatchHand was tested with 40 participants in four studies, a total of 36 hours of gesture data. It was tested across a variety of smartwatches, right-handed and left-handed, and in noisy environments, and was found to accurately track finger movements and wrist rotation.

Its performance is not perfect, the researchers noted. At launch, it works on Android smartwatches, not Apple iOS. While it worked well in noisy environments, the WatchHand had trouble registering hands if the user was walking, for example.

“WatchHand reflects my lab’s broader vision of turning everyday wearables into smart platforms that recognize behavior,” he said. Cheng Zhangassociate professor of information science Cornell Bowers and director of Cornell’s Smart Computer Interfaces for Future Interactions (SciFi) Labwhich leverages machine learning and AI to develop technologies that make sense of – and make sense of – the data we generate through our movements.

In the past, SciFi Lab scientists have designed, developed, and manufactured the following sensors hand, arm, tonqu, face, body and even teeth. They’ve put their technology into rings, glasses, necklaces and headphones, and sewn it into the seams of clothes. In recent years, the lab has turned to using acoustic sensing in wearables due to its accuracy and low power consumption.

“With just a software update, we can unlock new features from millions of existing devices,” Zhang said.

Along with Zhang, Lee, and Kim, co-authors of the paper are Ian Oakley and Hohurn Jung of KAIST and Tianhong doctoral students Catherine Yu and Ruidong Zhang.

This research received support from the National Science Foundation and the South Korean Ministry of Science and Information Technology.

Louis DiPietro is a professor in the Cornell Ann S. Bowers College of Computing and Information Science.