Researchers demo AR hand tracking controls for smartphones

Leap Motion suggested years ago that it was working to bring real time hand tracking to mobile devices — an innovation with obvious applications for virtual and augmented reality headsets, which it and Facebook’s Oculus unit subsequently have worked to commercialize. Now a group of Brown University researchers is showing off a different spin on the same hand tracking technology, applying it to the more popular medium of smartphone AR.

As of today, Brown HCI’s Portal-ble concept is somewhat rough around the edges, with a goal of enabling a hand-held smartphone to detect the user’s other hand for interactions with digital objects. Rather than using a smartphone with an integrated rear time of flight camera, the researchers paired an Android smartphone with a Leap Motion sensor and a battery-powered Intel Compute Stick to achieve their results; the PC provides the necessary computing horsepower to power the hand tracking.

On a positive note, the researchers were able to create viable AR usage paradigms for Portal-ble, including digital objects that users could manipulate with a free hand and place within real world spaces — while looking at the smartphone screen. The Leap Motion sensor works quickly and accurately enough to enable a hand to select between multiple items, while the system supports both translucent, boxy hand representations and occlusion of a more realistic hand model.

Of course, the major issues are the size, expense, and kludginess of the current hardware required to enable the feature, as well as the question as to whether users of smartphones would actually use hand tracking with one hand as they hold the device in another. The former issues are likely to become moot with the release of time-of-flight camera-packing smartphones like Samsung’s Galaxy S10 5G, while the latter might be resolved as similar technologies appear in Apple’s ARKit and Google’s ARCore smartphone development tools.

Developers interested in playing with the Portal-ble solution can get open-source software and instructions on 3D printing a Leap Motion mount from the researchers’ GitHub page. The researchers have also developed a case to wear the battery and Compute Stick PC at waist level while the smartphone and sensor are held in the hand.


Leave a Reply