Imaginary Interfaces: Spatial Interaction with Empty Hands and Without Visual Feedback

title_scene_frame "Imaginary Interfaces" is a research project by Sean Gustafson and Daniel Bierwirth at the Human-Computer Interaction Lab of Prof. Patrick Baudisch at Hasso Plattner Institute in Potsdam, Germany.

http://www.hpi.uni-potsdam.de/baudisch/projects/imaginary_interfaces.html

Technical Insights (Video):

The prototype shown in this video was created to demonstrate Sean's idea of interacting with screen-less wearable devices. It was primarily developed by Daniel Bierwirth as part of the "Imaginary Interfaces" project.

Components of the prototype

The Imaginary Interfaces prototype consists of following components:
1. Chest-worn fisheye camera with an attached ring of infrared LEDs
2. Processing unit to sense the gestures performed by the user
3. Remote visualization unit to display the results

camera_device_frame

Technical Implementation

The Imaginary Interface prototype senses spatial interaction of the user with a chest-worn fisheye camera. The screen-less device provides no visual feedback. Despite this, users can interact effectively as their hand movements are maintained in their visual short-term memory. The gestures of the user's empty hands are processed and wirelessly transmitted to produce visual output for a remote communication partner.

camera_device_hand_front_frame camera_device_hand_front_interact_frame

Detection Process

The main task of the detection process is the segmentation of the user's hands as foreground objects from the background. Since the screen-less wearable device enables the user to interact in mobile context, background subtraction will not work in the case of imaginary interfaces. We therefore implemented the infrared tracking system by mounting a ring of infrared LEDs onto the fisheye camera. The self illuminated device will illuminate the hands of a user with invisible light. As a result, we are able to sense the hands as the brightest spots since (1) the skin will reflect the infrared light and (2) the hands are the closest objects towards the camera unit.

hand_processing_frame processing_hand_shape_frame

Once both hands are identified as bright spots we classify the spots to identify the L-shaped and the drawing hand. This is done by applying following steps to the raw camera image: (1) threshold and smooth image, (2) determine contours, (3) determine feature points of contours. We are analyzing the contours in order to identify each hand with the respective feature points. In addition, the contour of the drawing hand is processed to determine whether the user wants to draw or simply moves his hand in front of the camera. To interact with the imaginary interface the user pinches his hand and starts drawing. The pinching gesture enables us to search for the pinch hole which is the inner contour of the drawing hand. The contour of the L-shaped hand is used to define the coordinate origin of the imaginary drawing pane. The feature points of thumb and pointing finger are used to define the x-and y axis of the imaginary coordinate system.

Visualizing Results to the Remote Communication Partner

vis_result_with_hand_frame

The visualization of the sensed interaction can be one out of several possible prototype scenarios. As shown in the demo video, the prototype implementation allows the user to draw shapes on his imaginary interfaces. These shapes will then be visualized to the remote communication partner. In our prototype implementation, the remote communication partner simply starts the imaginary client application on his iPad. That enables him to receive the tracking results wirelessly on his device, shown as visual drawings. The visualized results will simply be the shapes drawn by the user on his imaginary interface with a line smoothing effect applied to it. In addition, we visualize the L-shaped hand defining the coordinate origin of the imaginary interface as well as the realtime position of the user's drawing hand.

vis_result_closeup_frame