With touch screens becoming increasingly popular, classic interaction techniques as a mouse and keyboard are becoming less frequently used. An example of a breakthrough was the Apple's iPhone which was released in summer 2007 Now an entire surface can be used for input. However, this form of interaction is specifically designed for 2D surfaces.

Now Fraunhofer FIT has developed the next generation of multi-touch environment in a system that detects multiple fingers and hands at the same time and allows the user to interact with objects on a display. The users move their hands and fingers in the air and the system automatically recognises and interprets the gestures accordingly.

The science-fiction thriller Minority Report from 2002 which starred Tom Cruise in a 3-D software arena able to interact with numerous programs at unimaginable speed, used special gloves and only three fingers from each hand.

The FIT prototype provides the next generation of gesture-based interaction tracks the user's hand in front of a 3-D camera. The 3-D camera uses the time of flight principle, in this approach each pixel is tracked and the length of time it takes light to be filmed travelling to and from the tracked object is determined. This allows for the calculation of the distance between the camera and the tracked object.

"A special image analysis algorithm was developed which filters out the positions of the hands and fingers. This is achieved in real-time through the use of intelligent filtering of the incoming data. The raw data can be viewed as a kind of 3-D mountain landscape, with the peak regions representing the hands or fingers." said (right) Georg Hackenberg, who developed the system as part of his Master's thesis.

In addition plausibility criteria are used, these are based around: the size of a hand, finger length and the potential coordinates.
A user study was conducted and found that the system both easy to use and fun. However, work remains to be done on removing elements which confuses the system, for example reflections caused by wristwatches and palms which are positioned orthogonal to the camera.

"With Microsoft announcing Project Natal, it is likely that similar techniques will very soon become standard across the gaming industry. This technology also opens up the potential for new solutions in the range of other application domains, such as the exploration of complex simulation data and for new forms of learning," predicts (left ) Prof. Dr. Wolfgang Broll of the Fraunhofer Institute for Applied Information Technology FIT.