Advances in haptic technology and a new approach to generating virtual reality content may help to create more realistic and immersive virtual experiences.  Where users could see and hear virtual surroundings, objects and avatars, now touch comes into play for new applications with telepresence in telemedicine, industrial design, gaming and entertainment.

“Audiovisual aspects of VR have come a long way in recent years, so adding a sense of touch is the next step,” says Andreas Schweinberger, Technische Universität München researcher. “We know that the more senses that can be used, the more interaction, the greater the sense of presence. And a stronger sense of presence means the experience is more immersive and realistic.”

Schweinberger led a team from nine universities and research institutes in developing technology to make VR objects & characters touchable.

With funding from the EU in the Immersence project, they developed innovative haptic and multi-modal interfaces, new signal processing techniques and a pioneering method to generate VR objects from real objects in real time.

That technology, developed at the Computer Vision Laboratory of Swiss project partner ETH Zürich, uses a 3D scanner and advanced modelling system to create a virtual representation of a real object, such as a cup, box or, in one experiment, a green fluffy toy frog. An object's 3D digital representation can then be transmitted to someone at a remote location, who, wearing VR goggles and touching a haptic interface, can move, prod and poke the object.

“Haptic technology is still in the early stages. For the haptic interface, we used a robotic arm a Personal HAptic iNTerface Mechanism (PHANTOM) that has one contact point."

(Right: VT'is Cybergrasp with exo-skeleton.)

"This gives the sense of touching an object, but you can’t pick it up or handle it.

"However, one of the other project partners, Universidad Politécnica de Madrid, is developing a haptic device with two contact points that should make it possible to grasp an object with a virtual hand,” Schweinberger explains.

The researchers also worked on techniques that would allow a user to feel different textures and sense the stiffness of an object, enabling differentiation between a hard box, a soft fluffy frog or even a liquid.

The Immersence researchers did not stop at human-object interaction, however. Technische Universität München also developed technology to enable human-human interaction in a virtual environment.

At the lab in Munich, they used a mobile robotic platform with two arms to serve as the dance partner for a real human dancer. By wearing VR goggles, the user would see a dancer of the opposite sex and could dance with them by holding the “hands” of the robot.

Left: iCub“baby” robot acts in cognitive scenarios, performing tasks  interacting with the environment and humans. Small (104cm tall), compact size (22kg ) and fitting within the volume of a child) and 53 degrees of freedom, with the Open Source approach, distinguishs RobotCub from other humanoid robotics projects developed worldwide.

“To program the robot we first recorded the forces, balance and movement of a real human dancer and apply these to the robot. In a VR environment, the robot could be a computer-controlled agent or avatar of another person,” says the project manager..

French partner Université d’Evry went one step further and studied how to give the sensation of two people handling an object, such as lifting a heavy box, all virtually.

“It is not as simple as one person taking the lead and another following. In reality, it is a negotiation process and the robotic interface has to be programmed for that,” says Schweinberger.

Gamers watch developments, which promise to bring a whole new dimension and realism to VR environments. But there are serious applications for haptic VR technology. Doctors could treat patients remotely. Physiotherapists use it for  training and rehabilitation. Industrial designers remotely collaborate by  “teleporting” touchable digital mock-ups of designs over the internet.

“The research will also help in the development of cognitive robots that are better able to interact with humans,” adds Schweinberger. His  team is continuing research on that aspect of the Immersence project, which received funding under the FET-proactive strand of the EU’s Sixth Framework Programme. ETH Zürich, meanwhile, is set to continue developing its virtual teleporter.

Several of the project partners are also continuing their work in the EU-funded BEAMING project where they plan to develop a virtual reality room in which several mobile robots will move around autonomously and simulated objects, table, chair or door, will be experienced by the user immersed in a virtual world.

Beaming to develop virtual transport
Beaming, is a 4 year collaborative project to develop science and technology to give people a real sense of physically being in a remote location with other people without physically travelling.  Hosted by the University of Barcelona and its project coordinator,  R+D company Starlab.

The project started in January is carried out by a consortium of 11 leading technology R+D and neuroscience groups and companies from seven countries in Europe. The budget for Beaming is over €12m

Despite advanced video conferencing, shared virtual environments, and Second Life environments it is still much more efficient to  travel physically to remote location for business, scientific or family meetings, even if at a huge environmental, energetic and opportunity cost.

Beaming will address this with a new kind of virtual transportation, where the person can be physically embodied interacting with life-sized people half a globe away. it will be achieved by shifting their means for perception into the destination, and decomposing their actions, physiological and even emotional state into a stream of data transferred across the internet.

Beaming will bring  networking, computer vision, graphics, VR, haptics, robotics and user interface technology together  transcending what is currently possible. It will be underpinned by the practical use of recent cognitive neuroscience advances in understanding the process where the brain represents our own body.

Simultaneous streams of data from the destination site to the visitor’s perceptual apparatus, and from the actions and state of the visitor to the destination site, will cohere  to form a unified virtual environment representing the destination's physical space in real-time, including beamed people. Beaming will endow this process with physicality. 


The Beaming consortium comprises:
• Starlab (Spain) - Project Coordinator
• Universitat de Barcelona (Spain)
• University College London (United Kingdom)
• Eidgenössische Technische Hochschule Zürich (Switzerland)
• Scuola Superiore di Studi Universitari e di Perfezionamento Sant’ Anna (Italy)
• Technion - Israel Institute of Technology (Israel)
• Interdisciplinary Center Herzliya (Israel)
• IBM Haifa Research Lab (Israel)
• Consorci Institut d’Investigacions Biomediques August Pi i Sunyer  (Spain)
• Aalborg Universitet (Denmark)
• Technische Universitaet Muenchen (Germany




The AlloSphere
Beaming is another take on the immersive data sets chamber experience, the AlloSphere. Essentially a house-size digital microscope powered by a supercomputer with high-resolution video projectors, this can project images across the entire inner surface and is a unique virtual reality environment at the University of California, Santa Barbara, which makes turning large data sets into immersive experiences of sight and sound viable.

Inside its three-story metal sphere researchers can interpret and interact with their data in new and intriguing ways, including watching electrons spin from inside an atom or "flying" through an MRI scan of a patient's brain as blood density levels play as music.

Housed in a 5,760m2 space in California NanoSystems Institute  building, the AlloSphere's outer chamber is a cube covered with sound-absorbing material, making it one of the largest near-anechoic world spaces. Inside are two joined hemispheres of perforated aluminum that contain a suspended bridge.

More than 500 audio elements, woofers, tweeters etc are suspended in rings just outside the hemispheres. High-resolution video can project images across the entire inner surface. The result is  far beyond other VR systems such as a Cave Automatic Virtual Environment (CAVE) or a planetarium: 360 degrees of sounds and images in a chamber large enough to hold 30 or more researchers at once.

"It's a place where you can use all of your senses" to find new patterns in data, says (left) JoAnn Kuchera-Morin, AlloSphere's director.  "You can almost say researchers are shrunk down to the size of their data, immersed at a perceptual level."

Trained as an orchestral composer and director of the school's Center for Research in Electronic Art Technology (CREATE), she designed the AlloSphere to straddle the line between art and science.

Still, she emphasizes that it is a real research instrument, not a virtual-reality environment for entertainment. The bridge is often crowded with physicists, engineers, computer scientists and artists working on projects for weeks or months at a time.

Researchers interact with their data, which can be streamed live, using 3-D glasses, special wireless controllers, and sensors embedded in the bridge's railings. (Gesture control and voice recognition are in the works.)

Presence Research: Enabling technologies
Main feature