The World Is Your Playground

Microsoft is developing PlayAnywhere, an interactive projection system that is capable of displaying images on everyday surfaces and letting users manipulate the projected objects with their hands, by rotating or resizing them. The device, which the company says demonstrates the latest improvements in projection technology, senses users’ movements as they touch and drag the projected items, and reacts immediately by re-projecting the images.

PlayAnywhere was designed using advanced image-processing techniques. According to Microsoft, this compact product addresses installation, calibration and portability issues that are typical in most vision-based table systems. PlayAnywhere’s creators say they aimed to create a system capable of sensing a variety of objects and displaying animated graphics over large display surfaces; a system specifically designed for consumer applications where distribution, installation, mounting and calibration considerations are paramount.

A number of different techniques for interactive projection have been developed over the past few years. A popular approach is to mount a camera and projector high on a shelf or on the ceiling. This configuration allows users to project images on broad surfaces. On the down side, the images are often distorted when users unwittingly physically block the projector's beam.

Another approach is to place the projector and camera behind a diffusion screen. Microsoft says that while this eliminates occlusion problems, it puts limitations on image resolution. A number of systems use printed 2-D visual codes to track object movement and embed sensor electronics onto the surface itself, resulting in very fast and precise detection of touch, compared with vision-based approaches. The drawback of this method is that it lacks the flexibility to sense other objects.

Unlike many other projection systems, PlayAnywhere’s projector and camera are placed on the side of a table, and the device can sense and project images onto the target area from a relatively short distance.

Andrew Wilson, one of the researchers on the project, says the system demonstrates a number of important sensing capabilities that exploit the flexibility of computer-vision techniques, such as a novel touch-detection algorithm based on the observation of shadows; and an optical-flow algorithm for the manipulation of on-screen objects, which does not rely on fragile tracking algorithms. The developers’ main desire was to create a user-friendly device. “We believe that portability, ease of installation and ability to utilize any available surface without calibration are all features required for mainstream consumer acceptance,” Wilson says.

The Ins and Outs

PlayAnywhere consists of primarily of three components: a projector, a camera and an infrared illuminant. The three components are all constructed as a single piece, which also includes the required computing resources, such as CPU and storage. The NEC WT600 projector stands on a short pedestal, which is placed directly on the targeted surface, and uses four aspheric mirrors to project a visibly correct rectangular image from oblique angles at short distances.

In order to project a 40-inch diagonal image, the WT600 requires a minimum distance of 2.5 inches between its leading face and the projection surface; 100-inch diagonal images can be obtained from a distance of 26 inches. PlayAnywhere developers say that the oblique projection technique minimizes occlusion effects, while the unique structure ensures the system’s stability. Multiple cameras and the infrared illuminant are mounted on the projector itself, in order to preserve the calibration of the vision system to the display, regardless of where PlayAnywhere is placed. Because the height above the surface is constant, there are no problems related to adjusting the focal length of the camera and the projector when the unit is moved.

A very wide angle micro lens (2.9mm focal length), which is used to capture the entire projected surface, imparts significant barrel distortion on the input image, while the oblique position of the camera imparts a projective distortion or foreshortening. In order to preserve the correct geometric parameters of the projected images, PlayAnywhere eliminates both distortions using an image-rectification process, so that each projected object will appear identical to its source image in terms of shape — a rectangular object on the surface will be projected as a rectangular object in the image at the same (scaled) coordinates. One of the weaker points of the system, however, remains the sloped view of the camera, which causes objects further away from the unit to appear at a lower resolution.

PlayAnywhere does not require special instrumentation of the surface on which it operates. The system can be used on any available flat space because of a novel approach to touch detection developed by Microsoft engineers. The system is based on a technique that exploits the change in appearance of shadows as objects approach the surface. For example, it can detect whether a finger is touching or hovering above the surface by analyzing the shape of its shadow. As it approaches the surface, the image of the finger and its shadow come closer to each other, until the finger completely obscures the shadow of its tip, indicating it has touched the tabletop.

The current prototype is able to track only one finger per hand, but the scientists say that more sophisticated finger-shape analysis can

How It Works

PlayAnywhere implements a new format of visual codes that can be used to distinguish among various game pieces, printed pages, media containers, knobs and other objects on the tabletop having similar shapes but different application semantics. The system also allows easy manipulation of projected documents. Users can move and rotate virtual pages as the system tracks position changes in real time.

Thanks to a special page-tracking algorithm based on optical-flow computations, it is also possible to continuously track multiple paper pages of known dimensions, while preserving the individual images projected on each page. The user can also resize various objects shown on the tabletop, such as photos or videos, by spreading or contracting a single hand, or by moving two hands closer or further apart.

The system can instantly recognize plain sheets of paper and project images and video strictly within their dimensions, allowing users to rotate and move the sheets in any direction without interrupting the continuous and simultaneous data stream.

A map projected onto the surface can be resized, and users can zoom in and out using specific motions. An image of a bouncing ball can be blocked by the user's hand, making it bounce in the opposite direction. Microsoft envisions an all-digital future office environment in which your desk will be cluttered, not with piles of papers, but with the projected images of those papers.

PlayAnywhere is based entirely on computer-vision techniques, which provide greater flexibility to the projection system. This approach has its disadvantages. For example, its high computational cost could become a strong concern when there is a need for higher frame rates. Wilson says that the vision-based system makes sense only if the flexibility it provides is used, and that this depends on the envisioned applications.

“If we intend to emulate the single cursor in Windows on a table, the computer-vision approach is probably unnecessarily complex and inferior in many respects,” he says. “However, as demonstrated by PlayAnywhere and other systems, computer vision–based systems can support many scenarios that go beyond the usual GUI, including the merging of paper and virtual documents, gaming, collaborative and other multitouch applications, image-based artistic applications, and a variety of scenarios related to ubiquitous computing and tangible interfaces.”

Wilson says he believes that the advancements being made today in sensing and display technology will boost the development of home and office interactive-display applications. Microsoft envisions such systems being used as virtual desks that will conveniently project needed documents, or even as virtual playgrounds.

“Imagine a child pulling such a device out of the closet and placing it on a table or the floor of their room to transform the nearby surface into an active play space,” Wilson says.