Personal projectors for pervasive computing

12 July 2012 Projectors are a flexible medium for large, scalable, and transitory display. They’re pervasive as a presentation infrastructure, widely deployed for public viewing, and they’re increasingly replacing TV screens in the home. Beyond display, projectors can augment real-world…

12 July 2012

Projectors are a flexible medium for large, scalable, and transitory display. They’re pervasive as a presentation infrastructure, widely deployed for public viewing, and they’re increasingly replacing TV screens in the home. Beyond display, projectors can augment real-world objects with visual overlay and, in combination with camera systems, give rise to new forms of user interfaces—from interactive surfaces to “everywhere” interfaces.

Because of their size, projectors have developed as an infrastructure device, typically permanently installed or set up ad hoc in a fixed position. However, miniaturization has led to the first generation of pico projectors, which are small enough for use in a truly mobile fashion as handheld, wearable, or stand-alone devices or integrated with other personal devices. As a mobile personal device, pico projectors will be used quite differently from projectors as we know them—similar to how the use of handheld and wearable computers greatly differs from the use of PCs.

Here, we survey current research on mobile personal projectors to understand how pico projector technology can be embraced for pervasive computing. We review developments from a conceptual perspective, analyzing personal projector concepts, structuring the input and output space for interaction with projectors, and discussing sensing challenges and emerging applications.

Personal Projectors
We present an overview of current pico projector technologies and products in the related sidebar. The reasons for adopting pico projectors for mobile personal use are varied—they offer display portability with pocket-size devices, can overcome the screen limitations of personal devices, can facilitate shared viewing of content from personal devices, provide hands-free displays for mobile work, and enable new ways of interacting with physical environments.

Concepts for Devices
In response, various concepts for new or enhanced devices are emerging, which we’ve grouped into four categories (see Figure 1).

Peripheral. Portable projectors can be viewed as notebook peripheral, and pico projectors are now similarly available as a self-contained external display for personal devices. As a peripheral, they depend on a host device for application but can be flexibly used with different devices, such as handsets or digital cameras. Devices in this category include the Optoma Pico PK-101 and the Microvision SHOWWX laser pico projector (see Figure 1a).

Pico Projector Technologies and Projects

Three different technologies have emerged that enable projection with very small form-factor devices: Digital light processing (DLP), laser pico projectors, and Holographic laser projection (HLP).

DLP from Texas Instruments is based on emitting light onto a micromirror matrix that’s manipulated to control the reflection and intensity of projected pixels. It’s the most mature of the three technologies and was adopted in many of the early pico projector and projector phone products currently on the market. This includes the Samsung W9600/Galaxy Beam i8520, lG Expo, nTT DOCOMO Keitai F-04B, and Optoma pK 102.

Laser pico projectors, for instance Microvision’s SHOWWX, are based on a laser beam that’s steered across the projection surfaces to “paint the picture.” The advantages are that the image is always in focus and the process is more efficient, because images are formed by steering (as opposed to blocking) light.

A different laser-based technology, HLP, is under development by light Blue Optics. Here, the laser light is used to illuminate a micro display that’s diffracted to generate the projected image.

The first products available on the market have a brightness of up to 50 AnSI lumens; battery lifetime of up to two hours; VGA, wide VGA, and super VGA resolution; and can easily project images up to 100 inches in size. The form factor is compact—some projector phones are barely larger than comparable smartphones.

Handset-integrated. Mobile phones have become a central device for mobile information services, but their screen size and resolution severely limit the display of rich media, including Web content, maps, photos, and video. This is a strong reason to integrate pico projectors as a built-in component in handsets. The first products in this space include the Epoq EGP-PP01, NTT DOCOMO’s Fujitsu F-04B, and Samsung’s W9600 (see Figure 1b). Projectors are also becoming integrated with other handheld devices—for example, the COOLPIX S1100pj digital camera has a built-in projector.

Wearable. Pico projectors also provide new opportunities for supporting mobile activity as part of wearable system solutions. Mounting projectors on the body lets users keep their hands free for other tasks, and researchers have demonstrated projectors worn on the wrist, shoulder,head, and chest.

Stand-alone. Researchers are also exploring how to use personal projectors as “first class” (stand-alone) devices. Here, the projection isn’t an add-on—it’s central to the interaction, letting users interact with
their environment in novel ways. For example, users might “shine” a projector in a torch-like fashion at objects in the environment to project related information and explore the environment with a projected magic lens.


How we conceive personal projectors—whether as handset-integrated, wearable, or stand-alone device—raises distinct questions and challenges. Handsets, for example, are associated with a private viewing experience using the built-in screen, in contrast to the public-facing display that a built-in projector provides. In initial products, output is mirrored from the screen to the projector, which doesn’t reflect their different properties and potential to complement each other.

Wearable projectors, on the other hand, raise questions of placement on the body and of appropriation for use in mobile tasks, including in difficult environments. How a projector is held, worn, or integrated with another device affects the interaction design and application in terms of where users can display the projections, how they can control them (by hand, body, or head movement), and how they can interact with projectors and projected content.

Interaction with Personal Projectors

Projectors as we know them are predominantly used for passive display and are controlled indirectly via host computers. Direct interaction with projectors or projected content has received limited attention—though researchers have explored interaction at a distance and steerable interfaces. Personal projectors, however, open up a novel interaction space because of their mobile nature and the different ways in which they can be carried, worn, and manipulated.

Input and Control
In our research, we identified four conceptually different approaches to interacting with a personal projector system (see Figure 2).
Input on the projector. This approach leverages common interfaces such as buttons (for example, the Epoq EGPPP01), a scroll wheel, or soft controls on a touchscreen (for example, the Samsung W9600). Some research work has emphasized interface simplicity to let users provide input while still focusing on the projection by, for example, operating a two-button interface with their thumb.

Projector phones provide richer interfaces with touchscreens or keypad-screen combinations that can be leveraged for projector interaction. A problem with these devices is that users typically must switch their attention between projection (output) and handset (input), resulting in a higher task-completion time, error rate, and task load.
Movement of the projector. If the projector can track its movement, then it can use that data as input. This approach offers an implicit control for selecting the projection area. For example, some projector systems reveal content depending on the projector’s location and orientation. Also, handheld projectors could use movement for gestural input. For example, by tilting the projector up or down and left or right, the user could pan an image, map, or webpage. Such gestures are intuitive and let the user focus on the projected image as opposed to the input device. However, implementation can be challenging for use beyond environments instrumented with a motion tracker.

Direct interaction with the projection.
Researchers have extensively explored direct interaction with projected content using steerable projectors as well as wearable camera-projector systems. Toshikazu Karitsuka and Kosuke Sato were first to present a
shoulder-worn system that included an IR light module to illuminate a notepad fitted with retroreflective markers. This enabled camera tracking of the pad’s relative position and orientation for distortion-free projection. In addition, the system could track interaction on the pad by having the user wear IR light-emitting finger caps, which let the user select controls and draw on the projection. The research group also explored projection onto the user’s palm and control with finger gestures.
The Brainy Hand system takes this concept further by envisioning integration of the camera-projector system into an earpiece for interactive projection onto the user’s hand. The Wear Ur World (better known as
SixthSense) system expands on interaction with projections by supporting in-the-air hand gestures. Users wear the system either as a pendant or head-mounted device that employs color markers to track fingers. Researchers have proposed using such devices in scenarios ranging from augmented-reality (AR) projection onto user-held objects to projection onto walls in the user’s environment.

Researchers have also considered a variety of direct-interaction methods in a prototype designed for military use, including projecting images onto the floor and using a laser pointer, telescopic stick with a light-emitting tip, or the user’s boots (with a reflective marker) for pointer input.

Manipulation of the projection surface. Manipulating location and orientation of a handheld projection surface provides implicit input for prewarping projections in systems such as Karitsuka and Sato’s. However, surface manipulation could be used as a more explicit control—for example, a handheld surface might be bent inward or outward to zoom in or out of a projected image and tilted for panning.

Output and Presentation
In terms of output, we observed four distinct ways of using personal projectors for presentation (see Figure 4).

Anywhere display. The simplest of these four presentation concepts, the “anywhere display” doesn’t require projection tracking with respect to a virtual space, real objects, or other projections. However, it requires adapting to surfaces selected for presentation—in particular, users must align the projector to achieve distortion-free projection, unless the projector system can automatically prewarp projected images.
Some application domains, such as military operations, need displays “in the wild,” where the only available surfaces might be curved or textured, so the content might have to be adapted accordingly. The anywhere display could also use two pico projectors in tandem—one as output and the other for projecting a virtual keyboard for input.

Spotlight interaction. This interactive presentation style involves moving the projector to reveal information that forms part of a larger virtual space. It lets users explore large, high-resolution virtual information layers, perhaps mapped onto a wall. For example, the user could move the projector closer to the projection to semantically zoom in on an information space. Users could make a selection in such a space by simulating a mouse cursor with a crosshair in the center of the projection and pressing a button on the projector.
Augmented reality. Researchers have studied a host of scenarios—from mobile tourism to maintenance work—that could benefit from augmenting real-world objects with mobile and wearable systems. The advantage of handheld and wearable projectors over previously used systems, such as head-mounted displays, is that they can project an AR display directly onto the objects of interest. The
projector system first must register the object, but to address this, objects can broadcast their location and shape, or the system can rely on external location and information systems or can itself visually identify the
object. Handheld AR projection can help control objects—for example, by projecting tracks for a robot to follow—and can offer mixed-reality experiences—for example, by projecting content that augments a user’s hand

Multiprojector interactions.A group of users can use their personal devices in a collaborative manner to produce multiprojector interactions. The View & Share system lets a group of users share personal projections, so that users can interact with the projection using each other’s devices.

You can also use multiple projections side by side to create a larger shared display or to create a “focus and context” display in which one projection is moved like a spotlight over the other to show additional detail. For example, the “context” projection might show a map of a large area, and the “focus” projection moving over the map could highlight details as if viewed through a magnifying glass.

Sensing and Tracking Challenges

All but the most basic use of personal projectors (that is, as a display manually aligned to a surface) requires tracking system components and their relative spatial arrangement to underpin projector control, interaction, and output.

Tracking the projector. This type of tracking is particularly relevant when the projector is handheld or wristworn as opposed to fixed rigidly on the body, because this provides freedom of movement for projection onto different objects around the user, spotlight scanning of virtual or physical spaces, and gestural input. Accurately tracking the projector position and orientation requires extrinsic sensing—for example, with a camera infrastructure that tracks visual markers on the device, infrared reflective markers, or active light-emitting markers. A projector can use intrinsic sensing to track the device’s orientation and relative movement using accelerometers (which can be used alone or combined with a gyroscope or magnetic compass).

Tracking projected images. This approach helps detect and remove distortions. Integrating a camera with the projector unit helps track reference points to determine the projection plane’s relative orientation with respect to the projector. Tracking of the projected content can also support multiprojector interaction—for example, to associate different projections with different users and combine projections.

Tracking the projection surface. Most personal projector systems are independent of the projection surface; however, researchers have proposed systems that include a surface that the user can move and manipulate. In these cases, surface tracking must be incorporated into the system. A common approach is to place markers on the surface, such as passive markers (visible or infrared) that the projector-mounted camera
can track, or photosensing tags that let surfaces detect their arrangement with respect to a projector. Alternatively, a system could determine the surface position and orientation by tracking and analyzing the projected image. There’s also a larger body of work on analyzing the projected image to derive other surface properties, such as texture, to adapt the projection.

Tracking users’ hands.Systems that support input on the projection, using either direct touch on the surface or gestures at a distance, must track the input device—typically the user’s hands and fingers but, in some cases, sticks, pens, or the user’s feet. Tracking is generally performed with a projector-mounted camera and supported by markers (either passive or active infrared light-emitting markers). There are approaches that, instead of markers, use image processing to separate out the user’s hand in a scene and track pointing and potentially other gestures.

Application of Personal Projectors

Xiang Cao identified three main application areas: personal information processing, interaction with the physical
world, and interpersonal information exchange. Others have suggested projected desktop applications, projected
AR, and the selection of physical regions of interest as application classes. From our own survey and research, we
predict the following application areas will be particularly significant.

Games and Entertainment

Personal projectors facilitate new mobile interaction experiences with a strong potential for playful use, especially by teenagers and young adults. Youth culture has played a major role in the adoption of mobile technologies, such as camera phones, and we expect to see comparable cultural impacts from integrating projectors into handsets (and application stores provide a ready vehicle for creative development).

Among the features that render projector phones intriguing for game development is the availability of a private display alongside a public-facing projection. Research has also presented concepts for playful projector interaction with real-world props and user drawings and for games that build on interaction with multiple projectors. For example, a collaborative puzzle or treasure hunt that lets players uncover certain targets only when projections overlap.

Augmented Reality

Personal projectors have application potential for AR interaction with the physical world, although this requires registering projected content with realworld entities and thus more elaborate tracking technology. Radio frequency identity and geometry (RFIG) lamps are an early exploration of how physical objects can be detected, tracked, and augmented with mobile projectors. The general concept is to project a visual information overlay, such as task guidance in maintenance scenarios or travel directions on paper maps.

Typical scenarios discussed in the literature include logistics in warehouses, but researchers have also proposed using handheld AR projection for control applications, such as robot navigation, and for mixed-reality game experiences.

Data Visualization and Manipulation

Mobile projectors can display visual data anywhere and facilitate new ways of exploring large data visualizations using spatial metaphors, as demonstrated with spotlight interaction. Researchers have demonstrated applications ranging from standard desktops that use the projector like a mouse pointer to select visualizations of stock information for further analysis. Researchers have studied a range of input methods for interacting with visualizations, but gestural input on the projection or directed toward the projection seems most intuitive.

Group Collaboration

Personal projectors let users share information from a personal device without requiring a display infrastructure in the environment. This alone is significant for supporting mobile collaborative practices, but mobile projections might also be appropriated as ad hoc single-display groupware on which multiple users can interact simultaneously (for example, using their mobile phones for input).
A further dimension for social interaction opens up with collaborative use of multiple personal projectors. The applications explored by Kazuhiro Hosoi and his colleagues and Xiang Cao and his colleagues exemplify the potential of combining projections for practical aspects (larger display, document exchange) and collaborative problem solving (such as collaborative visual analysis).

A User-Centered Research Agenda

Prior to widespread adoption, personal projectors must shrink in size and become brighter and more energy efficient. Likewise, significant technical advances are required in sensing and tracking technology for mobile projection “in the wild,” such as methods for projectors to determine the geometry of their environment and to visually register real-world objects. However, in addition to advancing technology, we need a user-centered research agenda to better understand interaction with projectors and the usability and social factors of their application.

Understanding Interaction and Usability
Our survey shows that a wide range of interaction concepts and techniques are being considered for mobile personal
projectors. However, there has only been limited research into understanding the interaction and usability aspects
of personal projectors. Researchers have performed empirical studies with experimental projector phone configurations, providing insight into the task performance and qualitative aspects of displaying interfaces on a handset versus projector and into user preferences for sharing media. Researchers have also conducted a formative study on the usability of wearable projectors in difficult environments. As the technology matures, further work will be required to gain a principled understanding of the usability of different interaction concepts and modalities and their suitability for different applications and contexts.

Personal projector interfaces have thus far been explored with bespoke developments, or interface concepts as known from the desktop. We propose that abstractions and widgets tailored to mobile projection will be required for more effective interface and application development. This might include notions such as “floating window” and “flashlight.”

Social Acceptance, Visual Pollution, and Privacy
Overusing mobile projectors could ultimately overwhelm users by visually cluttering their environments. Research must establish practices that avoid the distracting effects of such projections and the problem of invading other people’s personal spaces. We have conducted a formative study, which indicated that personal projections in public spaces—such as pubs, clubs, or train stations—are socially accepted, but this might change as the technology becomes more commonplace and intrusive to bystanders.

Personal projectors also raise new privacy challenges, as they extend inherently private devices in a public-facing
manner. Mobile phones, in particular, are highly personal and rarely sharedwith others. We’ll need to develop guidelines and interface frameworks that ensure a privacy-aware design of interfaces and information displays across the private built-in screen and public-facing projector, including easyto-use safeguards against unintended projection of sensitive information.

Related work on the symbiotic use of private and public displays has proposed blurring sensitive information on large display while using the private display to explore the information in more detail. Related concerns have also been explored in work on multiprojector collaborative use, considering visual indication of ownership, access, and visibility of content.

Pico projectors are on the brink of wider adoption, particularly in high-end mobile handsets. The anywhere display capability they bring to small devices has compelling use cases in everyday life, for viewing and sharing media at largerthan-pocket sizes. Component costs, limited quality (brightness), and energy demands still present barriers for
widespread use, but these are increasingly being overcome by technological advances. With improvements in sensing and tracking, personal projectors will be able to support rich and innovative forms of interaction. Ultimately, they could transform how we display and consume information, regardless of where we are. As the technology matures, it will be important to advance our understanding of the human and social factors of personal projectors.

IEEE Computer Society