Method and system for a user-following interface

• US 8,589,796 B2
• Filed: 04/03/2008
• Issued: 11/19/2013
• Est. Priority Date: 08/23/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Pingali, Gopal Sarma, Levas, Anthony, Pinhanez, Claudio Santos, Moesgaard K{umlaut over (j)}eldsen, Frederik Carl
• Primary Examiner(s): Vu, Kieu
• Assistant Examiner(s): BASOM, BLAINE T
• Application Number:
  US12/062,150
• Publication Number:
  US 20080218641A1
• Time in Patent Office: 2,056 Days
• Field of Search: 715/761, 715/730, 715/850, 715/863, 715/731, 715/732, 715/848, 353/71
• US Class Current: 715/730
• CPC Class Codes: G03B 21/28 Reflectors in projection beam in illumination beam G03B21/2066
• CPC Class Codes: H04N 5/74 Projection arrangements for image reproduction, e.g. using eidophor
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

Techniques are disclosed for projecting an image onto a surface suitable for interaction with a user while avoiding user occlusion, and while correcting for distortion due to oblique projection. The displayed image moves to a suitable surface at a suitable size and orientation as a user moves around an environment, resulting in a user-following interface. Surfaces are selected in which the projected interface is not occluded by the user or other objects in the environment. Displayed images may be interactive, and moved into an interaction area on a suitable surface that is convenient for the user. The interaction area may or may not coincide with the display area. Adaptation of the projected interface is allowed so that the content of the display and the style of interaction widgets are modified based on distance from the user and orientation of the user with respect to a projected interface.

Claims

1. A user-following interface system comprising:

one or more projectors adapted to project one or more images onto one or more selected surfaces in a real environment, wherein the one or more selected surfaces are selected from a plurality of surfaces;

a user tracking module adapted to track positions and orientations of one or more users in the real environment; and

a controller coupled to the one or more projectors and the user tracking module, the controller adapted to select the one or more selected surfaces and to dynamically determine one or more configuration parameters of the one or more projectors based on the positions and orientations of the one or more users in the real environment, and to provide information to the one or more projectors suitable for allowing the one or more projectors to project the one or more images onto the one or more selected surfaces, wherein each selected surface comprises a display area and wherein the controller is further adapted to select a display area for an image by;

mapping the position and orientation of a selected user into a three-dimensional environment model;

determining a volume, in the three-dimensional model, that is viewable by the selected user;

determining a set of display areas on a set of displayable surfaces in the viewable volume; and

selecting one of the display areas;

wherein the controller is further adapted to identify occlusion of a projected image, and to move the projected image to a surface that is not occluded.

2. The user-following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, wherein the user tracking module is further adapted to track orientations of the one or more users by using information from the one or more tracking devices.

3. The user-following interface system of claim 1, further comprising a model of the real environment, wherein the controller is coupled to the model and is adapted to access, from the model, information about surfaces and objects in the real environment.

4. The user-following interface system of claim 1, wherein the controller is further adapted to determine warping parameters to prewarp an image to ensure it appears substantially undistorted to a user when the image is displayed on a selected surface.

5. The user-following interface system of claim 1, wherein the controller is further adapted to determine an orientation of a selected image when displayed on a selected surface, and to provide information to the one or more projectors so that the selected image, when displayed on a selective surface, looks upright and approximately orthogonal to a selected user.

6. The user-following interface system of claim 1, wherein the controller is further adapted to determine a size of a projected image based on an available area on a display area and on a distance of the display area from a selected user.

7. The user following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, and wherein at least one of the tracking devices comprises one or more cameras that observe one or more of the users in the real environment.

8. The user-following interface system of claim 7, wherein the user tracking module is coupled to one or more tracking devices, and wherein user tracking module is further adapted to analyze camera images from the one or more cameras for one or more of motion, shape, and color to determine presence of people in each camera image.

9. The user-following interface system of claim 7, wherein the user tracking module is further adapted to combine a detected location of an individual in multiple camera images and calibration parameters of the cameras to yield a three-dimensional location of the individual in the real environment.

10. The user-following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, and wherein at least one of the tracking devices comprises an electromagnetic device or an active badge worn by at least one user.

11. The user-following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, wherein the at least one tracking device is further adapted, when analyzing the camera images, to:

find a difference between a current image from a selected camera and a previous image from the selected camera to create a difference image;

threshold and filter the difference image to remove noise and to yield foreground regions;

select foreground regions with sufficient human skin color;

search the selected foreground regions for a shape of a human head; and

determining a position of the head in the difference image.

12. The user-following interface system of claim 11, wherein the at least one tracking device is further adapted to analyze a detected head shape in a camera image to determine an orientation of the detected head shape, thereby determining which way a selected user is facing.

13. The user-following interface system of claim 1, further comprising a sensing mechanism adapted to detect user interactions with a selected image projected onto one of the selected surfaces.

14. The user-following interface system of claim 13, wherein the sensing mechanism is further adapted to detect user hand gesture interactions with the selected image, and wherein the sensing mechanism is further adapted to detect a variety of interaction modes resulting from the user hand gesture interactions.

15. The user-following interface system of claim 1, further comprising an interface adapter adapted to change content of the one or more images based on user and surface parameters.

16. A method for creating a user-following interface, the method comprising the steps of:

tracking positions and orientations of one or more users in a real environment;

mapping the position and orientation of a selected user into a three-dimensional environment model;

determining a volume, in the three-dimensional model, that is viewable by the selected user;

determining a set of display areas on a set of displayable surfaces in the viewable volume;

selecting one or more selected surfaces from said set of display areas and dynamically determining one or more configuration parameters of one or more projectors based on the positions and orientations of the one or more users in the real environment;

projecting one or more images onto the one or more selected surfaces, wherein the interface comprises the one or more images;

identifying occlusion of a projected image; and

moving the projected image to a surface that is not occluded.

17. An article of manufacture for creating and using a user-following interface, comprising:

a non-transitory computer-readable recordable storage medium having computer-readable code means embodied thereon, the computer-readable program code means comprising

a step to track positions and orientations of one or more users in a real environment;

a step to map the position and orientation of a selected user into a three-dimensional environment model;

a step to determine a volume, in the three-dimensional model, that is viewable by the selected user;

a step to determine a set of display areas on a set of displayable surfaces in the viewable volume;

a step to select one or more selected surfaces from said set of display areas and to dynamically determine one or more configuration parameters of one or more projectors based on the positions and orientations of the one or more users in the real environment;

a step to project one or more images onto the one or more selected surfaces, wherein the interface comprises the one or more images;

a step to identify occlusion of a projected image; and

a step to move the projected image to a surface that is not occluded.

18. An apparatus for creating a user-following interface, comprising:

at least one processor operable to;

track positions and orientations of one or more users in a real environment;

map the position and orientation of a selected user into a three-dimensional environment model;

determine a volume, in the three-dimensional model, that is viewable by the selected user;

determine a set of display areas on a set of displayable surfaces in the viewable volume;

select one or more selected surfaces from said set of display areas and to dynamically determine one or more configuration parameters of one or more projectors based on the positions and orientations of the one or more users in the real environment;

project one or more images onto the one or more selected surfaces, wherein the interface comprises the one or more images;

identify occlusion of a projected image; and

move the projected image to a surface that is not occluded.