Method and system for a user-following interface

• US 20070013716A1
• Filed: 09/19/2006
• Published: 01/18/2007
• Est. Priority Date: 08/23/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kjeldsen, Frederik, Pingali, Gopal, Levas, Anthony, Pinhanez, Claudio
• Granted Patent: US 7,530,019 B2
• Time in Patent Office: Days
• US Class Current: 345/594
• CPC Class Codes: G03B 21/28 Reflectors in projection beam in illumination beam G03B21/2066
• CPC Class Codes: H04N 5/74 Projection arrangements for image reproduction, e.g. using eidophor
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

Techniques are disclosed for projecting an image onto a surface suitable for interaction with a user while avoiding user occlusion, and while correcting for distortion due to oblique projection. The displayed image moves to a suitable surface at a suitable size and orientation as a user moves around an environment, resulting in a user-following interface. Surfaces are selected in which the projected interface is not occluded by the user or other objects in the environment. Displayed images may be interactive, and moved into an interaction area on a suitable surface that is convenient for the user. The interaction area may or may not coincide with the display area. Adaptation of the projected interface is allowed so that the content of the display and the style of interaction widgets are modified based on distance from the user and orientation of the user with respect to a projected interface.

Claims

1. A user-following interface system comprising:

one or more projectors adapted to project one or more images onto one or more selected surfaces in a real environment, wherein the one or more selected surfaces are selected from a plurality of surfaces;

a user tracking module adapted to track positions of one or more users in the environment; and

a controller coupled to the one or more projectors and the user tracking module, the controller adapted to select the one or more surfaces based on the positions of the one or more users in the real environment, and to provide information to the one or more projectors suitable for allowing the one or more projectors to project the one or more images onto the one or more selected surfaces.

2. The user-following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, wherein the user tracking module is further adapted to track orientations of the one or more users by using information from the one or more tracking devices, and wherein the controller is further adapted to select the one or more surfaces based on the orientations of the one or more users.

3. The user-following interface system of claim 1, further comprising a model of the real environment, wherein the controller is coupled to the model and is adapted to access, from the model, information about surfaces and objects in the real environment.

4. The user-following interface system of claim 1, wherein the controller is further adapted to identify occlusion of a projected image.

5. The user-following interface system of claim 4, wherein the controller is further adapted to move the projected image to a surface that is not occluded.

6. The user-following interface system of claim 1, wherein the controller is further adapted to determine warping parameters to prewarp an image to ensure it appears substantially undistorted to a user when the image is displayed on a selected surface.

7. The user-following interface system of claim 1, wherein the controller is further adapted to determine an orientation of a selected image when displayed on a selected surface, and to provide information to the one or more projectors so that the selected image, when displayed on a selective surface, looks upright and approximately orthogonal to a selected user.

8. The user-following interface system of claim 1, wherein the controller is further adapted to determine a size of a projected image based on an available area on a display area and on a distance of the display area from a selected user.

9. The user following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, and wherein at least one of the tracking devices comprises one or more cameras that observe one or more of the users in the real environment.

10. The user-following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, and wherein at least one of the tracking devices comprises an electromagnetic device or an active badge worn by at least one user.

11. The user-following interface system of claim 9, wherein the user tracking module is coupled to one or more tracking devices, and wherein user tracking module is further adapted to analyze camera images from the one or more cameras for one or more of motion, shape, and color to determine presence of people in each camera image.

12. The user-following interface system of claim 1, wherein the user tracking module is coupled to one or more tracking devices, wherein the at least one tracking device is further adapted, when analyzing the camera images, to:

find a difference between a current image from a selected camera and a previous image from the selected camera to creating a difference image;

threshold and filter the difference image to remove noise and to yield foreground regions;

select foreground regions with sufficient human skin color;

search the selected foreground regions for a shape of a human head; and

determining a position of the head in the difference image.

13. The user-following interface system of claim 9, wherein the user tracking module is further adapted to combine a detected location of an individual in multiple camera images and calibration parameters of the cameras to yield a three-dimensional location of the individual in the real environment.

14. The user-following interface system of claim 12, wherein the at least one tracking device is further adapted to analyze a detected head shape in a camera image to determine an orientation of the detected head shape, thereby determining which way a selected user is facing.

15. The user-following interface system of claim 1, further comprising a sensing mechanism adapted to detect user interactions with a selected image projected onto one of the selected surfaces.

16. The user-following interface system of claim 15, wherein the sensing mechanism comprises a camera adapted to observe the selected image.

17. The user-following interface system of claim 15, wherein the sensing mechanism is further adapted to detect user hand gesture interactions with the selected image, and wherein the sensing mechanism is further adapted to detect a variety of interaction modes resulting from the user hand gesture interactions.

18. The user-following interface system of claim 15, wherein the sensing mechanism is further adapted to detect user interaction when the user uses a laser pointing device.

19. The user-following interface system of claim 15, wherein an interaction area, comprising a first image, is independent of a display area, comprising a second image, wherein the interaction area and display area are completely separate, completely overlapping, or partially overlapping.

20. The user-following interface system of claim 15, wherein a configuration is defined by a user or application in application space and where the controller is further adapted to automatically determine a view by a camera of the configuration for any surface that a selected image is projected on, so that no separate configurations for each camera view need to be defined.

21. The user-following interface system of claim 15, wherein the controller is further adapted to select an interaction area for an image by:

mapping position and orientation of the user into a three-dimensional environment model;

determining a volume in the three-dimensional model that is reachable by the user;

determining a set of surfaces available for interaction in the reachable volume;

determining available interaction areas on each surface;

determining a distance from an appendage of the user to a center of an available interaction area;

eliminating interaction areas that are occluded by the user to create remaining interaction areas; and

selecting one of the remaining interaction areas, within the reachable volume, that is closest to the user while being approximately orthogonal to a viewing direction of the user.

22. The user-following interface system of claim 15, wherein the controller determines warping parameters that map an interaction area to a camera image of the interaction area.

23. The user-following interface system of claim 1, further comprising an interface adapter adapted to change content of the one or more images based on user and surface parameters.

24. The user-following interface system of claim 23, wherein the interface adapter is further adapted to determine an effective resolution of the one or more images with respect to a user.

25. The user-following interface system of claim 23, wherein the interface adapter is further adapted to change content of the one or more images to be visible and readable by the user.

26. The user-following interface system of claim 23, wherein the interface adapter is further adapted to modify one or more of size or position of interaction widgets in the one more images based on user position and orientation parameters.

27. The user-following interface system of claim 23, wherein the interface adapter is further adapted to modify a mode of interaction based on user position and orientation parameters.

28. The user-following interface system of claim 23, wherein the interface adapter is further adapted to modify parameters of one or more interaction sensing mechanisms based on user position and orientation parameters.

29. A method for creating a using and user-following interface, the method comprising the steps of:

tracking positions of one or more users in the real environment;

selecting one or more surfaces from a plurality of surface based on the position of the one or more users in the real environment; and

projecting one or more images onto the one or more selected surfaces, wherein the interface comprises the one or more images.

30. An article of manufacture for creating and using a user-following interface, comprising:

a computer-readable medium having computer-readable code means embodied thereon, the computer-readable program code means comprising

a step to track positions of one or more users in a real environment;

a step to select one or more surfaces from a plurality of surface based on the position of the one or more users in the real environment; and

a step to project one or more images onto the one or more selected surfaces, wherein the interface comprises the one or more images.

31. An apparatus for creating a using and user-following interface, comprising:

at least one processor operable to;

track positions of one or more users in a real environment;

select one or more surfaces from a plurality of surface based on the position of the one or more users in the real environment; and

project one or more images onto the one or more selected surfaces, wherein the interface comprises the one or more images.