Apparatus and method for interacting with a subject in an environment

• US 20060158616A1
• Filed: 01/06/2006
• Published: 07/20/2006
• Est. Priority Date: 01/15/2005
• Status: Abandoned Application

• Current Assignee: Lenovo Singapore Pte Limited (Lenovo Group Ltd.)
• Original Assignee: International Business Machines Corporation
• Inventors: Pingali, Gopal Sarma, Borger, Sergio, Pinhanez, Claudio Santos, Kjeldsen, Frederik C.M.
• Application Number:
  US11/327,570
• Publication Number:
  US 20060158616A1
• Time in Patent Office: Days
• US Class Current: 353/69
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/302 specially adapted for receiving control signals not targeted to a display device or game input means, e.g. vibrating driver's seat, scent dispenser
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: F16M 11/10 around a horizontal axis F16M11/12 takes precedence
• CPC Class Codes: F16M 11/2014 around a vertical axis F16M11/2035 takes precedence
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 21/14 Details
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: H04N 21/422 : Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/4402 : involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 7/163 : by receiver means only

Abstract

Techniques are provided for interacting with a subject in an environment are provided. In one aspect, an inventive apparatus includes a display device configured to display images on a number of surfaces in the environment, at least one sensing device, and at least one display controller. The at least one sensing device is configured to sense interactions between the subject and the environment. At least some of the interactions can be interactions dispersed through the environment and remote from the displayed image. The display controller can be coupled to the display device and the sensing device, and can be configured to modify image display by the display device, in response to one or more of the interactions that are sensed.

Claims

1. An apparatus for interacting with a subject in an environment, said apparatus comprising:

at least one display device configured to display images on a plurality of surfaces in the environment;

at least one sensing device configured to sense interactions between the subject and the environment, at least some of said interactions comprising interactions that are dispersed through the environment and remote from said images displayed by said display device; and

at least one display controller coupled to said display device and said sensing device, said at least one display controller being configured to modify an image display of said display device responsive to at least one of said interactions sensed by said sensing device.

2. The apparatus of claim 1, wherein said modification of said image display of said display device comprises modification of at least one of image location, image orientation, and image content.

3. The apparatus of claim 1, further comprising at least one sensor processor coupled to said sensing device and said display controller, said sensor processor being configured to receive a data stream from said sensing device and to output a set of object parameters O and a set of interaction parameters K to said display controller, said set of object parameters including at least one of:

a number of objects;

a location of each of the objects; and

a label of each of the objects;

said set of interaction parameters including at least one of;

a number of interactions detected;

a location of each of said interactions; and

a type of each of said interactions.

4. The apparatus of claim 1, wherein at least some of said images include interactive items, and wherein said display controller is configured to activate a given one of said interactive items when an object is proximate thereto.

5. The apparatus of claim 4, wherein said display controller is configured to activate said given one of said interactive items when the object proximate thereto comprises at least one of a body of the subject, a hand of the subject, a finger of the subject, an obstruction manipulated by the subject, and a laser-pointer-illuminated region that is illuminated by action of the subject.

6. The apparatus of claim 1, wherein at least some of said interactions that said sensing device is configured to sense comprise at least one of:

proximity of an object to one of the surfaces; and

performing of a predefined motion by the subject proximate to one of the surfaces.

7. The apparatus of claim 6, wherein said predefined motion is at least one of touching and pointing.

8. An apparatus for interacting with a subject in an environment, said apparatus comprising:

at least one projector configured to project images on a plurality of surfaces in the environment;

at least one physical display located in the environment and configured to display images to the subject;

at least one sensing device configured to sense interactions between the subject and the environment; and

at least one display controller coupled to said projector, said physical display, and said sensing device, said at least one display controller being configured to modify an image display of said projector and said physical display responsive to at least one of said interactions sensed by said sensing device.

9. The apparatus of claim 8, wherein said display controller is adapted to distort an undistorted image into a distorted image for projection by said projector, further comprising a projector mounting to which said projector is mounted, said projector mounting being configured to perform at least one of rotating and translating said projector to project said distorted image on one of the plurality of surfaces, said distorted image being distorted so as to appear in substantially undistorted form on the one of the plurality of surfaces.

10. The apparatus of claim 9, wherein said display controller includes a video adapter and is configured to set parameters of a correction surface in said video adapter and place said undistorted image as a texture on said correction surface.

11. The apparatus of claim 10, wherein said projector is configured for both rotational and translational movement with respect to the environment.

12. The apparatus of claim 11, wherein said correction surface has parameters comprising three rotational coordinates, two translational coordinates, a lens parameter, and a scale parameter.

13. The apparatus of claim 9, wherein said display controller includes a video adapter and is configured to set parameters of a transformation matrix to distort said undistorted image into said distorted image.

14. The apparatus of claim 8, wherein said sensing device comprises a camera configured to sense said interactions, further comprising an image processor configured to receive a video data stream from said camera and to output a set of object parameters O and a set of interaction parameters K to said display controller, said set of object parameters including at least one of:

a number of objects;

a location of each of the objects; and

a label of each of the objects;

said set of interaction parameters including at least one of;

a number of interactions detected;

a location of each of said interactions; and

a type of each of said interactions.

15. The apparatus of claim 14, further comprising a combined mounting, wherein said camera and said projector are collocated with said combined mounting for substantially coupled motion.

16. The apparatus of claim 14, wherein said camera is configured to receive a camera image formed by a reflection, from one of the surfaces, of one of said images projected by said projector.

17. The apparatus of claim 16, wherein said image processor is further configured to:

un-distort said camera image to create an undistorted camera image;

find a difference image between said undistorted camera image and an undistorted form of said image projected by said projector;

threshold and filter said difference image to yield foreground regions corresponding to objects within a region of the environment viewed by said camera; and

analyze shape and movement of said foreground regions to detect interactions by the objects on the plurality of surfaces.

18. The apparatus of claim 16, wherein said image processor is further configured to:

un-distort said camera image to create an undistorted camera image;

find a difference image between said undistorted camera image and a previous undistorted camera image;

threshold and filter said difference image to yield foreground regions corresponding to objects within a region of the environment viewed by said camera; and

analyze shape and movement of said foreground regions to detect interactions by the objects on the plurality of surfaces.

19. The apparatus of claim 8, wherein said display controller is configured to communicate zoom and focus information to said projector.

20. An apparatus for interacting with a subject in an environment, said apparatus comprising:

at least one display device configured to display images on a plurality of surfaces in the environment;

at least one camera configured to sense interactions between the subject and at least some of said images;

at least one sensor configured to sense interactions between the subject and the environment remote from the images; and

at least one display controller coupled to said display device, said camera, and said sensor, said at least one display controller being configured to modify an image display of said display device responsive to said interactions sensed by said camera and said interactions sensed by said sensor.

21. The apparatus of claim 20, further comprising an image processor coupled to said camera and said display controller, said image processor being configured to process images from said camera to detect objects in the environment and to further detect interactions of the objects with the plurality of surfaces.

22. The apparatus of claim 21, wherein said image processor is configured to:

determine a difference between a current image from said camera and a previous image from said camera, in order to create a difference image;

threshold and filter said difference image to yield foreground regions corresponding to given ones of the objects within a region of the environment viewed by said camera; and

analyze shape and movement of said foreground regions to detect interactions by the given ones of the objects on the plurality of surfaces.

23. An apparatus for interacting with a subject in an environment, said apparatus comprising:

at least one video projector configured to display images on a plurality of surfaces in the environment;

at least one video projector mounting coupled to said video projector and configured to move said video projector in at least two rotational degrees of freedom;

at least one video camera configured to sense interactions between the subject and the environment; and

at least one controller coupled to said video projector, said video projector mounting, and said video camera, said at least one controller in turn comprising;

a memory; and

at least one processor, coupled to said memory, operative to modify an image display by said video projector responsive to at least one of said interactions sensed by said video camera.

24. The apparatus of claim 23, wherein the surfaces on which said images are displayed are divided into a plurality of display zones, and wherein said processor is further operative to:

determine whether any of said display zones are adjacent a given one of said interactions; and

responsive to a determination that at least one of said display zones is an adjacent display zone that is adjacent said given one of said interactions, select said adjacent display zone for display of an updated image modified in accordance with said given one of said interactions.

25. The apparatus of claim 23, wherein said two degrees of rotational freedom comprise pan and tilt, and wherein said processor is further operative to:

distort an undistorted projection image to form a distorted projection image, said distorted projection image being distorted so as to form, in an undistorted manner, one of said images displayed on one of the plurality of surfaces;

output said distorted projection image to said video projector;

cause said video projector mounting to move so as to cause said video projector to project said distorted projection image on a desired one of the plurality of surfaces; and

communicate desired pan and tilt parameters to said video projector mounting.

26. The apparatus of claim 25, wherein said processor is further configured to un-distort an image from said camera to obtain an undistorted camera image and to compare said undistorted projection image with said undistorted camera image to determine whether an object is present in said image from said camera.

27. The apparatus of claim 25, wherein said processor is further configured to un-distort a series of images from said camera to obtain a series of undistorted camera images and to compare successive undistorted camera images to determine whether an object is present in a given one of said images from said series of images from said camera.

28. The apparatus of claim 23, wherein said video projector mounting is further configured to move said video projector in at least one translational degree of freedom.

29. A computer-implemented method for interacting with a subject in an environment, said method comprising the steps of:

displaying images on a plurality of surfaces in the environment;

sensing interactions between the subject and the environment, at least some of said interactions comprising interactions that are dispersed through the environment and remote from said images; and

modifying an image display responsive to at least one of said sensed interactions.

30. The method of claim 29, wherein the surfaces on which said images are displayed are divided into a plurality of display zones, further comprising the additional steps of:

determining whether any of said display zones are adjacent a given one of said interactions; and

responsive to a determination that at least one of said display zones is an adjacent display zone that is adjacent said given one of said interactions, selecting said adjacent display zone for display of an updated image modified in accordance with said given one of said interactions.

31. The method of claim 30, wherein said images are displayed by display devices in said display zones, further comprising the additional steps of:

determining whether at least one of said display devices in said adjacent display zone is moveable; and

responsive to determining that said display device in said adjacent display zone is moveable, moving said display device in said adjacent display zone to a new location.

32. The method of claim 29, wherein the surfaces on which said images are displayed are divided into a plurality of display zones, further comprising the additional steps of:

determining whether any of said display zones are adjacent a given one of said interactions; and

responsive to a determination that none of said display zones are adjacent said given one of said interactions, determining whether other notification to the subject is possible.

33. The method of claim 31, further comprising the additional step of eliminating said given one of said interactions from a list, responsive to a determination that no other notification is possible to the subject.

34. The method of claim 31, further comprising the additional step of sending said other notification to the subject, responsive to a determination that said other notification is possible to the subject.

35. A computer program product comprising a computer usable medium having computer usable program code for interacting with a subject in an environment, said computer program product including:

computer usable program code for displaying images on a plurality of surfaces in the environment;

computer usable program code for sensing interactions between the subject and the environment, at least some of said interactions comprising interactions that are dispersed through the environment and remote from said images; and

computer usable program code for modifying an image display responsive to at least one of said sensed interactions.