Interactive video display system

• US 7,259,747 B2
• Filed: 05/28/2002
• Issued: 08/21/2007
• Est. Priority Date: 06/05/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Reactrix Systems Incorporated
• Inventors: Bell, Matthew
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US10/160,217
• Publication Number:
  US 20020186221A1
• Time in Patent Office: 1,911 Days
• Field of Search: 345156-160, 345/174, 345418-434, 345471-475, 250/221, 348/140, 463 30- 40
• US Class Current: 345/156
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G03B 21/26 Projecting separately subsidiary matter simultaneously with main image
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A device allows easy and unencumbered interaction between a person and a computer display system using the person'"'"'s (or another object'"'"'s) movement and position as input to the computer. In some configurations, the display can be projected around the user so that that the person'"'"'s actions are displayed around them. The video camera and projector operate on different wavelengths so that they do not interfere with each other. Uses for such a device include, but are not limited to, interactive lighting effects for people at clubs or events, interactive advertising displays, etc. Computer-generated characters and virtual objects can be made to react to the movements of passers-by, generate interactive ambient lighting for social spaces such as restaurants, lobbies and parks, video game systems and create interactive information spaces and art installations. Patterned illumination and brightness and gradient processing can be used to improve the ability to detect an object against a background of video images.

Claims

1. A system comprising:

a camera system operable to provide images of an object;

a display system operable to render video images onto a surface; and

a processor coupled to receive said images from said camera system and coupled to drive said display system, said processor operable to determine a strength and a direction of interaction between the object and at least one of the video images.

2. A system in accordance with claim 1, wherein said processor is further operable to determine an influence image that includes a region around an object image derived from the object, and wherein said strength and said direction of interaction are derived from said influence image.

3. A system in accordance with claim 2, wherein said processor is farther operable to determine a gradient of the influence image and to derive said direction of interaction based on said gradient.

4. A system in accordance with claim 3, wherein said processor is further operable to determine a brightness of the influence image and to derive said strength of interaction based on said brightness.

5. A system in accordance with claim 2, wherein said processor is further operable to determine a brightness of the influence image and to derive said strength of interaction based on said brightness.

6. A system in accordance with claim 1, wherein said interaction is the object pushing the at least one video image.

7. A system in accordance with claim 1, wherein said interaction is the object touching the at least one video image.

8. A system in accordance with claim 1, wherein said interaction is the object deforming the at least one video image.

9. A system in accordance with claim 1, wherein said interaction is the object manipulating the at least one video image.

10. A system in accordance with claim 1, wherein said processor is further operable to generate an adaptive model of a background to said object by analyzing changes over time in frames of the images from said camera system.

11. A system in accordance with claim 10, wherein said processor is further operable to generate said video images based on said adaptive model and said images from said camera system.

12. The system as described in claim 10 wherein said processor is further operable to generate said adaptive model of said background by identifying pixels in said frames having substantially constant brightness over time.

13. The system as described in claim 12 wherein said processor is further operable to generate said adaptive model of said background by computing median values over time for said pixels.

14. The system as described in claim 10 wherein said processor is further operable to generate said adaptive model of said background by computing median values over time for pixels in said frames.

15. The system as described in claim 10 wherein said processor is further operable to generate said adaptive model of said background by incorporating into said adaptive model of said background changes in said background that occur over time.

16. The system as described in claim 10 wherein said processor is further operable to generate said adaptive model of said background by incorporating into said adaptive model of said background changes in said background that occur due to changes in lighting affecting said background.

17. The system as described in claim 10 wherein said processor is further operable to generate said adaptive model of said background by creating a model image from said images from said camera system that classifies pixels in said model image as either foreground or background.

18. The system as described in claim 10 wherein said processor is further operable to generate said adaptive model of said background by incorporating into said adaptive model of said background information relating to at least a portion of said object that has remained substantially stationary for a period of time.

19. The system as described in claim 10 wherein said processor is further operable to generate said adaptive model of said background by computing a weighted average of a current frame of said images from said camera system with said adaptive model of said background.

20. The system as described in claim 19 wherein said processor is further operable to tune said weighted average to change a rate at which said model of said background adapts to changes in said images from said camera system.

21. The system as described in claim 10 wherein said processor is further operable to generate said display item by distinguishing between foreground that corresponds to said object and said background.

22. The system as described in claim 21 wherein said distinguishing comprises comparing a current frame of said images from said camera system with said adaptive model of said background.

23. The system as described in claim 22 wherein said distinguishing further comprises determining if differences between corresponding pixels in said current frame of said images from said camera system and said adaptive model of said background are greater than a threshold to determine location of said object.

24. The system as described in claim 10 wherein said camera system comprises two cameras to provide a stereo image, and wherein said processor is further operable to compute depth data based on said stereo image and to use said depth data to generate said model of said background.

25. A system in accordance with claim 1, wherein said camera system comprises a light source that is strobed.

26. A system in accordance with claim 25, wherein said processor is further operable to process said images from said camera system based on said strobed light source.

27. A system in accordance with claim 1, wherein said processor is further operable to detect a lighting phenomenon due to sources of light other than said camera system light source.

28. A system in accordance with claim 27, wherein said lighting phenomenon comprises shadows.

29. A system in accordance with claim 10, wherein at least a portion of said surface coincides with said background.

30. A system in accordance with claim 1, wherein said processor is further operable to detect said object and positions thereof.

31. A system in accordance with claim 1, wherein said video images comprise an advertisement.

32. A system in accordance with claim 1, wherein said processor is further operable to generate a display item having an animated characteristic.

33. A system in accordance with claim 32, wherein said processor is further operable to change said animated characteristic of said display item responsive to said interaction.