Stereo video for gaming

• US 9,855,496 B2
• Filed: 04/27/2015
• Issued: 01/02/2018
• Est. Priority Date: 11/14/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Graepel, Thore K H, Blake, Andrew, Herbrich, Ralf
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US14/696,775
• Publication Number:
  US 20150231490A1
• Time in Patent Office: 981 Days
• Field of Search: None
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

A real-time stereo video signal of a captured scene with a physical foreground object and a physical background is received. In real-time, a foreground/background separation algorithm is used on the real-time stereo video signal to identify pixels from the stereo video signal that represent the physical foreground object. A video sequence may be produced by rendering a 3D virtual reality based on the identified pixels of the physical foreground object.

Claims

1. A method for producing a video sequence in a game console, the method comprising:

receiving, from a stereo camera, a real-time stereo video signal comprising paired left and right images of a captured scene, the captured scene comprising a physical foreground object and a physical background;

performing, in real-time, a foreground/background separation algorithm on the real-time stereo video signal to identify pixels captured from the real-time stereo video signal that are indicative of the physical foreground object;

detecting attributes of the physical foreground object;

obtaining a plurality of different kernel images representing different types of inanimate objects;

comparing the detected attributes of the physical foreground object to respective attributes of the plurality of different kernel images to identify a matching kernel image representing a particular type of inanimate object; and

producing the video sequence by rendering a virtual reality having a virtual object of the particular type represented by the matching kernel image.

2. The method of claim 1, further comprising:

prior to the comparing, scaling the different kernel images based at least on a respective distance from the stereo camera to the physical foreground object.

3. The method of claim 1, the producing comprising:

including, in the video sequence, an avatar holding the virtual object.

4. The method of claim 1, the pixels identified by the foreground/background separation algorithm comprising arbitrary and substantially non-uniform colors.

5. The method of claim 1, the foreground/background separation algorithm being configured to identify at least an obscuring portion of the physical foreground object that at least partially obscures at least part of the physical background that has a substantially same color as the obscuring portion of the physical foreground object.

6. The method of claim 1, the performing the foreground/background separation algorithm comprising obtaining a stereo-match likelihood and fusing the stereo-match likelihood with a contrast-sensitive color model.

7. The method of claim 1, the performing the foreground/background separation algorithm being based at least in part upon at least two of:

stereo disparity, stereo matching, and a color of respective pixels from frames of the real-time stereo video signal.

8. A system, comprising:

a processor; and

memory comprising instructions that, when executed by the processor, cause the processor to;

receive a stereo video signal comprising paired left and right images of a captured scene, the captured scene comprising a physical foreground object and a physical background;

perform a foreground/background separation algorithm on the stereo video signal to identify pixels captured from the stereo video signal that are indicative of the physical foreground object;

obtain a plurality of different kernel images representing different types of inanimate objects;

use the pixels indicative of the physical foreground object to identify a matching kernel image from the plurality of different kernel images;

determine that the physical foreground object is a particular type of inanimate object represented by the matching kernel image; and

produce a video sequence by rendering a virtual reality that includes a virtual object of the particular type of inanimate object represented by the matching kernel image.

9. The system of claim 8, embodied as a game console.

10. The system of claim 8, the plurality of different kernel images including an additional kernel image representing at least a portion of a person'"'"'s body.

11. The system of claim 8, wherein the virtual reality includes an avatar holding the virtual object.

12. The system of claim 8, the pixels indicative of the physical foreground object comprising arbitrary and substantially non-uniform colors.

13. The system of claim 8, the foreground/background separation algorithm being configured to identify at least an obscuring portion of the physical foreground object that at least partially obscures part of the physical background that has a substantially same color as the obscuring portion of the physical foreground object.

14. The system of claim 8, wherein the instructions, when executed by the processor, cause the processor to:

obtain a stereo match likelihood and fuse the stereo-match likelihood with a contrast-sensitive color model to identify the pixels indicative of the physical foreground object.

15. The system of claim 8, wherein the instructions, when executed by the processor, cause the processor to:

identify the pixels indicative of the physical foreground object based at least in part upon at least two of;

stereo disparity, stereo matching, and a color of respective pixels from frames of the stereo video signal.

16. A computer-readable storage device having computer-executable instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving a stereo video signal of a captured scene with a physical foreground object and a physical background;

performing a foreground/background separation algorithm on the stereo video signal to identify pixels captured from the stereo video signal that represent the physical foreground object, the foreground/background separation algorithm being performed based at least in part upon disparity; and

obtaining a plurality of different kernel images representing different types of inanimate objects;

using the pixels that represent the physical foreground object to identify a matching kernel image from the plurality of different kernel images, the matching kernel image representing a particular type of inanimate object; and

producing a video sequence by rendering a virtual reality that includes a virtual object of the particular type of inanimate object represented by the matching kernel image.

17. The computer-readable storage device of claim 16, the virtual object appearing to be held by an avatar in the virtual reality.

18. The computer-readable storage device of claim 16, the virtual object being cylindrical.

19. The computer-readable storage device of claim 16, the stereo video signal comprising paired left and right images of the captured scene.

20. A game console comprising one or more computer readable storage devices according to claim 16.