STEREO VIDEO FOR GAMING

• US 20150231490A1
• Filed: 04/27/2015
• Published: 08/20/2015
• Est. Priority Date: 11/14/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Graepel, Thore KH, Blake, Andrew, Herbrich, Ralf
• Granted Patent: US 9,855,496 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

A real-time stereo video signal of a captured scene with a physical foreground object and a physical background is received. In real-time, a foreground/background separation algorithm is used on the real-time stereo video signal to identify pixels from the stereo video signal that represent the physical foreground object. A video sequence may be produced by rendering a 3D virtual reality based on the identified pixels of the physical foreground object.

Claims

1. A method for producing a video sequence in a game console, comprising:

receiving from a stereo camera a real-time stereo video signal comprising paired left and right images of a captured scene, the captured scene comprising a physical foreground object and a physical background;

performing, in real-time, a foreground/background separation algorithm on the real-time stereo video signal to identify pixels captured from the real-time stereo video signal that are indicative of the physical foreground object;

detecting attributes of the physical foreground object;

comparing the detected attributes of the physical foreground object to one or more kernel images, respective kernel images associated with a set of attributes;

identifying a kernel image associated with a set of attributes that substantially correspond to the detected attributes; and

producing a video sequence by rendering a virtual reality based at least in part upon the identified kernel image.

2. The method of claim 1, the physical foreground object not a human being.

3. The method of claim 1, producing the video sequence by rendering the virtual reality based at least in part upon the identified kernel image, comprising at least one of:

providing an avatar in the virtual reality with an object to hold based at least in part upon the identified kernel image;

providing an avatar in the virtual reality with a costume based at least in part upon the identified kernel image;

or

providing an avatar in the virtual reality with an object to use based at least in part upon the identified kernel image.

4. The method of claim 1, identifying the pixels indicative of the physical foreground object comprising identifying pixels that comprise arbitrary and substantially non-uniform colors.

5. The method of claim 1, the foreground/background separation algorithm configured to identify at least a portion of the physical foreground object that at least partially obscures at least a portion of the physical background that has a substantially same color as the obscuring portion of the physical foreground object.

6. The method of claim 1, identifying the pixels indicative of the physical foreground object comprising obtaining a stereo-match likelihood and fusing the stereo-match likelihood with a contrast-sensitive color model.

7. The method of claim 1, identifying the pixels indicative of the physical foreground object based at least in part upon at least two of:

stereo disparity, stereo matching, or a color of respective pixels from frames of the stereo video signal.

8. A system, comprising:

a processor; and

memory comprising instructions that, when executed by the processor, perform a method comprising;

receiving from a stereo camera a stereo video signal comprising paired left and right images of a captured scene, the captured scene comprising a physical foreground object and a physical background;

performing a foreground/background separation algorithm on the stereo video signal to identify pixels captured from the stereo video signal that are indicative of the physical foreground object;

detecting attributes of the physical foreground object;

comparing the detected attributes of the physical foreground object to one or more kernel images, respective kernel images associated with a set of attributes; and

identifying a kernel image associated with a set of attributes that substantially correspond to the detected attributes.

9. The system of claim 8, the method comprising producing a video sequence by rendering a virtual reality based at least in part upon the identified kernel image.

10. The system of claim 8, the physical foreground object not a human being.

11. The system of claim 9, producing the video sequence by rendering the virtual reality based at least in part upon the identified kernel image, comprising at least one of:

providing an avatar in the virtual reality with an object to hold based at least in part upon the identified kernel image;

providing an avatar in the virtual reality with a costume based at least in part upon the identified kernel image;

or

providing an avatar in the virtual reality with an object to use based at least in part upon the identified kernel image.

12. The system of claim 8, identifying the pixels indicative of the physical foreground object comprising identifying pixels that comprise arbitrary and substantially non-uniform colors.

13. The system of claim 8, the foreground/background separation algorithm configured to identify at least a portion of the physical foreground object that at least partially obscures at least a portion of the physical background that has a substantially same color as the obscuring portion of the physical foreground object.

14. The system of claim 8, identifying the pixels indicative of the physical foreground object comprising obtaining a stereo-match likelihood and fusing the stereo-match likelihood with a contrast-sensitive color model.

15. The system of claim 8, identifying the pixels indicative of the physical foreground object based at least in part upon at least two of:

stereo disparity, stereo matching, or a color of respective pixels from frames of the stereo video signal.

16. A computer-readable storage device having computer-executable instructions that, when executed by a processor, are configured to perform a method, the method comprising:

receiving from a stereo camera a stereo video signal of a captured scene with a physical foreground object and a physical background;

performing a foreground/background separation algorithm on the stereo video signal to identify pixels captured from the stereo video signal that represent the physical foreground object, the performing based at least in part upon disparity; and

producing a video sequence by rendering a virtual reality based at least in part upon the identified pixels of the physical foreground object, the identified pixels used to determine at least one object.

17. The computer-readable storage device of claim 16, the at least one object appearing to be at least one of held, worn, or used by an avatar in the virtual reality.

18. The computer-readable storage device of claim 16, the at least one object appearing to be at least one of held or worn by an avatar in the virtual reality.

19. The computer-readable storage device of claim 16, the stereo video signal comprising paired left and right images of the captured scene.

20. A game console comprising one or more computer readable storage devices according to claim 16.