Providing an interactive experience using a 3D depth camera and a 3D projector

• US 9,329,469 B2
• Filed: 03/29/2011
• Issued: 05/03/2016
• Est. Priority Date: 02/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Benko, Hrvoje, Jota Costa, Ricardo Jorge, Wilson, Andrew D.
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US13/074,041
• Publication Number:
  US 20120212509A1
• Time in Patent Office: 1,862 Days
• Field of Search: 345/419, 345/420, 348/50
• US Class Current: 1/1
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G03B 35/00 Stereoscopic photography
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

An interaction system is described which uses a depth camera to capture a depth image of a physical object placed on, or in vicinity to, an interactive surface. The interaction system also uses a video camera to capture a video image of the physical object. The interaction system can then generate a 3D virtual object based on the depth image and video image. The interaction system then uses a 3D projector to project the 3D virtual object back onto the interactive surface, e.g., in a mirrored relationship to the physical object. A user may then capture and manipulate the 3D virtual object in any manner. Further, the user may construct a composite model based on smaller component 3D virtual objects. The interaction system uses a projective texturing technique to present a realistic-looking 3D virtual object on a surface having any geometry.

Claims

1. A method for providing an interactive experience to a user using an interaction system, comprising:

capturing a depth image of a physical object placed on, or in proximity to, a surface using a depth camera, the depth image comprising information regarding the distance of different parts of the physical object to the depth camera;

capturing a video image of the physical object using a video camera, wherein the depth camera is not the video camera, and wherein the video image is captured in conjunction with the depth image;

generating a 3D virtual object which is a counterpart of the physical object based on the depth image and the video image; and

projecting, using a 3D projector, the 3D virtual object on, or in proximity to, the surface.

2. The method of claim 1, further comprising dynamically repeating said capturing of the depth image, capturing of the video image, generating the 3D virtual object, and projecting the 3D virtual object, to provide a dynamic and interactive experience to the user.

3. The method of claim 1, further comprising:

receiving an instruction from the user to capture an instance of the physical object; and

capturing the instance of the physical object in response to the instruction, to provide a captured 3D virtual object.

4. The method of claim 3, further comprising:

receiving an instruction from the user to manipulate the generated 3D virtual object; and

manipulating the generated 3D virtual object in response to the instruction to manipulate.

5. The method of claim 4, further comprising using said capturing and manipulating to aggregate two or more 3D virtual objects into a composite object, said two or more 3D virtual objects being derived from one or more physical objects.

6. The method of claim 1, further comprising removing background from the depth image using a previously-captured background image.

7. The method of claim 1, further comprising, prior to generating the 3D virtual object:

partitioning the depth image into a head region and an objects region; and

estimating a direction of attention of the user based on the head region.

8. The method of claim 1, wherein said generating of the 3D virtual object involves:

identifying a geometrical shape based on points associated with the depth image; and

applying a texture, obtained from the video image, to the geometrical shape, to add an appearance to the geometrical shape.

9. The method of claim 1, wherein said generating of the 3D virtual object involves:

creating a texture image from a point of view of the user; and

re-rendering the texture image from a point of view of the 3D projector, to provide a representation of the 3D virtual object in a form of a perspective-adjusted image,

said creating and re-rendering taking into account at least a geometry of a scene on which the 3D virtual object is to be projected, as represented in the depth image.

10. The method of claim 1, wherein said generating of the 3D virtual object involves using a projective texturing technique to generate a perspective-adjusted image based on a consideration of a direction of attention of the user and a position of the 3D projector.

11. The method of claim 10, wherein said projective texturing technique also takes into consideration a geometry of a scene on which the 3D virtual object is to be projected, as represented in the depth image.

12. The method of claim 11, wherein the geometry is irregular.

13. The method of claim 1, wherein said projecting of the 3D virtual object comprises projecting the 3D virtual object on, or in proximity to, the surface in a mirrored relationship with the physical object.

14. The method of claim 1, wherein the user is a local user who is present at a local location, and further comprising:

projecting the 3D virtual object to a remote user, who interacts with another interaction system at a remote location;

generating, using the other interaction system, another 3D virtual object based on a physical object that is present at the remote location; and

projecting the other 3D virtual object to the local user at the local location.

15. The method of claim 1, further comprising:

detecting a triggering action performed by the user with respect to the 3D virtual object that invokes a simulation effect, the triggering action being detected based on an interaction of a manipulating object with the 3D virtual object; and

applying the simulation effect to the 3D virtual object, the simulation effect defining a behavior of the 3D virtual object in response to the triggering action,

the manipulating object corresponding to a representation of at least one of;

a body part of the user and a tool controlled by the user.

16. An interaction system, comprising:

an illumination source for projecting radiation onto a surface;

a depth camera for capturing a depth image of at least one physical object by receiving radiation, emitted by the illumination source, the depth image comprising information regarding the distance of different parts of the physical object to the depth camera, which illuminates the at least one physical object on, or in proximity to, the surface;

a video camera for capturing a video image of said at least one physical object, wherein the depth camera and the video camera are separate cameras, and wherein the video camera captures the video image of the at least one physical object in conjunction with the depth camera capturing the depth image of the at least one physical object;

a virtual scene generation module for generating at least one 3D virtual object based on the depth image and the video image; and

a stereoscopic projector for projecting said at least one 3D virtual object on, or in proximity to, the surface.

17. The interaction system of claim 16, further comprising a control device for manipulating said at least one 3D virtual object that has been projected on, or in proximity to, the surface.

18. The interaction system of claim 16, further comprising two or more depth cameras and two or more video cameras.

19. A computer readable storage device for storing computer readable instructions, the computer readable instructions providing a virtual scene generation module when executed by one or more processing devices, the computer readable instructions comprising:

logic configured to receive a depth image from a depth camera of a physical object placed on, or in proximity to, a surface, the depth image comprising information regarding the distance of different parts of the physical object to the depth camera;

logic configured to receive a video image from a video camera of the physical object, wherein the depth camera is not the video camera, and wherein the video camera captures the video image of the physical object in conjunction with the depth camera capturing the depth image of the physical object;

logic configured to generate a 3D virtual object, the 3D virtual object being a 3D representation of the physical object based on the depth image and the video image; and

logic configured to apply a texture, obtained from the video image, to the 3D virtual object in a form of a perspective-adjusted 3D image,

the perspective-adjusted 3D image being projected, by a 3D projector, to a location on, or in proximity to, the surface.

20. The computer readable storage device of claim 19, wherein said logic configured to apply the texture comprises logic for using a projective texturing technique to generate the perspective-adjusted image based on:

a consideration of a direction of attention of a user;

a position of the 3D projector; and

a geometry of a scene on which the 3D virtual object is to be projected, as represented by the depth image.