USING DEPTH INFORMATION FOR DRAWING IN AUGMENTED REALITY SCENES

• US 20160140761A1
• Filed: 11/19/2014
• Published: 05/19/2016
• Est. Priority Date: 11/19/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Saunders, Justin, Henshaw, Jeff, Berrett, Doug, Cooper, Matthew, Palotas, Michael
• Application Number:
  US14/547,619
• Publication Number:
  US 20160140761A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Optimizing augmented reality scenes by using depth information to accurately display interactions between real objects and synthetic objects is described. A stream of depth data associated with a real scene of an augmented reality display and a stream of color data associated with the real scene may be received. The stream of depth data may be processed to construct a first mesh and the first mesh may be projected into a color space associated with the stream of color data to construct a second mesh. In some examples, a position of the synthetic objects respective to real objects in the real scene may be determined and/or queries may be conducted to determine how the synthetic objects interact with the real objects in the real scene. Based at least on constructing the second mesh, determining positions, and/or conducting queries, one or more synthetic objects may be drawn into the real scene.

Claims

1. A computer-implemented method comprising:

receiving a stream of depth data associated with a real scene of an augmented reality display;

receiving a stream of color data associated with the real scene;

processing the stream of depth data to construct a first mesh;

projecting the first mesh into a color space associated with the stream of color data to construct a second mesh; and

drawing one or more synthetic objects into the real scene based at least in part on boundaries of real objects in the real scene that are defined by the second mesh.

2. The computer-implemented method of claim 1, wherein:

the stream of depth data comprises a point cloud; and

processing the stream of depth data comprises performing triangulation calculations between individual depth pixels in the point cloud to construct the first mesh.

3. The computer-implemented method of claim 1, wherein the second mesh maps precisely to the real scene.

4. The computer-implemented method of claim 1, further comprising rendering the second mesh into a visibility buffer for rendering against one or more real objects in the real scene.

5. The computer-implemented method of claim 1, wherein the processing the stream of depth data to construct the first mesh and the projecting the first mesh into the color space to construct the second mesh are performed in a recurring loop to construct a dynamic mesh that is reflective of the real scene in substantially real time.

6. The computer-implemented method of claim 1, further comprising performing one or more queries to determine how the one or more synthetic objects interact with one or more real objects in the real scene.

7. The computer-implemented method of claim 6, wherein the one or more queries include:

collision queries to determine how the one or more synthetic objects interact with the one or more real objects before, during, and after a collision between the any one or more synthetic objects and any one of the one or more real objects; and

geometric queries to determine surface angles of the one or more real objects and/or contours of the one or more real objects.

8. The computer-implemented method of claim 1, further comprising determining a visibility of a particular synthetic object of the one or more synthetic objects in the real scene based at least in part on the boundaries defined by the second mesh.

9. The computer-implemented method of claim 8, wherein determining the visibility comprises:

determining that the particular synthetic object is positioned at least partially behind one of the one or more real objects in the real scene; and

drawing the particular synthetic object so that it is at least partially occluded behind the one of the one or more real objects in the real scene.

10. A system comprising:

computer-readable media storing at least a rendering module;

a processing unit operably coupled to the computer-readable media, the processing unit adapted to execute at least the rendering module, the rendering module comprising;

an input module for receiving at least two data streams associated with one or more real objects in a real scene, wherein a first data stream of the at least two data streams includes a depth data stream and a second data stream of the at least two data streams includes a color data stream;

a reconstruction module for constructing a mesh defining surfaces associated with the one or more real objects in the real scene, the constructing based at least in part on projecting the depth data from the first data stream into color data from the second data stream; and

a drawing module for drawing one or more synthetic objects into the real scene based at least in part on boundaries of the one or more real objects that are defined by the mesh.

11. The system of claim 10, further comprising position module for determining how to position the one or more synthetic objects in the real scene based at least in part on the boundaries of the one or more real objects that are defined by the mesh, wherein positioning the one or more synthetic objects comprises:

determining that the one or more synthetic objects are positioned behind the one or more real objects and partially occluding the one or more synthetic objects behind the one or more real objects; and

determining that the one or more real objects are positioned behind the one or more synthetic objects and partially occluding the one or more real objects behind the one or more synthetic objects.

12. The system of claim 10, further comprising a query module for performing queries against the one or more real objects in the real scene to determine how the one or more synthetic objects interact with the one or more real objects, the queries based at least in part on the boundaries of the one or more real objects that are defined by the mesh.

13. The system of claim 12, wherein the queries comprise collision queries to determine how the one or more synthetic objects and the one or more real objects interact responsive to one of the one or more synthetic objects colliding with one of the one or more real objects.

14. The system of claim 12, wherein the queries comprise geometric queries to determine at least an angle, contour, or size of the one or more real objects.

15. The system of claim 10, wherein the mesh comprises a transformed mesh, and constructing the transformed mesh comprises:

extracting a point cloud from the stream of depth data;

processing the point cloud using triangulation calculations to construct a first mesh that is mapped to a surface of the one or more real objects;

projecting the first mesh into a color space associated with the stream of color data; and

interpolating depth data between depth pixels in the first mesh to construct the transformed mesh.

16. The system of claim 15, wherein the reconstruction module constructs the transformed mesh in substantially real time.

17. One or more computer-readable media encoded with instructions that, when executed by a processor, configure a computer to perform acts comprising:

receive depth data comprising a plurality of depth pixels arranged in a point cloud representative of a real scene of an augmented reality display;

receive a stream of color data associated with the real scene;

construct a mesh based at least in part on the plurality of depth pixels;

update the mesh based at least in part on projecting the mesh into a color space associated with the stream of color data; and

draw at least one synthetic object into the real scene based at least in part on surface boundaries defined by the mesh.

18. One or more computer-readable media as claim 17 recites, wherein the acts further comprise:

determining a position of the at least one synthetic object with respect to one or more real objects in the real scene based at least in part on the surface boundaries defined by the mesh;

based at least in part on determining that the at least one synthetic object is positioned behind the one or more real objects, drawing the at least one synthetic object sub-imposed over the one or more real objects;

based at least in part on determining that the at least one synthetic object is positioned in front of the one or more real objects, drawing the at least one synthetic object superimposed in front of the one or more real objects.

19. One or more computer-readable media as claim 18 recites, wherein the acts further comprise performing one or more queries to determine how the at least one synthetic object interacts with one or more real objects in the real scene.

20. One or more computer-readable media as claim 17 recites, wherein constructing the mesh based at least in part on the plurality of depth pixels and updating the mesh based at least in part on projecting the mesh into a color space associated with the stream of color data are performed on a frame by frame basis so that every frame of the augmented reality display reflects a substantially real time physical, real-world view.