Methods and systems for rendering virtual reality content based on two-dimensional (“2D”) captured imagery of a three-dimensional (“3D”) scene

• US 10,417,810 B2
• Filed: 05/31/2017
• Issued: 09/17/2019
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Lodato, Michael, Sepulveda, Rene, Moodlagiri, Pai, Breitenfeld, Denny
• Primary Examiner(s): Hajnik, Daniel F
• Application Number:
  US15/610,586
• Publication Number:
  US 20180350134A1
• Time in Patent Office: 839 Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation

Abstract

An exemplary method includes a virtual reality content rendering system receiving two-dimensional (“2D”) color data and depth data captured by a plurality of capture devices disposed at different vantage points in relation to a three-dimensional (“3D”) scene, receiving metadata, generating, for each vantage point associated with each respective capture device included in the plurality of capture devices, and based on the metadata and the depth data, a partial 3D mesh projected into a virtual 3D space to produce a partial representation of the 3D scene in the virtual 3D space, and generating, based on the partial 3D meshes projected into the virtual 3D space, and from an arbitrary viewpoint within the virtual 3D space, an image view of the virtual 3D space. The generating of the image view may comprise accumulating the partial 3D meshes projected into the virtual 3D space.

Claims

1. A method comprising:

receiving, by a virtual reality content rendering system, two-dimensional (“

2D”

) color data and depth data captured by a plurality of capture devices disposed at different vantage points in relation to a three-dimensional (“

3D”

) scene;

receiving, by the virtual reality content rendering system, metadata for the 2D color data and the depth data;

generating, by the virtual reality content rendering system, for each vantage point associated with each respective capture device included in the plurality of capture devices, and based on the metadata and the depth data, a partial 3D mesh projected into a virtual 3D space to produce a partial representation of the 3D scene in the virtual 3D space; and

generating, by the virtual reality content rendering system based on the partial 3D meshes projected into the virtual 3D space, and from an arbitrary viewpoint within the virtual 3D space, an image view of the virtual 3D space, the generating of the image view comprising

accumulating the partial 3D meshes projected into the virtual 3D space, wherein the accumulating of the partial 3D meshes projected into the virtual 3D space comprises accumulating color samples for the partial 3D meshes in a frame buffer of a graphics processing unit (“

GPU”

), and

additively blending, based on the 2D color data, the color samples for overlapping sections of the partial 3D meshes in the frame buffer of the GPU to form the image view of the virtual 3D space, wherein the additively blending of the color samples for the overlapping sections of the partial 3D meshes comprises writing each color sample to the frame buffer when that color sample is sampled and selected to be written to the frame buffer, and additively blending each color sample with any previously written color samples in response to each color sample be written to the frame buffer.

2. The method of claim 1, wherein the generating of the partial 3D mesh for each vantage point associated with each respective capture device included in the plurality of capture devices comprises:

projecting, for each vantage point associated with each respective capture device included in the plurality of capture devices, a 2D input mesh along perspective rays to depths within the virtual 3D space based on the depth data to produce 3D coordinates of vertices of primitives of the partial 3D mesh in the virtual 3D space.

3. The method of claim 2, wherein the generating of the partial 3D mesh for each vantage point associated with each respective capture device included in the plurality of capture devices further comprises:

identifying, for each partial 3D mesh, one or more of the primitives of the partial 3D mesh as extraneous primitives; and

removing the identified extraneous primitives from each partial 3D mesh.

4. The method of claim 3, wherein the identifying, for each partial 3D mesh, one or more of the primitives of the partial 3D mesh as extraneous primitives comprises at least one of:

identifying a first primitive of the partial 3D mesh as a first extraneous primitive based at least in part on the first primitive having a depth change that exceeds a predetermined depth change threshold; and

identifying a second primitive of the partial 3D mesh as a second extraneous primitive based at least in part on the second primitive having a surface not facing the respective vantage point associated with the partial 3D mesh.

5. The method of claim 2, wherein the additively blending of the color samples for the overlapping sections of the partial 3D meshes to form the image view of the virtual 3D space comprises:

determining blend weights for the primitives of the partial 3D meshes;

weighting the color samples for the partial 3D meshes based on the determined blend weights for the primitives of the partial 3D meshes; and

determining weighted averages of the weighted color samples for superimposed primitives included in the overlapping sections of the partial 3D meshes.

6. The method of claim 1, further comprising:

determining, by the virtual reality content rendering system based on the partial 3D meshes projected into the virtual 3D space, and from the arbitrary viewpoint within the virtual 3D space, an accumulation region;

wherein the accumulating of the partial 3D meshes and the additively blending of the color samples for overlapping sections of the partial 3D meshes to form the image view of the virtual 3D space are performed only for fragments that are within the accumulation region.

7. The method of claim 1, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

8. A non-transitory computer-readable medium storing instructions that, when executed, direct at least one processor of a computing device to:

receive two-dimensional (“

2D”

) color data and depth data captured by a plurality of capture devices disposed at different vantage points in relation to a three-dimensional (“

3D”

) scene,

receive metadata for the 2D color data and the depth data,

generate, for each vantage point associated with each respective capture device included in the plurality of capture devices, and based on the metadata and the depth data, a partial 3D mesh projected into a virtual 3D space to produce a partial representation of the 3D scene in the virtual 3D space, and

generate, based on the partial 3D meshes projected into the virtual 3D space, and from an arbitrary viewpoint within the virtual 3D space, an image view of the virtual 3D space, the generating of the image view comprising accumulating color samples for the partial 3D meshes in a frame buffer of a graphics processing unit (“

GPU”

), and additively blending, based on the 2D color data, the color samples for overlapping sections of the partial 3D meshes in the frame buffer of the GPU to form the image view of the virtual 3D space, wherein the additively blending of the color samples for the overlapping sections of the partial 3D meshes comprises writing each color sample to the frame buffer when that color sample is sampled and selected to be written to the frame buffer, and additively blending each color sample with any previously written color samples in response to each color sample be written to the frame buffer.

9. The non-transitory computer-readable medium of claim 8, wherein the generating of the partial 3D mesh for each vantage point associated with each respective capture device included in the plurality of capture devices comprises:

the instructions, when executed, directing the at least one processor to project, for each vantage point associated with each respective capture device included in the plurality of capture devices, a 2D input mesh along perspective rays to depths within the virtual 3D space based on the depth data to produce 3D coordinates of vertices of primitives of the partial 3D mesh in the virtual 3D space.

10. The non-transitory computer-readable medium of claim 9, wherein the generating of the partial 3D mesh for each vantage point associated with each respective capture device included in the plurality of capture devices further comprises the instructions, when executed, directing the at least one processor to:

identify, for each partial 3D mesh, one or more of the primitives of the partial 3D mesh as extraneous primitives; and

remove the identified extraneous primitives from each partial 3D mesh.

11. The non-transitory computer-readable medium of claim 9, wherein the additively blending of the color samples for overlapping sections of the partial 3D meshes to form the image view of the virtual 3D space comprises the instructions, when executed, directing the at least one processor to:

determine blend weights for the primitives of the partial 3D meshes;

weight the color samples for the partial 3D meshes based on the determined blend weights for the primitives of the partial 3D meshes; and

determine weighted averages of the weighted color samples for superimposed primitives included in the overlapping sections of the partial 3D meshes.

12. The non-transitory computer-readable medium of claim 8, further comprising instructions that, when executed, direct the at least one processor to determine, based on the partial 3D meshes projected into the virtual 3D space, and from the arbitrary viewpoint within the virtual 3D space, an accumulation region;

wherein the additively blending of the color samples for the overlapping sections of the partial 3D meshes to form the image view of the virtual 3D space is performed only for color samples that are within the accumulation region.

13. A system comprising:

at least one computer processor; and

a virtual reality rendering facility that directs the at least one computer processor to;

receive two-dimensional (“

2D”

) color data and depth data captured by a plurality of capture devices disposed at different vantage points in relation to a three-dimensional (“

3D”

) scene,

receive metadata for the 2D color data and the depth data,

generate, for each vantage point associated with each respective capture device included in the plurality of capture devices, and based on the metadata and the depth data, a partial 3D mesh projected into a virtual 3D space to produce a partial representation of the 3D scene in the virtual 3D space, and

generate, based on the partial 3D meshes projected into the virtual 3D space, and from an arbitrary viewpoint within the virtual 3D space, an image view of the virtual 3D space, the generating of the image view comprising accumulating color samples for the partial 3D meshes in a frame buffer of a graphics processing unit (“

GPU”

), and additively blending, based on the 2D color data, the color samples for overlapping sections of the partial 3D meshes in the frame buffer of the GPU to form the image view of the virtual 3D space, wherein the additively blending of the color samples for the overlapping sections of the partial 3D meshes comprises writing each color sample to the frame buffer when that color sample is sampled and selected to be written to the frame buffer, and additively blending each color sample with any previously written color samples in response to each color sample be written to the frame buffer.

14. The system of claim 13, wherein the generating of the partial 3D mesh for each vantage point associated with each respective capture device included in the plurality of capture devices comprises:

the virtual reality rendering facility directing the at least one computer processor to project, for each vantage point associated with each respective capture device included in the plurality of capture devices, a 2D input mesh along perspective rays to depths within the virtual 3D space based on the depth data to produce 3D coordinates of vertices of primitives of the partial 3D mesh in the virtual 3D space.

15. The system of claim 14, wherein the generating of the partial 3D mesh for each vantage point associated with each respective capture device included in the plurality of capture devices further comprises:

the virtual reality rendering facility directing the at least one computer processor to identify, for each partial 3D mesh, one or more of the primitives of the partial 3D mesh as extraneous primitives; and

the virtual reality rendering facility directing the at least one computer processor to remove the identified extraneous primitives from each partial 3D mesh.

16. The system of claim 14, wherein the additively blending of the color samples for the overlapping sections of the partial 3D meshes to form the image view of the virtual 3D space comprises:

the virtual reality rendering facility directing the at least one computer processor to determine blend weights for the primitives of the partial 3D meshes;

the virtual reality rendering facility directing the at least one computer processor to weight the color samples for the partial 3D meshes based on the determined blend weights for the primitives of the partial 3D meshes; and

the virtual reality rendering facility directing the at least one computer processor to determine weighted averages of the weighted color samples for superimposed primitives included in the overlapping sections of the partial 3D meshes.

17. The system of claim 13, further comprising the virtual reality rendering facility directing the at least one computer processor to determine, based on the partial 3D meshes projected into the virtual 3D space, and from the arbitrary viewpoint within the virtual 3D space, an accumulation region;

wherein the additively blending of the color samples for the overlapping sections of the partial 3D meshes to form the image view of the virtual 3D space is performed only for color samples that are within the accumulation region.