Methods and Systems for Generating Depth Data by Converging Independently-Captured Depth Maps

• US 20180124371A1
• Filed: 10/31/2016
• Published: 05/03/2018
• Est. Priority Date: 10/31/2016
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Kamal, Syed Meeran, Smith, Steven L., Pan, Yongsheng, Virodov, Sergey, Globerson, Jonathan A., Chandrasiri, Lama Hewage Ravi Prathapa, Khalid, Mohammad Raheel
• Granted Patent: US 10,271,033 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An exemplary depth data generation system (“system”) accesses a first depth map and a second depth map of surfaces of objects included in a real-world scene. The first and second depth maps are captured independently from one another. The system converges the first and second depth maps into a converged depth map of the surfaces of the objects included in the real-world scene. More specifically, the converging comprises assigning a first confidence value to a first depth data point in the first depth map, assigning a second confidence value to a second depth data point in the second depth map, and generating a third depth data point representing a same particular physical point as the first and second depth data points based on the first and second confidence values and on at least one of the first depth data point and the second depth data point.

Claims

1. A method comprising:

accessing, by a depth data generation system, a first depth map of surfaces of objects included in a real-world scene, the first depth map including a first plurality of depth data points each representative of a different physical point included in a plurality of physical points on the surfaces of the objects included in the real-world scene;

accessing, by the depth data generation system, a second depth map of the surfaces of the objects included in the real-world scene, the second depth map captured independently from the first depth map and including a second plurality of depth data points each representative of a different physical point included in the plurality of physical points on the surfaces of the objects included in the real-world scene; and

converging, by the depth data generation system, the first and second depth maps into a converged depth map of the surfaces of the objects included in the real-world scene, the converged depth map including a third plurality of depth data points each representative of a different physical point included in the plurality of physical points on the surfaces of the objects included in the real-world scene, the converging comprising;

assigning a first confidence value to a first depth data point in the first plurality of depth data points of the first depth map, the first depth data point representing a particular physical point included in the plurality of physical points,

assigning a second confidence value to a second depth data point in the second plurality of depth data points of the second depth map, the second depth data point representing the particular physical point, and

generating, based on the first and second confidence values and on at least one of the first depth data point in the first plurality of depth data points and the second depth data point in the second plurality of depth data points, a third depth data point in the third plurality of depth data points, the third depth data point representing the particular physical point.

2. The method of claim 1, further comprising:

generating, by the depth data generation system based on the converged depth map, a data stream representative of a dynamic volumetric model of the surfaces of the objects included in the real-world scene, the dynamic volumetric model of the surfaces of the objects in the real-world scene configured to be used to generate virtual reality media content representative of the real-world scene as experienced from a dynamically selectable viewpoint corresponding to an arbitrary location within the real-world scene, the dynamically selectable viewpoint selected by a user of a media player device while the user is experiencing the real-world scene using the media player device; and

providing, by the depth data generation system to the media player device and based on the data stream, the virtual reality media content representative of the real-world scene.

3. The method of claim 1, wherein:

the assigning of the first confidence value to the first depth data point is performed based on at least one of

a first attribute of a particular surface that includes the particular physical point, the particular surface belonging to a particular object in the objects included in the real-world scene, and

a first viewing perspective, with respect to the particular surface that includes the particular physical point, of a component of the depth data generation system capturing the first depth map; and

the assigning of the second confidence value to the second depth data point is performed based on at least one of

a second attribute of the particular surface that includes the particular physical point, and

a second viewing perspective, with respect to the particular surface that includes the particular physical point, of a component of the depth data generation system capturing the second depth map.

4. The method of claim 1, wherein the generating of the third depth data point comprises at least one of:

identifying a maximum confidence value from the first and second confidence values and,

if the maximum confidence value is the first confidence value, assigning the first depth data point to the third depth data point, and

if the maximum confidence value is the second confidence value, assigning the second depth data point to the third depth data point; and

generating the third depth data point as a weighted average of the first and second depth data points in accordance with the first and second confidence values.

5. The method of claim 1, wherein:

the accessing of the first depth map includes capturing the first depth map by way of a first depth map capture technique; and

the accessing of the second depth map includes capturing the second depth map by way of a second depth map capture technique different from the first depth map capture technique.

6. The method of claim 5, wherein:

the depth data generation system includes a plurality of nodes each disposed at a different fixed node position in a plurality of fixed node positions with respect to the real-world scene;

the capturing of the first depth map and the capturing of the second depth map are performed from a particular node in the plurality of nodes; and

the first and second depth map capture techniques are different depth map capture techniques each selected from;

a structured light depth map capture technique,

a stereoscopic depth map capture technique, and

a time-of-flight depth map capture technique.

7. The method of claim 1, wherein:

the accessing of the first depth map includes capturing the first depth map by way of a depth map capture technique; and

the accessing of the second depth map includes capturing the second depth map by way of the depth map capture technique.

8. The method of claim 7, wherein:

the depth data generation system includes a plurality of nodes each disposed at a different fixed node position in a plurality of fixed node positions with respect to the real-world scene;

the capturing of the first depth map is performed from a first node in the plurality of nodes;

the capturing of the second depth map is performed from a second node in the plurality of nodes, the second node disposed at a different fixed node position than the first node in the plurality of fixed node positions; and

the depth map capture technique is selected from;

a structured light depth map capture technique,

a stereoscopic depth map capture technique, and

a time-of-flight depth map capture technique.

9. The method of claim 1, wherein:

the accessing of the first depth map includes converging a first plurality of antecedent depth maps into the first depth map, the antecedent depth maps in the first plurality of antecedent depth maps each captured by way of a first depth map capture technique; and

the accessing of the second depth map includes converging a second plurality of antecedent depth maps into the second depth map, the antecedent depth maps in the second plurality of antecedent depth maps each captured by way of a second depth map capture technique.

10. The method of claim 1, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

11. A method comprising:

capturing, by a depth data generation system, a first depth map of surfaces of objects included in a real-world scene, the first depth map including a first plurality of depth data points each representative of a different physical point included in a plurality of physical points on the surfaces of the objects included in the real-world scene, the capturing of the first depth map performed from a particular node disposed at a particular fixed node position with respect to the real-world scene and using a structured light depth map capture technique;

capturing, by the depth data generation system independently from the capturing of the first depth map, a second depth map of the surfaces of the objects included in the real-world scene, the second depth map including a second plurality of depth data points each representative of a different physical point included in the plurality of physical points on the surfaces of the objects included in the real-world scene, the capturing of the second depth map performed from the particular node disposed at the particular fixed node position and using a stereoscopic depth map capture technique;

converging, by the depth data generation system, the first and second depth maps into a converged depth map of the surfaces of the objects included in the real-world scene, the converged depth map including a third plurality of depth data points each representative of a different physical point included in the plurality of physical points on the surfaces of the objects included in the real-world scene, the converging comprising;

assigning a first confidence value to a first depth data point in the first plurality of depth data points of the first depth map, the first depth data point representing a particular physical point included in the plurality of physical points,

assigning a second confidence value to a second depth data point in the second plurality of depth data points of the second depth map, the second depth data point representing the particular physical point, and

generating, based on the first and second confidence values and on at least one of the first depth data point in the first plurality of depth data points and the second depth data point in the second plurality of depth data points, a third depth data point in the third plurality of depth data points, the third depth data point representing the particular physical point.

12. The method of claim 11, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

13. A system comprising:

at least one physical computing device that

accesses a first depth map of surfaces of objects included in a real-world scene, the first depth map including a first plurality of depth data points each representative of a different physical point included in a plurality of physical points on the surfaces of the objects included in the real-world scene;

accesses a second depth map of the surfaces of the objects included in the real-world scene, the second depth map captured independently from the first depth map and including a second plurality of depth data points each representative of a different physical point included in the plurality of physical points on the surfaces of the objects included in the real-world scene; and

converges the first and second depth maps into a converged depth map of the surfaces of the objects included in the real-world scene, the converged depth map including a third plurality of depth data points each representative of a different physical point included in the plurality of physical points on the surfaces of the objects included in the real-world scene, by

assigning a first confidence value to a first depth data point in the first plurality of depth data points of the first depth map, the first depth data point representing a particular physical point included in the plurality of physical points,

assigning a second confidence value to a second depth data point in the second plurality of depth data points of the second depth map, the second depth data point representing the particular physical point, and

generating, based on the first and second confidence values and on at least one of the first depth data point in the first plurality of depth data points and the second depth data point in the second plurality of depth data points, a third depth data point in the third plurality of depth data points, the third depth data point representing the particular physical point.

14. The system of claim 13, wherein the at least one physical computing device further:

generates, based on the converged depth map, a data stream representative of a dynamic volumetric model of the surfaces of the objects included in the real-world scene, the dynamic volumetric model of the surfaces of the objects in the real-world scene configured to be used to generate virtual reality media content representative of the real-world scene as experienced from a dynamically selectable viewpoint corresponding to an arbitrary location within the real-world scene, the dynamically selectable viewpoint selected by a user of a media player device while the user is experiencing the real-world scene using the media player device; and

provides, to the media player device and based on the data stream, the virtual reality media content representative of the real-world scene as experienced from the dynamically selectable viewpoint corresponding to the arbitrary location within the real-world scene.

15. The system of claim 13, wherein:

the at least one physical computing device assigns the first confidence value to the first depth data point based on at least one of

a first attribute of a particular surface that includes the particular physical point, the particular surface belonging to a particular object in the objects included in the real-world scene, and

a first viewing perspective, with respect to the particular surface that includes the particular physical point, of a component of the system capturing the first depth map; and

the at least one physical computing device assigns the second confidence value to the second depth data point based on at least one of

a second attribute of the particular surface that includes the particular physical point, and

a second viewing perspective, with respect to the particular surface that includes the particular physical point, of a component of the system capturing the second depth map.

16. The system of claim 13, wherein the at least one physical computing device generates the third depth data point by at least one of:

identifying a maximum confidence value from the first and second confidence values and,

if the maximum confidence value is the first confidence value, assigning the first depth data point to the third depth data point, and

if the maximum confidence value is the second confidence value, assigning the second depth data point to the third depth data point; and

generating the third depth data point as a weighted average of the first and second depth data points in accordance with the first and second confidence values.

17. The system of claim 13, wherein:

the at least one physical computing device accesses the first depth map by capturing the first depth map by way of a first depth map capture technique; and

the at least one physical computing device accesses the second depth map by capturing the second depth map by way of a second depth map capture technique different from the first depth map capture technique.

18. The system of claim 17, wherein:

the at least one physical computing device is distributed across a plurality of nodes each disposed at a different fixed node position in a plurality of fixed node positions with respect to the real-world scene;

the capturing of the first depth map and the capturing of the second depth map are performed from a particular node in the plurality of nodes; and

the first and second depth map capture techniques are different depth map capture techniques each selected from;

a structured light depth map capture technique,

a stereoscopic depth map capture technique, and

a time-of-flight depth map capture technique.

19. The system of claim 13, wherein:

the at least one physical computing device accesses the first depth map by capturing the first depth map by way of a depth map capture technique; and

the at least one physical computing device accesses the second depth map by capturing the second depth map by way of the depth map capture technique.

20. The system of claim 19, wherein:

the at least one physical computing device is distributed across a plurality of nodes each disposed at a different fixed node position in a plurality of fixed node positions with respect to the real-world scene;

the capturing of the first depth map is performed from a first node in the plurality of nodes;

the capturing of the second depth map is performed from a second node in the plurality of nodes, the second node disposed at a different fixed node position than the first node in the plurality of fixed node positions; and

the depth map capture technique is selected from;

a structured light depth map capture technique,

a stereoscopic depth map capture technique, and

a time-of-flight depth map capture technique.