Methods, Systems and Computer Program Products for Creating Three Dimensional Meshes from Two Dimensional Images

• US 20130141433A1
• Filed: 01/23/2012
• Published: 06/06/2013
• Est. Priority Date: 12/02/2011
• Status: Abandoned Application

• Current Assignee: Sony Ericsson Mobile Communications USA Incorporated (Sony Group Corp.)
• Original Assignee: Sony Ericsson Mobile Communications USA Incorporated (Sony Group Corp.)
• Inventors: Astrand, Per
• Application Number:
  US13/355,960
• Publication Number:
  US 20130141433A1
• Time in Patent Office: Days
• US Class Current: 345/420
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/50 Depth or shape recovery

Abstract

Methods for obtaining a three-dimensional (3D) mesh from two dimensional images are provided. The methods include obtaining a series of 2D images using a camera array; calculating a depth map using the obtained series of 2D images; identifying portions of the calculated depth map that need additional detail; applying a textured based algorithm to the identified portions of the calculated depth map to obtain the additional detail in the depth map; and combining the calculated depth map with the obtained additional detail to provide a more accurate 3D mesh. Related systems and computer program products are also provided.

Claims

1. A method for obtaining a three-dimensional (3D) mesh from two dimensional images, the method comprising:

obtaining a series of 2D images using a camera array;

calculating a depth map using the obtained series of 2D images;

identifying portions of the calculated depth map that need additional detail;

applying a textured based algorithm to the identified portions of the calculated depth map to obtain the additional detail in the depth map; and

combining the calculated depth map with the obtained additional detail to provide a more accurate 3D mesh,

wherein at least one of the obtaining, calculating, identifying, applying and combining are implemented by at least one processor.

2. The method of claim 1, wherein the camera array is one of a matrix of 4×

4 camera and a matrix of 4×

5 cameras.

3. The method of claim 1, wherein the camera array is included in a computational camera of a wireless communication device.

4. The method of claim 1, wherein identifying portions of the calculated depth map comprises marking regions in the depth map having a distance greater than d, where d is the distance into the depth map defining when the additional detail is needed.

5. The method of claim 4, wherein applying a texture based algorithm comprises applying a texture based algorithm to the regions marked to obtain an improved mesh for the marked regions.

6. The method of claim 5, wherein combining further comprises combining the calculated depth map and the improved mesh for the marked regions to obtain the more accurate 3D mesh.

7. The method of claim 5, wherein combining is preceded by:

assigning a higher weight to the improved mesh for the marked regions of the depth map for regions with a distance greater than d; and

assigning a higher weight to calculated depth map for regions in the depth map having a distance less than d.

8. A system for obtaining a three-dimensional (3D) mesh of two-dimensional (2D) images, the system comprising:

a camera configured to obtain a series of 2D images using a camera array; and

a processor including;

a depth map module configured to calculate a depth map using the obtained series of 2D images;

a refinement module configured to identify portions of the calculated depth map that need additional detail in the depth map; and

a texture based acquisition module configured to apply a textured based algorithm to the identified portions of the calculated depth map to obtain the additional detail,

wherein the refinement module is further configured to combine the calculated depth map with the obtained additional detail to provide a more accurate 3D mesh.

9. The system of claim 8, wherein the system is included in a wireless communication device.

10. The system of claim 8, wherein the camera is included in a wireless communications device.

11. The system of claim 8, wherein the refinement module is further configured to mark regions in the depth map having a distance greater than d, where d is the distance into the depth map defining when the additional detail is needed.

12. The system of claim 11, wherein the texture based acquisition module is further configured to apply a texture based algorithm to the regions marked to obtain an improved mesh for the marked regions.

13. The system of claim 12, wherein the refinement module is further configured to combine the calculated depth map and the improved mesh for the marked regions to obtain the more accurate 3D mesh.

14. The system of claim 12, wherein the refinement module is further configured to:

assign a higher weight to the improved mesh for the marked regions of the depth map for regions with a distance greater than d; and

assign a higher weight to calculated depth map for regions in the depth map having a distance less than d.

15. A computer program product for obtaining a three-dimensional (3D) mesh from two dimensional images, the computer program product comprising:

a non-transitory computer readable storage medium including computer readable program code embodied therein, the computer readable program code comprising;

computer readable program code configured to obtain a series of 2D images using a camera array;

computer readable program code configured to calculate a depth map using the obtained series of 2D images;

computer readable program code configured to identify portions of the calculated depth map that need additional detail;

computer readable program code configured to apply a textured based algorithm to the identified portions of the calculated depth map to obtain the additional detail in the depth map; and

computer readable program code configured to combine the calculated depth map with the obtained additional detail to provide a more accurate 3D mesh.

16. The computer program product of claim 15, wherein the computer readable program code configured to identify portions of the calculated depth map comprises computer readable program code configured to mark regions in the depth map having a distance greater than d, where d is the distance into the depth map defining when the additional detail is needed.

17. The computer program product of claim 16, wherein the computer readable program code configured to apply a texture based algorithm comprises computer readable program code configured to apply a texture based algorithm to the regions marked to obtain an improved mesh for the marked regions.

18. The computer program product of claim 16, wherein the computer readable program code configured to combine further comprises computer readable program code configured to combine the calculated depth map and the improved mesh for the marked regions to obtain the more accurate 3D mesh.

19. The computer program product of claim 16, further comprising:

computer readable program code configured to assign a higher weight to the improved mesh for the marked regions of the depth map for regions with a distance greater than d; and

computer readable program code configured to assign a higher weight to calculated depth map for regions in the depth map having a distance less than d.

20. The computer program product of claim 15, wherein the computer program product if executed by a processor of a wireless communications device.