Systems and methods 2-D to 3-D conversion using depth access segiments to define an object

• US 9,082,224 B2
• Filed: 07/13/2012
• Issued: 07/14/2015
• Est. Priority Date: 03/12/2007
• Status: Active Grant

• Current Assignee: Intellectual Discovery Co., Ltd.
• Original Assignee: Intellectual Discovery Co., Ltd.
• Inventors: Birtwistle, Steven, SPOONER, DAVID A., Wallner, Natascha, Lowe, Danny D., Keech, Gregory R., Adelman, Jonathan, Simmons, Christopher Levi
• Primary Examiner(s): AMIN, JWALANT B
• Application Number:
  US13/548,723
• Publication Number:
  US 20120275689A1
• Time in Patent Office: 1,096 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

The present invention is directed to systems and methods for controlling 2-D to 3-D image conversion. The system and method includes receiving an image and masking the objects in the image using segmentation layers Each segmentation layer can have weighted values for static and dynamic features. Iterations are used to form the final image which, if desired, can be formed using less than all of the segmentation layers. A final iteration can be run with the weighted values equal for static and dynamic features.

Claims

1. A non-transitory, machine-readable storage medium comprising instructions stored therein, the instructions executable by one or more processors to facilitate performing a method for creating one or more 3-D images, said method comprising:

obtaining object models of each said image to be created;

generating a plurality of segmentation layer masks from each said obtained model, each said segmentation layer mask having a plurality of features;

rendering a 3-D image from an assembled set of said generated masks;

classifying each of said plurality of features as being a static feature or a dynamic feature based on how closely a 3-D point in a point cloud associated with said 3-D image is mapped to a corresponding 2-D point in one or more 2-D images;

assigning weighted values to said plurality of features such that static features receive a higher weight value than do dynamic features; and

calculating a scene geometry by iterating said segmentation layer masks, said iterating comprising iteratively determining one or more features associated with one or more respective 3-D points that do not satisfy a threshold for mapping to one or more respective 2-D points, and iteratively adjusting one or more of said weighted values based on said determined one or more features, said iterating terminating after a plurality of iterations when a predetermined condition is satisfied.

2. The non-transitory, machine-readable storage medium of claim 1 wherein said generating a plurality of segmentation layer masks comprises:

rendering said point cloud according to a camera model;

obtaining a boolean image from said point cloud;

identifying a sequence of boundary points in said point cloud;

obtaining said polycurve from said sequence of boundary points; and

refining said polycurve.

3. The non-transitory, machine-readable storage medium of claim 2 wherein said rendering comprises:

forming, from each said mask, a 2-D mesh, and converting each said mesh into a 3-D planar mesh.

4. The non-transitory, machine-readable storage medium of claim 3 further comprising:

forming a 3-D mesh from said planar mesh.

5. The non-transitory, machine-readable storage medium of claim 4 wherein said 2-D mesh is constructed using vertices and wherein said 3-D forming comprises:

replicating surface vertices of said 2-D mesh.

6. The non-transitory, machine-readable storage medium of claim 4 wherein said rendering comprises:

utilizing less than a full set of said segmentation masks.

7. The machine-readable storage medium of claim 1, wherein said plurality of segmentation layer masks are generated using polycurve techniques.

8. A non-transitory, machine-readable storage medium comprising instructions stored therein, the instructions executable by one or more processors to facilitate performing a method, said method comprising:

rendering a point cloud according to a camera model, said camera model mapping a plurality of 3-D points in said point cloud to a plurality of 2-D points in one or more 2-D images;

obtaining a boolean image from said point cloud;

segmenting said boolean image to identify a plurality of segmentation layers within said boolean image, each said layer having a plurality of features;

classifying each of said features as being a static feature or a dynamic feature based on how closely a 3-D point in said point cloud is mapped to a corresponding 2-D point;

assigning weighted values to said plurality of features such that static features receive a higher weight value than do dynamic features;

calculating a scene geometry; and

iteratively determining one or more features associated with one or more respective 3-D points that do not satisfy a threshold for mapping to one or more respective 2-D points, and iteratively adjusting one or more of said weighted values based on said determined one or more features, said iterating terminating after a plurality of iterations when a predetermined condition is satisfied.

9. The machine-readable storage medium of claim 8, wherein said method further comprises:

terminating said iterations when at least one of the following occurs;

a feature classification does not change between iterations;

specified number of iterations has been performed;

specified condition has been achieved.

10. The machine-readable storage medium of claim 8, wherein said method further comprises:

identifying a sequence of boundary points in said point cloud;

obtaining a polycurve from said sequence of boundary points; and

refining said polycurve.

11. The machine-readable storage medium of claim 8, wherein said method further comprises:

rendering a 3-D volume image from an assembled set of generated masks.

12. The machine-readable storage medium of claim 11, wherein said rendering comprises:

forming, from each said mask, a 2-D mesh, and

converting each said mesh into a 3-D volume mesh.

13. The machine-readable storage medium of claim 12, wherein said 2-D mesh is constructed using vertices and wherein said 3-D volume mesh converting comprises:

forming a planar 3-D mesh from said 2-D mesh; and

lofting by replicating said surface vertices of said 2-D mesh.

14. The machine-readable storage medium of claim 8, wherein said method further comprises:

resetting said weighted values to be even to each other;

running at least one additional iteration with said reset values.

15. The machine-readable storage medium of claim 14, wherein said weighted values associated with said dynamic features are reset to zero and another iteration is run.

16. The machine-readable storage medium of claim 8, wherein said scene geometry is calculated based on iterating said segmentation layers.

17. The machine-readable storage medium of claim 16, wherein said iterating said segmentation layers comprises said step of iteratively determining said one or more features associated with said one or more respective 3-D points that do not satisfy a threshold for mapping to said one or more respective 2-D points.

18. A system for converting 2-D images to 3-D images, said system comprising:

a memory storing executable instructions;

one or more processors for executing executable instructions, the one or more processors being configured to;

execute control instructions for rendering a point cloud according to a camera model, said camera model mapping a plurality of 3-D points in said point cloud to a plurality of 2-D points in one or more 2-D images;

execute control instructions for obtaining a boolean image from said point cloud;

execute control instructions for segmenting said boolean image to identify a plurality of segmentation layers within said boolean image, each said layer having a plurality of features;

execute control instructions for classifying each of said features as being a static feature or a dynamic feature based on how closely a 3-D point in said point cloud is mapped to a corresponding 2-D point;

execute control instructions for assigning weighted values to said plurality of features such that static features receive a higher weight value than do dynamic features;

execute control instructions for calculating a scene geometry; and

execute control instructions for iteratively determining one or more features associated with one or more respective 3-D points that do not satisfy a threshold for mapping to one or more respective 2-D points, and iteratively adjusting one or more of said weighted values based on said determined one or more features, said iterating terminating after a plurality of iterations when a predetermined condition is satisfied.

19. A system for creating one or more 3-D images, said system comprising:

a memory storing executable instructions;

one or more processors for executing executable instructions, the one or more processors being configured to;

execute control instructions for obtaining object models of each said image to be created;

execute control instructions for generating a plurality of segmentation layer masks from each said obtained model, each said segmentation layer mask having a plurality of features;

execute control instructions for rendering a 3-D image from a set of said masks;

execute control instructions for classifying each of said plurality of features as being a static feature or a dynamic feature based on how closely a 3-D point in a point cloud associated with said 3-D image is mapped to a corresponding 2-D point in one or more 2-D images;

execute control instructions for assigning weighted values to said plurality of features such that static features receive a higher weight value than do dynamic features; and

execute control instructions for calculating a scene geometry by iterating said segmentation layer masks, said iterating comprising iteratively determining one or more features associated with one or more respective 3-D points that do not satisfy a threshold for mapping to one or more respective 2-D points, and iteratively adjusting one or more of said weighted values based on said determined one or more features, said iterating terminating after a plurality of iterations when a predetermined condition is satisfied.

20. The system of claim 19, wherein the executable instructions stored on the memory comprise all of said control instructions, said one or more processors being configured to execute said control instructions for generating said plurality of segmentation layer masks comprise said one or more processors being configured to:

execute control instructions for rendering said point cloud according to a camera model;

execute control instructions for obtaining a boolean image from said point cloud;

execute control instructions for identifying a sequence of boundary points in said point cloud;

execute control instructions for obtaining said polycurve from said sequence of boundary points; and

execute control instructions for refining said polycurve.

21. The system of claim 19, wherein the executable instructions stored on the memory comprise all of said control instructions, said one or more processors being configured to execute said control instructions for rendering said 3-D image comprise said one or more processors being configured to:

execute control instructions for forming, from each said mask, a 2-D mesh, and

execute control instructions for converting each said 2-D mesh into a 3-D planar mesh.

22. The system of claim 21, wherein said one or more processors being configured to execute said control instructions for converting each said 2-D mesh comprise said one or more processors being configured to execute control instructions for reverse projection.

23. The system of claim 21, wherein said one or more processors are further configured to:

execute control instructions for forming a 3-D mesh from said planar mesh.

24. The system of claim 23, wherein said one or more processors are configured to execute control instructions to construct said 2-D mesh using vertices, and wherein said one or more processors being configured to execute control instructions for forming a 2-D mesh comprise said one or more processors being configured to:

execute control instructions for lofting by replicating surface vertices of said 2-D mesh.

25. The system of claim 19, wherein the executable instructions stored on the memory comprise all of said control instructions, said one or more processors being configured to execute control instructions for generating a plurality of segmentation layer masks comprise said one or more processors being configured to execute control instructions for generating said plurality of segmentation layer masks using polycurve techniques.