COLOR-BASED DEPTH SMOOTHING OF SCANNED 3D MODEL TO ENHANCE GEOMETRY IN 3D PRINTING

• US 20170061701A1
• Filed: 08/26/2015
• Published: 03/02/2017
• Est. Priority Date: 08/26/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Mittal, Amit, Chopra, Harsh Vardhan, Goel, Naveen Prakash, Hemani, Mayur
• Granted Patent: US 9,818,232 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2200/12 involving antialiasing
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/005 Retouching; Inpainting; Scratch removal detecting, correction, reducing or removing defects, e.g. non-responsive pixels of solid state image sensors H04N25/68, scratch removal for cinematographic films scanned by electronic image sensor H04N5/253
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

Methods and systems for correcting depth irregularities in a three-dimensional scanned model. In particular, one or more embodiments obtain depth data and color data for an object using a depth scanner and a color scanner, respectively. One or more embodiments identify, based on edge information in the color data, regions that have no depth data that are part of the object. One or more embodiments then correct the depth frame by assigning depth data to the identified regions based on a gradient of depth data from areas near the identified regions. The methods and systems use the corrected depth data to generate a corrected three-dimensional model of the object.

Claims

1. A method of correcting depth errors in a three-dimensional scanned image, comprising:

obtaining, by at least one processor, a scan of an object with a depth frame and a color frame for the object;

identifying, by the at least one processor, one or more regions in the depth frame with missing or irregular depth data;

determining, by the at least one processor, whether the one or more regions in the depth frame with missing or irregular depth data should have depth data using the color data from the color frame;

correcting, by the at least one processor, the depth frame by assigning depth data to the one or more regions that should have depth data based on depth data from an area surrounding the one or more regions that should have depth data; and

generating, by the at least one processor, a smoothed three-dimensional model of the object by combining the corrected depth frame and the color frame.

2. The method as recited in claim 1, wherein correcting the depth frame by assigning depth data to the one or more regions that should have depth data comprises:

determining gradients of the area surrounding the one or more regions that should have depth data; and

applying a surface-fitting algorithm using the determined gradients to calculate the depth data to assign to the one or more regions that should have depth data.

3. The method as recited in claim 2, wherein determining gradients of the area surrounding the one or more regions that should have depth data comprises:

calculating for each point (x,y,z) inside a first region that should have depth data, at a distance e from a boundary of the first region that should have depth data;

4. The method as recited in claim 1, wherein identifying one or more regions in the depth frame with missing or irregular depth data using the color data from the color frame comprises generating a binary mask from the depth frame that indicates one or more regions of the scan that have regular depth information and the one or more regions that lack regular depth information.

5. The method as recited in claim 4, wherein identifying one or more regions in the depth frame with missing or irregular depth data comprises:

identifying portions of the binary mask with depth data marked as zero;

identifying connected portions of the binary mask with depth data marked as zero.

6. The method as recited in claim 4, further comprising:

detecting an outer boundary of the object by identifying a threshold difference in depth values between a background of the scan and the outer boundary of the object; and

classifying any regions in the depth frame with missing or irregular depth data that are outside of the outer boundary of the object as not being a region that should have depth data.

7. The method as recited in claim 6, further comprising:

performing edge detection on the color frame of the scan to identify edges in the color frame;

comparing the identified edges to a region with missing or irregular depth data that is inside the outer boundary of the object to determine if one or more edges of the identified edges run parallel to a boundary of the region with missing or irregular depth data that is inside the outer boundary of the object; and

if no edges run parallel to the boundary of the region with missing or irregular depth data, classifying the region with missing or irregular depth data as being the a region that should have depth data;

if edges run parallel to the boundary of the region with missing or irregular depth data, classifying the region with missing or irregular depth data as not being a region that should have depth data.

8. The method as recited in claim 7, wherein performing edge detection on the color frame of the scan to identify edges in the color frame comprises applying an edge detection algorithm to the color frame.

9. The method as recited in claim 1, further comprising identifying depth data for the area surrounding the one or more regions that should have depth data by:

identifying a plurality of pixels in the area surrounding the one or more regions; and

determining a plurality of depth values corresponding to the plurality of pixels from the depth frame.

11. The method as recited in claim 9, wherein calculating second depth data for the region missing depth data using the determined gradients of the first depth data comprises applying a surface-fitting algorithm using the determined gradients of the first depth data to generate the second depth data.

13. The method as recited in claim 11, wherein identifying that the region missing depth data is an error created during generation of the three-dimensional scan of the object comprises:

detecting an outer boundary of the object by identifying a difference in depth values in the depth frame between a background of the three-dimensional scan and the object;

identifying that the region missing depth data is within the outer boundary of the object;

identifying edges in the color frame; and

determining that the identified edges do not correspond to a boundary of the region missing depth data.

10. A method of correcting depth errors in a three-dimensional scanned image, comprising:

generating, by at least one processor, a three-dimensional scan of an object, the scan comprising a depth frame of the object and a color frame of the object;

identifying, by the at least one processor and based on color data from the color frame, a region of the object for which the depth frame is missing depth data;

determining gradients of first depth data of one or more areas surrounding the region missing depth data; and

calculating, by the at least one processor, second depth data for the region missing depth data using the determined gradients of the first depth data.

12. The method as recited in claim 10, further comprising identifying that the region missing depth data is an error created during generation of the three-dimensional scan of the object.

14. The method as recited in claim 12, wherein determining that the identified edges do not correspond to the boundary of the region missing depth data comprises determining that no edges in the color frame run parallel to the boundary of the region.

15. A system for correcting depth errors in a three-dimensional scanned image, comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the system to;

capture a scan of an object to obtain a depth frame for the object and a color frame for the object;

identify one or more regions in the depth frame with missing or irregular depth data;

determine whether the one or more regions in the depth frame with missing or irregular depth data should have depth data using the color data from the color frame;

correct the depth frame by assigning depth data to the one or more regions that should have depth data based on depth data from an area surrounding the one or more regions that should have depth data; and

generate a smoothed three-dimensional model of the object by combining the corrected depth frame and the color frame.

16. The system as recited in claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to correct the depth frame by assigning depth data to the one or more regions that should have depth data by:

determining gradients of the area surrounding the one or more regions that should have depth data; and

applying a surface-fitting algorithm using the determined gradients to calculate the depth data to assign to the one or more regions that should have depth data.

17. The system as recited in claim 16, wherein the instructions, when executed by the at least one processor, cause the system to determine gradients of the area surrounding the one or more regions that should have depth data by:

calculating for each point (x,y,z) inside a first region that should have depth data, at a distance e from a boundary of the first region that should have depth data;

18. The system as recited in claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to detect that a region from the one or more regions is located within the object by:

detecting an outer boundary of the object based on depth data from the depth frame; and

determining that the region is within the outer boundary of the object based on a position of the region relative to the outer boundary.

19. The system as recited in claim 18, further comprising instructions that, when executed by the at least one processor, cause the system to:

generate a binary mask from the depth frame that indicates one or more regions of the scan that have regular depth information and the one or more regions that lack regular depth information;

identify portions of the binary mask with depth data marked as zero;

identify connected portions of the binary mask with depth data marked as zero.

20. The system as recited in claim 19, further comprising instructions that, when executed by the at least one processor, cause the system to:

perform edge detection on the color frame of the scan to identify edges in the color frame;

compare the identified edges to a region with missing or irregular depth data that is inside the outer boundary of the object to determine if one or more edges of the identified edges run parallel to a boundary of the region with missing or irregular depth data that is inside the outer boundary of the object; and

if no edges run parallel to the boundary of the region with missing or irregular depth data, classify the region with missing or irregular depth data as being the a region that should have depth data;

if edges run parallel to the boundary of the region with missing or irregular depth data, classify the region with missing or irregular depth data as not being a region that should have depth data.