Depth masks for image segmentation for depth-based computational photography

• US 10,554,956 B2
• Filed: 10/29/2015
• Issued: 02/04/2020
• Est. Priority Date: 10/29/2015
• Status: Active Grant

• Current Assignee: Dell Products LP (Dell Technologies Inc.)
• Original Assignee: Dell Products LP (Dell Technologies Inc.)
• Inventors: Das, Sujata
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US14/927,272
• Publication Number:
  US 20170127046A1
• Time in Patent Office: 1,559 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

A depth-based computational photography method includes recording a first image using a stereoscopic camera system, establishing a disparity map, via a processor, of the first image including a determination of pixel distance between features in the first image and the features in a second image, determining a histogram of pixel disparity values from the disparity map, and removing statistical outlier pixel disparity values. The method further includes identifying holes in disparity map of the first image and computationally shrinking holes in the disparity map via averaging disparity map values based on radially sampled areas proximate to the hole and sharpening histogram pixel disparity value peaks by shifting pixel disparity values in a close neighborhood range of a histogram pixel disparity value peak maximum closer to a pixel disparity value at the histogram pixel disparity value peak maximum.

Claims

1. A method of depth-based computational photography comprising:

recording a first image using a stereoscopic camera system;

establishing a disparity map, via a processor executing codes instructions, of the first image including a determination of pixel distance between features in the first image and the features in a second image;

identifying holes in disparity map of the first image and computationally shrinking at least one hole in the disparity map via applying a weighted average of radially sampled disparity map values proximate to the at least one hole, wherein sampled disparity map values for pixels closer to the at least one hole have greater weighting than sampled disparity map values for pixels further from the at least one hole;

removing noise in the disparity map via edge smoothing;

determining pixel depth values from the disparity map; and

encoding the pixel depth values into a depth mask for a composite plural image.

2. The method of claim 1 further comprising:

encoding depth values into a depth mask for the composite plural image such that more bits in the depth mask are assigned to foreground pixel depth values than to background pixel depth values.

3. The method of claim 1 further comprising:

encoding depth values into a depth mask using range-inverse encoding.

4. The method of claim 1 comprising:

applying a post-processing image filter to at least one image depth segmentation of the depth mask, wherein the depth mask correlates to a plurality of ranges of pixel depth values into image depth segmentations.

5. The method of claim 4 wherein applying the post-processing image filter includes an image filter selected from a group consisting of a black and white filter, a focus filter, an exposure level filter, and a color filter.

6. The method of claim 1 wherein encoding the disparity map into a depth mask includes image depth segmentation that statistically correlates a spectrum of pixel depth values to represent an object in the composite plural image.

7. The method of claim 1 comprising:

generating a histogram of pixel disparity values from the disparity map;

identifying a histogram peak disparity value maximum corresponding to an image object in the image;

shifting pixel disparity values in a close neighborhood range of the histogram peak disparity value maximum to equal the histogram peak disparity value maximum to sharpen the histogram peak corresponding to the image object; and

assigning the shifted pixel disparity values to pixels in the disparity map.

8. An information handling system comprising:

a stereoscopic camera system for recording a first image;

a processor operatively coupled to the stereoscopic camera system to establish a disparity map associated with the first image including a determination of pixel distance between features in the first image and the features in a second image;

the processor to execute instructions to identify holes in the disparity map associated with the first image and process the disparity map by computationally shrinking the holes using scan conversion applied to the disparity map with weighted, averaged disparity map values radially sampled from areas proximate to the hole, where disparity map values of pixels closer to the holes have a greater weighting value than disparity map values of pixels further from the holes in the weighted averaging;

the processor to encode the disparity map into a depth mask of a composite plural image; and

a memory operatively coupled to the processor, the memory for storing the composite plural image with the depth mask.

9. The information handling system of claim 8 further comprising:

the processor to construct the composite image including the depth mask having image depth segmentation with a plurality depth value ranges.

10. The information handling system of claim 8 further comprising:

the processor executing instructions to remove noise in the disparity map values via edge smoothing.

11. The information handling system of claim 8 further comprising:

the processor encoding depth values from a disparity map via range-inverse encoding such that more bits in the depth mask are assigned to a foreground pixel depth values than background pixel depth values.

12. The information handling system of claim 8 further comprising:

the processor applying a post-processing image filter to at least one image depth segmentation of the depth mask.

13. The information handling system of claim 12 wherein the post-processing image filter includes a focus filter applied to the at least one image depth segmentation.

14. The information handling system of claim 8 further comprising:

the processor determining a histogram of disparity values from the disparity map;

the processor to identify peak disparity map values and to shift disparity values within a limited range of a peak disparity value histogram maximum closer to the peak disparity value histogram maximum to sharpen image depth segmentation in a depth mask associated with one or more image objects.

15. A method of depth-based computational photography comprising:

recording a first image using a stereoscopic camera system;

establishing a disparity map, via a processor executing code instructions, of the first image including a determination of pixel distance between features in the first image and the features in a second image;

determining a histogram of pixel disparity values from the disparity map;

removing statistical outlier pixel disparity values;

identifying holes in disparity map of the first image and computationally shrinking holes in the disparity map via averaging disparity map values proximate to the hole;

determining whether any scanned disparity map values are different in value to those proximate to the hole;

rejecting scanned pixel disparity map values with different value to those proximate to the hole and determining a weighted average based on radially sampled areas of remaining scanned disparity map values proximate to the hole; and

sharpening histogram pixel disparity value peaks by shifting pixel disparity values in a close neighborhood range of a histogram pixel disparity value peak maximum closer to a pixel disparity value at the histogram pixel disparity value peak maximum for a composite plural image of the first and second images.

16. The method of claim 15 further comprising:

applying bilateral filtering to the disparity map values reduce pixel disparity value noise in the histogram.

17. The method of claim 15 further comprising:

determining depth values from the disparity map to construct the composite plural image with a depth mask.

18. The method of claim 17 further comprising:

encoding the depth values in the composite plural image using range-inverse encoding.

19. The method of claim 15 wherein, sharpening histogram pixel disparity value peaks includes shifting pixel disparity values in the close neighborhood range of the histogram pixel disparity value peak maximum to equal the pixel disparity value at the histogram pixel disparity value peak maximum.

20. The method of claim 17 comprising:

applying a post-processing filter to at least one image depth segmentation of the depth mask.