System and methods for depth regularization and semiautomatic interactive matting using RGB-D images

• US 10,574,905 B2
• Filed: 10/01/2018
• Issued: 02/25/2020
• Est. Priority Date: 03/07/2014
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Srikanth, Manohar, Ramamoorthi, Ravi, Venkataraman, Kartik, Chatterjee, Priyam
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Bennett, Stuart D
• Application Number:
  US16/148,816
• Publication Number:
  US 20190037150A1
• Time in Patent Office: 512 Days
• Field of Search: None
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20096 Interactive definition of curve of interest
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Systems and methods in accordance with embodiments of this invention perform depth regularization and semiautomatic interactive matting using images. In an embodiment of the invention, the image processing pipeline application directs a processor to receive (i) an image (ii) an initial depth map corresponding to the depths of pixels within the image, regularize the initial depth map into a dense depth map using depth values of known pixels to compute depth values of unknown pixels, determine an object of interest to be extracted from the image, generate an initial trimap using the dense depth map and the object of interest to be extracted from the image, and apply color image matting to unknown regions of the initial trimap to generate a matte for image matting.

Claims

1. An array camera, comprising:

a plurality of cameras that capture images of a scene from different viewpoints;

memory containing an image processing pipeline application;

wherein the image processing pipeline application direct the processor to;

capture a set of images using a group of cameras from the plurality of cameras;

receive (i) an image comprising a plurality of pixel color values for pixels in the image and (ii) an initial depth map corresponding to the depths of the pixels within the image, wherein the initial depth map is generated using the set of images; and

regularize the initial depth map into a dense depth map using pixels for which depth is known to estimate depths of pixels for which depth is unknown, wherein regularizing the initial depth map into the dense depth map further comprises;

performing Laplacian matting to compute a Laplacian L;

obtain a binary confidence map C that indicates whether a depth at a given pixel is confident, where the confidence map C is obtained by a thresholded gradient of the image;

wherein the Laplacian matting is optimized by solving a reduced linear system for depth values of pixels that are marked as non-confident based on the confidence map C; and

using the dense depth map to perform image-based rendering.

2. The array camera of claim 1, wherein the image processing application further directs the processor to prune the Laplacian L, wherein pruning the Laplacian L comprises:

for each pair i,j of pixels in affinity matrix A, determine if i and j have depth differences beyond a threshold; and

if the difference is beyond the threshold, purge the pair i,j within the affinity matrix A.

3. The array camera of claim 2, wherein the image processing application further directs the processor to:

detect and correct depth bleeding across edges by computing a Laplacian residual R and removing incorrect depth values based on the Laplacian residual R.

4. The array camera of claim 3, wherein the computing the Laplacian residual R comprises computing R=Lz* where z* is the regularized depth map, wherein removing incorrect depth values comprises identifying regions where R>

0.

5. The array camera of claim 1, wherein a pixel for which depth is unknown is a pixel that has a confidence value below a particular threshold regarding the accuracy of the depth.

6. The array camera of claim 1, wherein the confidence map C is defined at texture and object edges within the image.

7. The array camera of claim 1, wherein the confidence map C is a sparse mn x mn diagonal matrix whose diagonal entries are binary confidence values.

8. The array camera of claim 3, wherein the image processing application further directs the processor to compute a new confidence map whenever the residual R is greater than a threshold.

9. The array camera of claim 1, wherein the image processing application further directs the processor to utilize the regularized dense depth map to perform depth-based segmentation that can be dilated to create a trimap.

10. The array camera of claim 1, wherein an unknown pixel'"'"'s depth is estimated as a weighted average of depths of the k-nearest super-pixels.

11. The array camera of claim 10, where the weights are derived as a function of distance of the RGBxy feature of the pixel from the super-pixel centroids.

12. The array camera of claim 1, wherein regularizing the initial depth map into the dense depth map comprises solving for depths in only unknown regions of the image.

13. The array camera of claim 1, wherein the Laplacian matting is optimized by solving a reduced linear system for alpha values only in unknown regions.

14. The array camera of claim 1, wherein the image processing application further directs the processor:

determine an object of interest to be extracted from the image;

generate an initial trimap using the dense depth map and the object of interest to be extracted from the image; and

apply color image matting to unknown regions of the initial trimap to generate a matte for image matting.

15. The array camera of claim 1, wherein the image processing pipeline application directs the processor to generate a trimap based on the regularized depth map.

16. The array camera of claim 1, wherein the Laplacian matting is a convention kNN-based (K nearest-neighbor) Laplacian that pairs similar pixels without regards to their depth when constructing the affinity matrix A and Laplacian L.

17. The array camera of claim 1, wherein the image processing application further directs the processor to detect and correct depth bleeding across edges.