Systems and methods for depth-assisted perspective distortion correction

• US 10,540,806 B2
• Filed: 02/19/2018
• Issued: 01/21/2020
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Yang, Samuel, Srikanth, Manohar, Lelescu, Dan, Venkataraman, Kartik
• Primary Examiner(s): He, Yingchun
• Application Number:
  US15/898,666
• Publication Number:
  US 20180240265A1
• Time in Patent Office: 701 Days
• Field of Search: None
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

Systems and methods for automatically correcting apparent distortions in close range photographs that are captured using an imaging system capable of capturing images and depth maps are disclosed. In many embodiments, faces are automatically detected and segmented from images using a depth-assisted alpha matting. The detected faces can then be re-rendered from a more distant viewpoint and composited with the background to create a new image in which apparent perspective distortion is reduced.

Claims

1. A camera system, comprising:

a plurality of cameras configured to capture image data from multiple viewpoints, wherein cameras in the plurality of cameras are situated in various positions corresponding to the multiple viewpoints;

a processor;

a memory containing an image processing application; and

a display;

wherein the image processing application stored in the memory directs the processor to;

obtain image data captured by the plurality of cameras from multiple viewpoints including an initial viewpoint;

generate depth map data indicating distances to faces within a scene from the initial viewpoint using information based on differences among the multiple viewpoints of the image data;

detect a face within the image data and a distance from the initial viewpoint to the face from the depth map data;

segment face image data from background image data using the depth map data;

rerender the face from a synthetic viewpoint by warping the segmented face image data based upon the depth map data to generate warped face image data, where the synthetic viewpoint is a greater distance from the face along an optical axis relative to the distance from the initial viewpoint to the face, and the warping corrects perspective distortion in the segmented face image data resulting from camera optics by;

selecting a desired viewpoint distance that specifies a distance from the synthetic viewpoint to the face;

projecting the segmented face image data to 3D locations based upon distances to pixels within the segmented face image data contained within the depth map data;

re-projecting the 3D locations to new 2D pixel locations based upon the desired viewpoint distance to create warped face image data; and

filling holes in the warped face image data;

combine the warped face image data with the background image data to create perspective distortion corrected image data; and

output the perspective distortion corrected image data to the display.

2. The camera system of claim 1, wherein the image processing application stored in the memory directs the processor to segment face image data from background image data using the depth map data by identifying a trimap for the image data using the image data and the depth map data.

3. The camera system of claim 2, wherein the trimap comprises pixels within the image data identified as:

foreground pixels;

background pixels;

or unknown pixels.

4. The camera system of claim 2, wherein the image processing application stored in the memory directs the processor to segment face image data from background image data using the depth map data by segmenting the face image data by generating an alpha matte using the trimap.

5. The camera system of claim 1, wherein the image processing application stored in the memory further directs the processor to warp the segmented face image data to create warped face image data by rescaling the warped face image data to correspond in size to the segmented face image data.

6. The camera system of claim 1, wherein the image processing application stored in the memory further directs the processor to inpaint the background image data.

7. The camera system of claim 1, wherein the image processing application stored in the memory directs the processor to inpaint the background image data by identifying holes in the background image data and performing interpolation to fill the holes.

8. The camera system of claim 1, wherein the memory further contains occluded pixel image data and the image processing application stored in the memory directs the processor to inpaint the background image data by identifying pixels in the occluded pixel image data corresponding to holes identified in the background image data and filling the holes with identified pixels.

9. The camera system of claim 1, wherein the image processing application stored in the memory further directs the processor to register the depth map image data to the perspective distortion corrected image data to generate a registered depth map.

10. The camera system of claim 9, the image processing application stored in the memory further directs the processor to store the perspective distortion corrected image data and a registered depth map in an image file.

11. The camera system of claim 1, wherein the image data and the depth map data are contained within an image file.

12. The camera system of claim 11, wherein the file further comprises occluded pixel image data.

13. A camera system, comprising:

a plurality of cameras configured to capture image data from multiple viewpoints, wherein cameras in the plurality of cameras are situated in various positions corresponding to the multiple viewpoints;

a processor;

a memory containing an image processing application; and

a display;

wherein the image processing application stored in the memory directs the processor to;

obtain image data captured by the plurality of cameras from multiple viewpoints including an initial viewpoint;

generate depth map data indicating distances to faces within a scene from the initial viewpoint using information based on differences among the multiple viewpoints of the image data;

detect a face within the image data and a distance from the initial viewpoint to the face from the depth map data;

segment face image data from background image data using the depth map data by;

identifying a trimap for the image data using the image data and the depth map data; and

generating an alpha matte using the trimap;

rerender the face from a synthetic viewpoint by warping the segmented face image data based upon the depth map data to generate warped face image data, where the synthetic viewpoint is a greater distance from the face along an optical axis relative to the distance from the initial viewpoint to the face, and the warping corrects perspective distortion in the segmented face image data resulting from camera optics by;

selecting a desired viewpoint distance that specifies a distance from the synthetic viewpoint to the face;

projecting the segmented face image data to 3D locations based upon distances to pixels within the segmented face image data contained within the depth map data;

re-projecting the 3D locations to new 2D pixel locations based upon the desired viewpoint distance to create warped face image data; and

filling holes in the warped face image data;

inpaint the background image data;

combine the warped face image data with the background image data to create perspective distortion corrected image data; and

output the perspective distortion corrected image data to the display.