Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information

• US 9,955,070 B2
• Filed: 09/07/2016
• Issued: 04/24/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited (Adeia Inc.)
• Inventors: Lelescu, Dan, Duong, Thang
• Primary Examiner(s): CUTLER, ALBERT H
• Application Number:
  US15/258,947
• Publication Number:
  US 20170070673A1
• Time in Patent Office: 594 Days
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/581 acquired simultaneously
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

Systems and methods for synthesizing high resolution images using image deconvolution and depth information in accordance embodiments of the invention are disclosed. In one embodiment, an array camera includes a processor and a memory, wherein an image deconvolution application configures the processor to obtain light field image data, determine motion data based on metadata contained in the light field image data, generate a depth-dependent point spread function based on the synthesized high resolution image, the depth map, and the motion data, measure the quality of the synthesized high resolution image based on the generated depth-dependent point spread function, and when the measured quality of the synthesized high resolution image is within a quality threshold, incorporate the synthesized high resolution image into the light field image data.

Claims

1. An array camera, comprising:

a processor; and

a memory connected to the processor and configured to store an image deconvolution application;

wherein the image deconvolution application configures the processor to;

obtain light field image data, where the light field image data comprises an image having a plurality of pixels, a depth map, and metadata describing the motion associated with a capturing device that captured the light field image data;

determine motion data based on the metadata contained in the light field image data;

generate a depth-dependent point spread function based on the image, the depth map, and the motion data, where the depth-dependent point spread function describes the blurriness of points within the image based on the motion of the capturing device and the depth of the pixels described in the depth map;

measure the quality of the image based on the generated depth-dependent point spread function;

when the measured quality of the image is within a quality threshold, incorporate the image into the light field image data; and

when the measured quality of the image is outside the quality threshold;

determine updated motion data based on the measured quality and the depth-dependent point spread function;

generate an updated depth-dependent point spread function based on the updated motion data; and

synthesize a new image based on the updated depth-dependent point spread function.

2. The array camera of claim 1, further comprising an array camera module comprising an imager array having multiple focal planes and an optics array configured to form images through separate apertures on each of the focal planes;

wherein the array camera module is configured to communicate with the processor; and

wherein the obtained light field image data comprises image data captured by the imager array.

3. The array camera of claim 1, wherein the metadata describing the motion associated with the capturing device that captured the light field image data is obtained from an inertial movement sensor that captures motion information related to rotation and translation of the array camera.

4. The array camera of claim 3, wherein the inertial measurement sensor is selected from the group consisting of a gyroscope and a microelectromechanical system (MEMS) accelerometer, an accelerometer, and a magnetometer.

5. The array camera of claim 3, wherein the inertial measurement sensor is integrated into the array camera.

6. The array camera of claim 1, wherein the metadata further comprises a timestamp associated with the motion.

7. The array camera of claim 1, wherein the quality threshold is dependent on a sharpness of the image.

8. The array camera of claim 1, wherein the image deconvolution application further configures the processor to:

apply ego-motion techniques to estimate motion data by;

detecting features in a first image and matching those features to a second image; and

generate an optical flow describing the estimate of the motion of the array camera.

9. The array camera of claim 1, wherein determining motion data further comprises correlating motion information along with time information contained in the light field image data, wherein the correlation describes how quickly the array camera moved as packets of image data included in the light field image data were read from cameras in the array camera.

10. The array camera of claim 1, wherein the image deconvolution application configures the processor to calibrate the array camera by performing intrinsic and extrinsic camera calibration.

11. The array camera of claim 1, wherein the point spread function includes a blur kernel that describes the response of the array to a point object.

12. The array camera of claim 11, wherein the blur kernel is generated using a pixel path calculated for at least one pixel.

13. The array camera of claim 1, wherein a point p in a scene is located at a location p(x, y, z), where (x, y, z) represents a three-dimensional location of the point in the scene, wherein the image deconvolution application configures the processor to apply an a priori rigid motion transformation having rotation R and translation T to a point p in a light field that results in the point p moving to:

M_R,T(p)=(RT)p

and the imaged position q=P(x,y,z) move to;

PM_R,T(p)

where P is the projective transformation of the array camera capturing the light field.

14. The array camera of claim 1, wherein a point p in a scene is located at a location p(x, y, z), where (x, y, z) represents a three-dimensional location of the point in the scene, and wherein an a priori rigid motion transformation M_R,T is represented by a rotation R_n and a translation T_n for a time sampling period N=1, . . . ,n₁ wherein the image deconvolution application configures the processor to apply an a priori rigid motion transformation to pixel p over the time sampling period by calculating pixel path:

q_n=PM_n(p)

where;

M_n(p)=P(R_nT_n. . . R₁T₁).

15. The array camera of claim 1, wherein the point spread functions contain a blur kernel that describes the response of the array camera to a point object, wherein the blur kernel K for a pixel path q_n for a time sampling period N=1, . . . , n is generated using the formula:

16. A method for performing image deconvolution, comprising:

obtain light field image data using an array camera having a plurality of cameras, wherein the light field image data comprises an image having a plurality of pixels, a depth map, and metadata describing the motion associated with the array camera that captured the light field image data;

determine motion based on the metadata contained in the light field image data;

generate a depth-dependent point spread function based on the image, the depth map, and the motion data, where the depth-dependent point spread function describes a blurriness of points within the image based on the motion of the array camera and the depth of the pixels described in the depth map;

measure the quality of the image based on the generated depth-dependent point spread function;

when the measured quality of the image is within a quality threshold, incorporate the image into the light field image data; and

when the measured quality of the image is outside the quality threshold;

determine updated motion data based on the measured quality and the depth-dependent point spread function;

generate an updated depth-dependent point spread function based on the updated motion data; and

synthesize a new image based on the updated depth-dependent point spread function.

17. The method of claim 16, wherein the metadata describing the motion associated with the array camera that captured the light field image data is obtained from an inertial movement sensor that captures motion information related to rotation and translation of the array camera.

18. The method of claim 17, wherein the inertial measurement sensor is selected from the group consisting of a gyroscope and a microelectromechanical system (MEMS) accelerometer, an accelerometer, and a magnetometer.

19. The method of claim 16, wherein determining motion data further comprises correlating motion information along with time information contained in the light field image data, wherein the correlation describes how quickly the array camera moved as packets of image data included in the light field image data were read from cameras in the array camera.

20. The method of claim 16, wherein the image deconvolution application configures the processor to calibrate the array camera by performing intrinsic and extrinsic camera calibration.