Image processing including image correction

• US 8,848,063 B2
• Filed: 04/23/2012
• Issued: 09/30/2014
• Est. Priority Date: 05/02/2011
• Status: Expired due to Fees

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Jo, Kensei, Kaizu, Shun, Mitsunaga, Tomoo
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): TEJANO, DWIGHT ALEX C
• Application Number:
  US13/452,977
• Publication Number:
  US 20120281111A1
• Time in Patent Office: 890 Days
• Field of Search: 34820899-20813, 348/222.1, 348/229.1, 348362-368
• US Class Current: 348/208.6
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Long and short exposure time pixel information are input to pixel information. A long exposure time image set with the pixel values assuming all of the pixels have been exposed for a long time and a short exposure time image set with the pixel values assuming all of the pixels have been exposed for a short time are generated. A point spread function corresponding to the long exposure time image is computed as a long exposure time image PSF. A corrected image is generated using the short exposure time image, the long exposure time image, and the long exposure time image PSF. The corrected image is generated as a wide dynamic range image utilizing the pixel information for the long and short exposure time image. Utilizing the pixel information for the short exposure time image with little blurring, makes the corrected image a high quality corrected image with little blurring.

Claims

1. An image processing device, comprising:

an image capture element that outputs long exposure time pixel information and short exposure time pixel information based on image capture processing under a control that provides different exposure times for individual pixels; and

an image correction portion that inputs the pixel information that the image capture element has output and generates a corrected image by performing image stabilization and dynamic range expansion processing,

wherein the image correction portion

generates a long exposure time image in which pixel values have been set on the assumption that all of the pixels have been exposed for a long time and a short exposure time image in which the pixel values have been set on the assumption that all of the pixels have been exposed for a short time,

computes a point spread function (a PSF) that corresponds to the long exposure time image as a long exposure time image PSF, and

generates the corrected image by using the short exposure time image, the long exposure time image, and the long exposure time image PSF, and

wherein the image correction portion generates, based on the short exposure time image, a first estimated image for which it is assumed that exposure was performed for a long time, computes a first correction amount that makes a difference between the first estimated image and the long exposure time image smaller, and generates the corrected image by performing processing that adds the computed first correction amount to an initial image that has been generated based on the short exposure time image.

2. The image processing device according to claim 1,

wherein the image correction portion

computes a PSF that corresponds to the short exposure time image as a short exposure time image PSF, and

generates the corrected image by using the short exposure time image, the long exposure time image, the long exposure time image PSF, and the short exposure time image PSF.

3. The image processing device according to claim 1,

wherein the image correction portion

performs division processing that takes a result of a discrete Fourier transform that is based on a pixel value for a specific color that has been selected from the long exposure time image and divides it by a result of a discrete Fourier transform that is based on a pixel value for a specific color that has been selected from the short exposure time image, and

computes the long exposure time image PSF by performing an inverse discrete Fourier transform on a result of the division processing and performing processing on a result of the inverse discrete Fourier transform that performs noise removal and identifies a linking component that passes through an origin point.

4. The image processing device according to claim 1,

wherein the image correction portion

performs division processing that takes a result of a discrete Fourier transform that is based on a pixel value for a specific color that has been selected from the long exposure time image and divides it by a result of a discrete Fourier transform that is based on a pixel value for a specific color that has been selected from the short exposure time image, and

computes the long exposure time image PSF by performing an inverse discrete Fourier transform on a result of the division processing, performing processing on a result of the inverse discrete Fourier transform that performs noise removal and identifies a linking closed region that passes through an origin point, extending a length of a line segment that links the origin point and a center of gravity of the closed region to twice its original length, with the origin point at the center, and defining the extended line segment as the long exposure time image PSF.

5. The image processing device according to claim 1, wherein the image correction portion computes a PSF that corresponds to the short exposure time image as a short exposure time image PSF, computing the short exposure time image PSF by multiplying, by a factor that is equal to two times a ratio of an exposure time for a second exposure condition to an exposure time for a first exposure condition, a length of a line segment that links an origin point and a center of gravity of the long exposure time image PSF, with the origin point at the center.

6. The image processing device according to claim 1,

wherein the image correction portion includes

a first correction amount computation portion that, from a long exposure time image PSF that is computed based on the long exposure time image, from an initial image that is generated based on the short exposure time image, and from the long exposure time image, computes a first correction amount for the initial image,

a second correction amount computation portion that, from a short exposure time image PSF that is computed based on the short exposure time image, and from the initial image, computes a second correction amount for the initial image, and

an addition portion that adds the first correction amount and the second correction amount for the initial image.

7. The image processing device according to claim 6, wherein the first correction amount computation portion includes

a first estimated image computation portion that computes a first estimated image that is a result of estimating an image that is similar to the long exposure time image, based on the initial image and the long exposure time image PSF,

a subtraction portion that computes a first difference image that is a difference between the long exposure time image and the first estimated image, and

a first correction amount estimation portion that computes the first correction amount based on the first difference image and the long exposure time image PSF.

8. The image processing device according to claim 7,

wherein the first estimated image computation portion includes

a color-specific PSF computation portion that computes a first color-specific PSF that combines characteristics of the long exposure time image PSF for each color (phase) of an image capture surface of the image capture element,

a first convolution computation portion that performs a convolution computation for each color of an object pixel of the initial image using the first color-specific PSF, and

a saturation processing portion that takes a results image that has been output from the first convolution computation portion and outputs an image by replacing pixel values that are not less than a value that is equivalent to a saturated pixel value of the image capture element with the value that is equivalent to the saturated pixel value.

9. The image processing device according to claim 8,

wherein the first correction amount estimation portion includes

an inverse color-specific PSF computation portion that computes a first inverse color-specific PSF in which the long exposure time image PSF has been inverted symmetrically with respect to a point,

a second convolution computation portion that performs a convolution computation for each color of an object pixel of the first difference image using the first inverse color-specific PSF, and

a multiplication portion that performs a correction amount adjustment by multiplying a correction amount adjustment parameter times a results image that has been output from the second convolution computation portion.

10. The image processing device according to claim 1,

wherein the image correction portion includes

a demosaic processing portion that performs demosaic processing on the long exposure time image and the short exposure time image, and

the image correction portion generates the corrected image based on a demosaiced image that is a processing result of the demosaic processing portion.

11. An image processing method that is implemented in an image processing device, comprising:

outputting by an image capture element of long exposure time pixel information and short exposure time pixel information based on image capture processing under a control that provides different exposure times for individual pixels; and

generating by an image correction portion of a corrected image by inputting of the pixel information that the image capture element has output and performing of image stabilization and dynamic range expansion processing on the pixel information,

wherein the generating of the corrected image includes

generating of a long exposure time image in which pixel values have been set on the assumption that all of the pixels have been exposed for a long time and generating of a short exposure time image in which the pixel values have been set on the assumption that all of the pixels have been exposed for a short time,

computing of a point spread function (a PSF) that corresponds to the long exposure time image as a long exposure time image PSF,

generating of the corrected image by using the short exposure time image, the long exposure time image, and the long exposure time image PSF,

generating, based on the short exposure time image, a first estimated image for which it is assumed that exposure was performed for a long time,

computing a first correction amount that makes a difference between the first estimated image and the long exposure time image smaller, and

generating the corrected image by performing processing that adds the computed first correction amount to an initial image that has been generated based on the short exposure time image.

12. A non-transitory computer readable medium on which is stored a program that causes image processing to be performed in an image processing device, comprising:

outputting by an image capture element of long exposure time pixel information and short exposure time pixel information based on image capture processing under a control that provides different exposure times for individual pixels; and

generating by an image correction portion of a corrected image by inputting of the pixel information that the image capture element has output and performing of image stabilization and dynamic range expansion processing on the pixel information,

wherein the generating of the corrected image includes

generating of a long exposure time image in which pixel values have been set on the assumption that all of the pixels have been exposed for a long time and generating of a short exposure time image in which the pixel values have been set on the assumption that all of the pixels have been exposed for a short time,

computing of a point spread function (a PSF) that corresponds to the long exposure time image as a long exposure time image PSF,

generating of the corrected image by using the short exposure time image, the long exposure time image, and the long exposure time image PSF,

generating, based on the short exposure time image,

a first estimated image for which it is assumed that exposure was performed for a long time,

computing a first correction amount that makes a difference between the first estimated image and the long exposure time image smaller, and

generating the corrected image by performing processing that adds the computed first correction amount to an initial image that has been generated based on the short exposure time image.