Image processing including image correction

• US 9,560,290 B2
• Filed: 08/28/2014
• Issued: 01/31/2017
• Est. Priority Date: 05/02/2011
• Status: Active Grant

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Jo, Kensei, Kaizu, Shun, Mitsunaga, Tomoo
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): TEJANO, DWIGHT ALEX C
• Application Number:
  US14/472,153
• Publication Number:
  US 20140368697A1
• Time in Patent Office: 887 Days
• Field of Search: 348/222.1, 348/229.1, 348362-368
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Long and short exposure time pixel information are input to pixel information. A long exposure time image set with the pixel values assuming all of the pixels have been exposed for a long time and a short exposure time image set with the pixel values assuming all of the pixels have been exposed for a short time are generated. A point spread function corresponding to the long exposure time image is computed as a long exposure time image PSF. A corrected image is generated using the short exposure time image, the long exposure time image, and the long exposure time image PSF. The corrected image is generated as a wide dynamic range image utilizing the pixel information for the long and short exposure time image. Utilizing the pixel information for the short exposure time image with little blurring, makes the corrected image a high quality corrected image with little blurring.

Claims

1. An image processing device, comprising:

an image capture unit configured to generate an initial output image, wherein the image capture unit comprises a plurality of pixels, and wherein the plurality of pixels include a first group of pixels with a first exposure time and a second group of pixels with a second exposure time;

an interpolation processing unit configured to generate at least a first interpolated image based on the first group of pixels with the first exposure time and a second interpolated image based on the second group of pixels with the second exposure time;

a bit shift unit configured to generate at least two gradation images, which include a first gradation image corresponding to the first interpolated image and a second gradation image corresponding to the second interpolated image;

a correction amount computation unit configured to compute at least two correction amounts,

wherein the at least two correction amounts include a first correction amount and a second correction amount, and

wherein the first correction amount is computed based, at least in part, on the first gradation image and the initial output image, and the second correction amount is computed based, at least in part, on the second gradation image and the initial output image; and

an addition unit configured to add the at least two correction amounts to the initial output image to produce a corrected output image.

2. The image processing device of claim 1, further comprising a noise reduction unit configured to reduce noise of data associated with the second interpolated image to generate the initial output image.

3. The image processing device of claim 1, wherein the first correction amount is further based, at least in part, on a first point spread function (PSF) corresponding to the first interpolated image, and wherein the second correction amount is further based, at least in part, on a second point spread function (PSF) corresponding to the second interpolated image.

4. The image processing device of claim 1, wherein the image capture unit is configured to control the first and second groups of pixels such that the first exposure time and the second exposure time start at different times and end at the same time.

5. The image processing device of claim 1, wherein the interpolation processing unit, to generate the first interpolated image, is configured to:

generate portions of the first interpolated image corresponding to positions of the first group of pixels based on pixel values of the first group of pixels;

generate interpolated pixel values by interpolation of the pixel values of the first group of pixels; and

generate portions of the first interpolated image corresponding to positions of the second group of pixels based, at least in part, on the interpolated pixel values.

6. The image processing device of claim 5, wherein interpolation of the pixel values of the first group of pixels further comprises interpolation of the pixel values of the first group of pixels by filters or linear interpolation of the pixel values of the first group of pixels.

7. The image processing device of claim 6, wherein generation of the interpolated pixel values comprises assignment of weights to the pixel values of the first group of pixels based, at least in part, on locations of edges in the first interpolated image.

8. An image processing method, comprising:

generating, with a plurality of pixels, an initial output image, the plurality of pixels including a first group of pixels with a first exposure time and a second group of pixels with a second exposure time;

generating at least a first interpolated image based on the first group of pixels with the first exposure time and a second interpolated image based on the second group of pixels with the second exposure time;

generating at least two gradation images by bit shifting at least the first and second interpolated images, wherein the at least two gradation images include a first gradation image corresponding to the first interpolated image and a second gradation image corresponding to the second interpolated image;

determining at least two correction amounts including a first correction amount and a second correction amount, wherein the first correction amount is determined based, at least in part, on the first gradation image and the initial output image, and the second correction amount is determined based, at least in part, on the second gradation image and the initial output image; and

adding the at least two correction amounts to the initial output image to produce a corrected output image.

9. The image processing method of claim 8, further comprising:

generating the initial output image by reducing noise of data associated with the second interpolated image.

10. The image processing method of claim 8, wherein the first correction amount is further determined based, at least in part, on a first point spread function (PSF) corresponding to the first interpolated image, and wherein the second correction amount is further determined based, at least in part, on a second point spread function (PSF) corresponding to the second interpolated image.

11. The image processing method of claim 8, wherein generating the initial output image comprises controlling the first and second groups of pixels such that the first exposure time and the second exposure time start at different times and end at the same time.

12. The image processing method of claim 8, wherein generating the first interpolated image based on the first group of pixels with the first exposure time comprises:

generating portions of the first interpolated image corresponding to positions of the first group of pixels based on pixel values of the first group of pixels;

generating interpolated pixel values by interpolating the pixel values of the first group of pixels; and

generating portions of the first interpolated image corresponding to positions of the second group of pixels based, at least in part, on the interpolated pixel values.

13. The image processing method of claim 12, wherein interpolating the pixel values of the first group of pixels further comprises interpolating the pixel values of the first group of pixels by filters or linearly interpolating the pixel values of the first group of pixels.

14. The image processing method of claim 13, wherein generating the interpolated pixel values comprises assigning weights to the pixel values of the first group of pixels based, at least in part, on locations of edges in the first interpolated image.

15. A camera, comprising:

an image processing device including;

an image capture unit configured to generate an initial output image, wherein the image capture unit comprises a plurality of pixels, and wherein the plurality of pixels include a first group of pixels with a first exposure time and a second group of pixels with a second exposure time,

an interpolation processing unit configured to generate at least a first interpolated image based on the first group of pixels with the first exposure time and a second interpolated image based on the second group of pixels with the second exposure time,

a bit shift unit configured to generate at least two gradation images, which include a first gradation image corresponding to the first interpolated image and a second gradation image corresponding to the second interpolated image,

a correction amount computation unit configured to compute at least two correction amounts,

wherein the at least two correction amounts include a first correction amount and a second correction amount, and

wherein the first correction amount is computed based, at least in part, on the first gradation image and the initial output image, and the second correction amount is computed based, at least in part, on the second gradation image and the initial output image, and

an addition unit configured to add the at least two correction amounts to the initial output image to produce a corrected output image; and

a display device configured to display the corrected output image.

16. The camera of claim 15, wherein the camera comprises a digital still camera.

17. The camera of claim 15, wherein the camera comprises a video camera.

18. The camera of claim 15, wherein the image capture unit includes a CMOS image sensor that comprises the plurality of pixels.