Image processing apparatus, imaging apparatus, solid-state imaging device, image processing method and program

• US 8,749,646 B2
• Filed: 09/01/2009
• Issued: 06/10/2014
• Est. Priority Date: 09/03/2008
• Status: Expired due to Fees

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Mitsunaga, Tomoo
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): BHUIYAN, FAYEZ A
• Application Number:
  US12/551,705
• Publication Number:
  US 20100053346A1
• Time in Patent Office: 1,743 Days
• Field of Search: 348/208.6
• US Class Current: 348/208.6
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/583 with different integration times
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors

Abstract

An image processing apparatus includes: a shaking estimating unit configured to estimate the shaking information of an image generated within a predetermined period of time; a short-time-exposure shaking correcting unit configured to correct the shaking of a plurality of short-time-exposure images generated due to intermittent exposure within the predetermined period of time based on the estimated shaking information; a long-time-exposure shaking correcting unit configured to correct the shaking of a long-time-exposure image generated due to consecutive exposure within the predetermined period of time based on the estimated shaking information; and an image synthesizing unit configured to synthesize the corrected short-time-exposure images and the corrected long-time-exposure image.

Claims

1. An image processing apparatus comprising:

a shaking estimating unit configured to;

estimate shaking information of an image generated within a predetermined period of time by comparing image data extracted from a plurality of short-time-exposure images generated due to intermittent exposure within the predetermined period of time, and

calculate a parametric curve that represents temporal change of image coordinates due to motion of an imaging apparatus within said predetermined period of time, as calculated shaking information; and

wherein said shaking estimating unit comprises;

a calculating unit configured to calculate discrete temporal change of each component of a coordinate transformation matrix between images making up said plurality of short-time-exposure images, and

a curvilinear fitting unit configured to fit the discrete temporal change of each component of said coordinate transformation matrix into said parametric curve;

a short-time-exposure shaking correcting unit configured to correct shaking of the plurality of short-time-exposure images based on said estimated shaking information;

a long-time-exposure shaking correcting unit configured to correct shaking of a long-time-exposure image generated due to consecutive exposure within said predetermined period of time, wherein the shaking of the long-time-exposure image is corrected based on said estimated shaking information generated by comparing image data extracted from the plurality of short-time-exposure images; and

an image synthesizing unit configured to synthesize said corrected short-time-exposure images and said corrected long-time-exposure image.

2. The image processing apparatus according to claim 1, said shaking estimating unit further comprising:

an image shaking correcting unit configured to correct shaking of said short-time-exposure images by aligning pixel positions of said plurality of short-time-exposure images to synthesize the short-time-exposure images based on said calculated shaking information;

a rendering unit configured to generate a point blurring function image at a plurality of target positions on said long-time-exposure images based on said calculated shaking information;

a point blurring function image correcting unit configured to correct said point blurring function image for each of said target positions based on an image adjacent to said target positions of said corrected short-time-exposure images, and an image adjacent to said target positions of said long-time-exposure image; and

a shaking information updating unit configured to update said estimated shaking information based on said calculated shaking information between images, said corrected point blurring function image, and said calculated shaking information.

3. The image processing apparatus according to claim 2:

wherein said calculating unit calculates matched points between images of said plurality of short-time-exposure images, thereby calculating said estimated shaking information between images;

and wherein said shaking estimating unit includes a reliable point extracting unit configured to extract a matched point where matching has been obtained from short-time-exposure images of said plurality of short-time-exposure images of said matched points, as a reliable point;

and wherein said image shaking correcting unit aligns the pixel positions of said plurality of short-time-exposure images based on said calculated shaking information regarding a region adjacent to said extracted reliable point to synthesize the short-time-exposure images, thereby correcting the shaking of said short-time-exposure images;

and wherein said rendering unit generates a point blurring function image at the position of said reliable point on said long-time-exposure image based on said calculated shaking information;

and wherein said point blurring function image correcting unit corrects said point blurring function image based on an image adjacent to said reliable point of said corrected short-time-exposure images, and an image adjacent to said reliable point of said long-time-exposure image.

4. The image processing apparatus according to claim 1, wherein said short-time-exposure shaking correcting unit aligns pixel positions of images making up said plurality of short-time-exposure images based on said estimated shaking information to synthesize the plurality of short-time-exposure images, thereby correcting the shaking of said short-time-exposure images.

5. The image processing apparatus according to claim 1, said long-time-exposure shaking correcting unit comprising:

a rendering unit configured to generate point blurring function images at a plurality of target positions on said long-time-exposure image based on said estimated shaking information; and

a deconvolution unit configured to subject said long-time-exposure image to a deconvolution process for each of said target positions based on each of said generated point blurring function images.

6. The image processing apparatus according to claim 1, wherein said image synthesizing unit adds a pixel value of said corrected short-time-exposure images and a pixel value of said corrected long-time-exposure image for each pixel, employs the inverse property of synthetic property between incident light intensity and pixel value response property according to short-time exposure, and incident light intensity and pixel value response property according to long-time exposure to correct said pixel values added for each pixel, thereby generating a synthetic image between said corrected short-time-exposure images and said corrected long-time-exposure image.

7. The image processing apparatus according to claim 1:

wherein said short-time-exposure images are mosaic images output from a solid-state imaging device;

and wherein said long-time-exposure image is a mosaic image output from said solid-state imaging device;

and wherein said image synthesizing unit disposes a pixel of said corrected short-time-exposure images, and a pixel of said corrected long-time-exposure image so as to obtain a mosaic image having the same pixel array as the mosaic image output from said solid-state imaging device, thereby executing a demosaic process regarding the mosaic image thereof.

8. An imaging apparatus comprising:

a solid-state imaging device where a first pixel used for generating a long-time-exposure image by consecutive exposure within a predetermined period of time, and a second pixel used for generating a plurality of short-time-exposure images by intermittent exposure within said predetermined period of time are arrayed on the imaging surface;

a shaking estimating unit configured to;

compare image data extracted from the plurality of short-time-exposure images in order to estimate shaking information within said predetermined period of time, and

calculate a parametric curve that represents temporal change of image coordinates due to motion of an imaging apparatus within said predetermined period of time, as calculated shaking information, and

wherein said shaking estimating unit comprises;

a calculating unit configured to calculate discrete temporal change of each component of a coordinate transformation matrix between images making up said plurality of short-time-exposure images, and

a curvilinear fitting unit configured to fit the discrete temporal change of each component of said coordinate transformation matrix into said parametric curve;

a short-time-exposure shaking correcting unit configured to correct shaking of said short-time-exposure images based on said estimated shaking information;

a long-time-exposure shaking correcting unit configured to correct shaking of a long-time-exposure image generated by using said first pixel, wherein the shaking of the long-time-exposure image is corrected based on said estimated shaking information generated by comparing image data extracted from the plurality of short-time-exposure images; and

an image synthesizing unit configured to synthesize said corrected short-time-exposure images and said corrected long-time-exposure image.

9. The imaging apparatus according to claim 8, wherein, with said solid-state imaging device, a first pixel line where said first pixels are arrayed in the horizontal direction, and a second pixel line where said second pixels are arrayed in said horizontal direction are alternately disposed in the vertical direction.

10. The imaging apparatus according to claim 9, wherein, with said solid-state imaging device, in order for pixels having first spectral sensitivity to be disposed in a checkered pattern, said pixels having first spectral sensitivity are disposed on said first pixel line and said second pixel line alternately, and in order for pixels having second spectral sensitivity and pixels having third spectral sensitivity to be disposed alternately between said pixels having first spectral sensitivity on said first pixel line and said second pixel line, said pixels having second spectral sensitivity according to said first pixels, and said pixels having second spectral sensitivity according to said second pixels, are arrayed alternately in an oblique direction, and also said pixels having third spectral sensitivity according to said first pixels, and said pixels having third spectral sensitivity according to said second pixels are arrayed alternately in said oblique direction.

11. The imaging apparatus according to claim 8, wherein, with said solid-state imaging device, said first pixels and said second pixels are disposed in a checkered pattern.

12. The imaging apparatus according to claim 11, wherein, with said solid-state imaging device, an array made up of said pixels having first spectral sensitivity, said pixels having second spectral sensitivity, and said pixels having third spectral sensitivity according to said first pixels is a Bayer array inclined obliquely, and an array made up of said pixels having first spectral sensitivity, said pixels having second spectral sensitivity, and said pixels having third spectral sensitivity according to said second pixels is a Bayer array inclined obliquely.

13. The imaging apparatus according to claim 11, wherein, with said solid-state imaging device, on the upper side of a horizontal line made up of said first pixels and said second pixels arrayed alternately in the horizontal direction, a line transfer signal line that supplies a control signal used for executing readout by consecutive exposure within said predetermined period of time in said horizontal direction, and a line rest signal line are disposed and connected to said first pixels included in said horizontal line, and on the lower side of said horizontal line, a line transfer signal line that supplies a control signal used for executing readout sequentially by intermittent exposure within said predetermined period of time in said horizontal direction, and a line rest signal line are disposed and connected to said second pixels included in said horizontal line.

14. The imaging apparatus according to claim 8, wherein, with said solid-state imaging device, a first pixel group where said three first pixels arrayed in the horizontal direction, and said three first pixels arrayed in the vertical direction are connected in a staircase pattern, and a second pixel group where said three second pixels arrayed in the horizontal direction, and said three second pixels arrayed in the vertical direction are connected in a staircase pattern, are disposed alternately in said horizontal direction.

15. The imaging apparatus according to claim 14, wherein, with said solid-state imaging device, the array made up of said pixels having first spectral sensitivity, said pixels having second spectral sensitivity, and said pixels having third spectral sensitivity, is a Bayer array.

16. An image processing method comprising the steps of:

comparing of image data extracted from a plurality of short-time-exposure images generated due to intermittent exposure within a predetermined period of time in order to estimate shaking information of an image generated within the predetermined period of time;

wherein said comparing includes;

calculating a parametric curve that represents temporal change of image coordinates due to motion of an imaging apparatus within said predetermined period of time,

calculating discrete temporal change of each component of a coordinate transformation matrix between images making up said plurality of short-time-exposure images, and

fitting the discrete temporal change of each component of said coordinate transformation matrix into said parametric curve;

correcting of shaking of the plurality of short-time-exposure images generated due to intermittent exposure within said predetermined period of time based on said estimated shaking information;

correcting of shaking of a long-time-exposure image generated due to consecutive exposure within said predetermined period of time, wherein the shaking of the long-time-exposure image is corrected based on said estimated shaking information generated by comparing image data extracted from the plurality of short-time-exposure images; and

synthesizing of said corrected short-time-exposure images and said corrected long-time-exposure image.

17. A non-transitory, computer-readable storage medium storing a program that, when executed by a computer, causes the computer to perform a method, the method comprising:

comparing of image data extracted from a plurality of short-time-exposure images generated due to intermittent exposure within a predetermined period of time in order to estimate shaking information of an image generated within the predetermined period of time;

wherein said comparing includes;

calculating a parametric curve that represents temporal change of image coordinates due to motion of an imaging apparatus within said predetermined period of time,

calculating discrete temporal change of each component of a coordinate transformation matrix between images making up said plurality of short-time-exposure images, and

fitting the discrete temporal change of each component of said coordinate transformation matrix into said parametric curve;

correcting of shaking of the plurality of short-time-exposure images generated due to intermittent exposure within said predetermined period of time based on said estimated shaking information;

correcting of shaking of a long-time-exposure image generated due to consecutive exposure within said predetermined period of time, wherein the shaking of the long-time-exposure image is corrected based on said estimated shaking information generated by comparing image data extracted from the plurality of short-time-exposure images; and

synthesizing of said corrected short-time-exposure images and said corrected long-time-exposure image.