Method and apparatus for electronically stabilizing digital images

• US 7,557,832 B2
• Filed: 08/12/2005
• Issued: 07/07/2009
• Est. Priority Date: 08/12/2005
• Status: Active Grant

• Current Assignee: Florent Maurice René Staley, Ivan Kisel, Volker Lindenstruth
• Original Assignee: Florent Maurice René Staley, Ivan Kisel, Volker Lindenstruth
• Inventors: Lindenstruth, Volker, Kisel, Ivan, Staley, Florent Maurice René
• Primary Examiner(s): Ometz; David L
• Assistant Examiner(s): Le; Quang V
• Application Number:
  US11/202,810
• Publication Number:
  US 20070035630A1
• Time in Patent Office: 1,425 Days
• Field of Search: 348/208.99, 348/208.4, 348294-297, 348/310
• US Class Current: 348/208.99
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

An electronic image stabilizer in a digital camera compensates for camera motion-induced blurriness by segmenting exposure times into multiple shorter exposure times and summing the individual pixels from successive frames after applying an appropriate motion correction. Motion is detected by computing the correlation function between successive images, and compensation is applied by maximizing the correlation function. This avoids the need for mechanical stabilization devices in order to detect or correct the motion as is done in prior art. This method further enables the detection of moving objects in a still background, and correction of blurriness images due to such motion.

Claims

1. A method of compensating for motion during an exposure time in a digital imaging system, comprising

dividing said exposure time into a plurality of shorter successive time intervals, each time interval selected such that image data for the time interval has negligible motion induced blurriness;

sensing image data for each of said time intervals, the image data corresponding to individual images;

determining a lateral offset between successive images; and

integrating said images to compensate for said lateral offset.

2. The method of claim 1, wherein said sensing comprises sensing said image data with a sensor having an array of pixels, and said dividing comprises selecting said time intervals such that an amount an object in an image moves on said sensor during a time interval is less than a pixel resolution of said sensor.

3. The method of claim 2, wherein said dividing comprises reading data out from said sensor at a high frame rate selected to provide said short time intervals.

4. The method of claim 2, wherein said integrating comprises interpolating between pixels of the sensor.

5. The method of claim 1, wherein said determining comprises determining an offset vector corresponding to the lateral offset between two successive images, and said integrating comprises using said vector to combine said images to maximize the correlation between the images.

6. The method of claim 5, wherein said determining comprises constructing horizontal and vertical intensity histograms of successive images, identifying areas of high contrast from first derivatives of said histograms, and determining maxima in said histograms to locate said lateral offset.

7. The method of claim 1, wherein said determining comprises computing a correlation between said images, and said integrating comprises combining each individual image with a single integrated image formed from a plurality of preceding images to maximize said correlation.

8. The method of claim 6, wherein said determining comprises summing one or more of color or brightness histograms of said images over a plurality of pixels.

9. The method of claim 1, wherein said integrating comprises integrating multiple images corresponding to multiple time intervals to increase the signal-to-noise ratio in the compensated image.

10. The method of claim 1 further comprising identifying a moving object with respect to a background within said images, and said determining comprises determining another offset between the positions of the moving object in successive images, and said integrating comprises integrating said images to compensate for said other offset.

11. The method of claim 10, wherein said identifying comprises identifying an area of the images corresponding to the moving object, and said determining comprises determining said other offset between images in said area.

12. The method of claim 10, wherein said lateral offset is due to imaging system motion, and said integrating comprises producing images compensated for said system motion, and wherein said other offset due to said object motion is determined from said compensated images.

13. The method of claim 1 further comprising storing image data read out from a sensor in a buffer, processing said stored image data to determine said offset, and storing integrated data corresponding to a compensated image in the buffer.

14. An apparatus for compensating for motion during an exposure time in a digital imaging system, the apparatus comprising:

an image sensor for producing image data corresponding to a scene; and

a processor for controlling the sensor to produce multiple shorter successive images during said exposure time, each of said successive images selected to correspond to a small fraction of said exposure time such that each said successive image has negligible motion induced blurriness, the processor having a control program that determines an offset between successive images due to said motion, and integrates the successive images to compensate for the offset.

15. The apparatus of claim 14, wherein the processor determines the offset between said images by comparing maxima of areas of high contrast in the images.

16. The apparatus of claim 14, wherein the processor determines maxima of high contrast areas by computing a lateral offset vector between successive images from intensity histograms of the images.

17. The apparatus of claim 14, wherein the processor determines the offset between the images by maximizing a correlation function between the images.

18. The apparatus of Claim 14, wherein the control program determines another offset between integrated successive images, said other offset corresponding to a moving object in a scene, and uses said other offset to compensate for image blurring.

19. The apparatus of claim 14 further comprising a buffer for storing image data read out from the sensor, said processor processing said image data stored in said buffer to determine said offset, and storing integrated data corresponding to a compensated image in said buffer.

20. A method of compensating for motion during an exposure time in a digital imaging system, comprising

dividing said exposure time into a plurality of shorter successive time intervals, each time interval selected such that image data for the time interval has negligible motion induced blurriness;

sensing image data for each of said successive time intervals, the image data corresponding to successive images;

determining a lateral offset between the successive images by constructing two dimensional intensity histograms of said images and determining maxima in said histograms; and

integrating said successive images to compensate for said lateral offset.