Apparatus and method for digital image stabilization using object tracking

• US 7,952,614 B2
• Filed: 10/31/2008
• Issued: 05/31/2011
• Est. Priority Date: 11/02/2007
• Status: Expired due to Fees

• Current Assignee: Pixelplus Co., Ltd.
• Original Assignee: Synergy Innovation Co., Ltd.
• Inventors: Lee, Young-Sin
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US12/262,968
• Publication Number:
  US 20090115858A1
• Time in Patent Office: 942 Days
• Field of Search: 348/208.99, 348/208.4, 348169-172, 396/55, 382/103
• US Class Current: 348/208.99
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 1/409 Edge or detail enhancement; Noise or error suppression
• CPC Class Codes: H04N 1/6047 wherein the test pattern is part of an arbitrary user image
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/73 by influencing the exposure time

Abstract

Apparatus for digital image stabilization using object tracking can produce a digital still image free of an image blurring phenomenon caused by shaky hands without separate hardware and can produce an image that is bright, clear, and noise free. The apparatus includes an image signal processing unit that outputs a first digital image obtained under a first exposure value condition and a plurality of second digital images obtained under a second exposure value condition. The apparatus also includes a shaky hand compensating unit that compensates for motion in comparative second digital images relative to a reference second digital image using object tracking. The compensating unit then generates a shaky hand compensated image by overlapping each compensated comparative image with the reference image. The apparatus further includes an image property correcting unit that corrects a property of the shaky hand compensated image based on a property of the first digital image.

Claims

1. An apparatus for digital image stabilization comprising:

an image signal processing unit operative to output a first digital image obtained under a first exposure value condition and a plurality of second digital images obtained under a second exposure value condition;

a compensating unit operative to compensate for motion in a plurality of comparative second digital images relative to a reference second digital image using object tracking, and generate a compensated image by overlapping each compensated comparative image with the reference image; and

an image property correcting unit operative to correct a property of the compensated image based on a property of the first digital image wherein an object labeling unit operative to classify the plurality of second digital images into a reference image and one or more comparative images, and label an object of the reference image and an object of each comparative image using object tracking;

a motion value computing unit operative to match the object of each comparative image with the object of the reference image, calculate a parameter of the matched object, and compute a motion value of each comparative image relative to the reference image with the calculated parameter;

a motion compensating unit operative to apply each computed motion value to a corresponding comparative image to compensate for motion of each comparative image;

and a compensated image generating unit operative to overlap each motion-compensated comparative image with the reference image to generate a compensated image.

2. The apparatus of claim 1, wherein:

the first exposure value condition is an exposure value condition of an auto exposure mode, and

the second exposure value condition is an exposure value condition corresponding to a luminance lower than a luminance of the auto exposure mode.

3. The apparatus of claim 1, wherein:

the image signal processing unit is operative to receive an image signal under the first exposure value condition and a plurality of image signals under the second exposure value condition from an image sensor; and

the apparatus further comprises an exposure mode converting unit operative to;

determine adaptively the second exposure value condition based on the first exposure value condition, and

change an exposure property of the image sensor from the first exposure value condition to the second exposure value condition.

4. The apparatus of claim 3, further comprising:

a sensor controller operative to change an exposure value condition of the image sensor; and

a sensor data sheet defining the second exposure value condition based on the first exposure value condition; and

wherein the exposure mode converting unit determines the second exposure value condition based on the first exposure value condition by referencing the sensor data sheet, and controls the sensor controller to change the exposure value condition of the image sensor to the second exposure value condition.

5. The apparatus of claim 1, wherein the first digital image is a preview image obtained in an auto exposure mode.

6. The apparatus of claim 5, wherein the first digital image is a preview image generated before an operation of a shutter where a preview image is generated at a regular frame interval in a preview mode.

7. The apparatus of claim 1, wherein the second digital images are full-size captured still images generated at a regular time interval in an image capture mode.

8. The apparatus of claim 1, wherein the object labeling unit comprises:

an image binarization unit operative to convert the reference image and each comparative image into binary images; and

an object labeler operative to;

extract objects from the reference binary image and each comparative binary image, and label the extracted objects.

9. The apparatus of claim 8, wherein the object labeling unit further comprises:

an image filter operative to filter the reference image and each comparative image to sharpen boundaries of the images; and

an image synthesizer operative to synthesize the boundary-sharpened images outputted from the image filter with the binary images,

wherein the object labeler labels objects from the synthesized images.

10. The apparatus of claim 8, further comprising an image resizer operative to resize the plurality of second digital images, before the second digital images are processed by the object labeling unit, by scaling sizes of the second digital images to a predetermined size or clipping boundaries of a predetermined width from the second digital images.

11. The apparatus of claim 8, further comprising an image selecting unit operative to exclude a comparative image from images for object labeling if:

a magnitude of the motion vector of the comparative image relative to the reference image exceeds a first critical value, or

a difference in brightness between the reference image and the comparative image does not exceed a second critical value.

12. The apparatus of claim 8, wherein the motion value of each comparative image comprises any one selected from a group consisting of a motion value according to translation, a motion value according to rotation, a motion value according to scaling, a motion value according to shear, or combinations thereof.

13. The apparatus of claim 1, wherein the image property correcting unit comprises a color correcting unit for correcting a color of the compensated image by:

replacing pixel data of the compensated image with pixel data of the first digital image, or

interpolating pixel data of the compensated image based on pixel data of the first digital image in consideration of a size ratio between the compensated image and the first digital image.

14. The apparatus of claim 1, wherein the image property correcting unit comprises a brightness correcting unit operative to:

compute a difference in brightness between the first digital image and the compensated image,

select adaptively a brightness correcting intensity, and

correct the brightness of the compensated image by the selected brightness correcting intensity.

15. A method for digital image stabilization using object tracking comprising:

receiving a first digital image obtained under a first exposure value condition and a plurality of second digital images obtained under a second exposure value condition;

compensating for motion in a plurality of comparative second digital images relative to a reference second digital image using object tracking;

generating a compensated image by overlapping each compensated comparative image with the reference image; and

correcting a property of the compensated image based on a property of the first digital image the compensating for motion comprises;

classifying the plurality of second digital images into a reference image and comparative images;

extracting objects from the reference image and each comparative image, labeling the extracted objects, and calculating a parameter of each labeled object;

matching the object of each comparative image with the object of the reference image, and computing a motion value of each comparative image using the parameter of the matched object;

applying each computed motion value to a corresponding comparative image to compensate for motion of each comparative image;

and overlapping each motion-compensated comparative image with the reference image to generate a shaky hand compensated image.

16. The method of claim 15, further comprising before the receiving of first and second digital images:

loading a sensor data sheet defining the second exposure value condition based on the first exposure value condition;

sensing the first exposure value condition under which the first digital image is taken;

determining adaptively the second exposure value condition based on the first exposure value condition by referencing the sensor data sheet; and

changing an exposure property of an image sensor from the first exposure value condition to the second exposure value condition.

17. The method of claim 15, wherein

the first exposure value condition is an exposure value condition of an auto exposure mode, and

the second exposure value condition is an exposure value condition corresponding to a luminance lower than a luminance of the auto exposure mode.

18. The method of claim 15, wherein the first digital image is a preview image generated before an operation of a shutter where a preview image is generated at a regular frame interval in an auto exposure mode.

19. The method of claim 15, wherein the second digital images are captured still images generated at a regular time interval in an image capture mode.

20. The method of claim 15, wherein the extracting of objects comprises:

converting an image for object extraction into a binary image;

detecting an object, the object comprising a plurality of white pixels connected to

each other in the binary image; and

extracting the object.

21. The method of claim 15, wherein the extracting of objects comprises:

filtering an image to sharpen a boundary of the image;

converting the image into a binary image;

synthesizing the boundary-sharpened image with the binary image;

detecting an object, the object comprising a plurality of white pixels connected to each other in the synthesized image; and

extracting the object.

22. The method of claim 15, further comprising before the extracting of objects:

scaling or boundary-clipping each second digital image to a predetermined size to resize the second digital image;

or

excluding a comparative image from images for object labeling if;

the magnitude of a motion vector of the comparative image exceeds a first critical value, or a difference in brightness between the reference image and the comparative image does not exceed a second critical value.

23. The method of claim 15, wherein the motion value of each comparative image comprises any one selected from a group consisting of a motion value according to translation, a motion value according to rotation, a motion value according to scaling, a motion value according to shear, or combinations thereof.

24. The method of claim 15, wherein the correcting of a property comprises correcting a color of the compensated image by:

replacing a pixel data of the compensated image with pixel data of the first digital image, or

interpolating the pixel data of the compensated image based on pixel data of the first digital image in consideration of a size ratio between the compensated image and the first digital image.

25. The method of claim 15, wherein the correcting of a property comprises:

computing a difference in brightness between the first digital image and the compensated image;

selecting adaptively a brightness correcting intensity; and

correcting the brightness of the compensated image by the selected brightness correcting intensity.