Apparatus for digital image stabilization using object tracking and method thereof

• US 7,952,615 B2
• Filed: 10/31/2008
• Issued: 05/31/2011
• Est. Priority Date: 11/02/2007
• Status: Expired due to Fees

• Current Assignee: Pixelplus Co., Ltd.
• Original Assignee: Synergy Innovation Co., Ltd.
• Inventors: Lee, Young-Sin
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): Dagnew; Mekonnen
• Application Number:
  US12/263,133
• Publication Number:
  US 20090115859A1
• Time in Patent Office: 942 Days
• Field of Search: 348/208.99, 348/208.4, 348169-172, 348/222.1, 396/55, 382/103
• US Class Current: 348/208.99
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Apparatus for digital image stabilization using object tracking includes an image signal processing unit that outputs a first digital image obtained under a first exposure value condition and a plurality of second digital images obtained under a second exposure value condition. The apparatus also includes a shaky hand compensating unit that compensates for motion in comparative second digital images relative to a reference second digital image by tracking an object in either a binary image or a reverse binary image of a respective second digital image that has the most objects (i.e., in the binary or reverse image). The compensating unit then generates a shaky hand compensated image by overlapping the motion-compensated images with the reference image. The apparatus further includes an image property correcting unit that corrects a property of the shaky hand compensated image based on a property of the first digital image.

Claims

1. An apparatus for digital image stabilization comprising:

an image signal processing unit operative to output a first digital image obtained under a first exposure value condition and a plurality of second digital images obtained under a second exposure value condition;

a compensating unit operative to;

compensate for motion in a plurality of comparative second digital images relative to a reference second digital image by tracking an object in an object tracking image where the object tracking image is either a binary image or a reverse binary image, whichever has more objects, of a respective comparative second digital image, and

generate a shaky hand compensated image by overlapping the motion-compensated images with the reference image; and

an image property correcting unit operative to correct a property of the shaky hand compensated image based on a property of the first digital image, wherein the compensating unit includes;

an object labeling unit operative to;

track an object in an image among binary images and reverse binary images of the reference image and each comparative image, and label the object;

a motion value computing unit operative to;

match an object parameter of each comparative image with an object of the reference image, calculate an object parameter of each comparative image, and compute a motion value for each comparative image using the calculated object parameter;

a motion compensating unit operative to apply each computed motion value to a corresponding comparative image to compensate for motion of each comparative image; and

a compensated image generating unit operative to overlap each motion-compensated comparative image with the reference image to generate a shaky hand compensated image.

2. The apparatus of claim 1, wherein:

the first digital image is a preview image taken under an exposure value condition of an auto exposure mode, and

the plurality of second digital images is a respective plurality of captured still images taken continually at a regular interval under an exposure value condition that is different than the exposure value condition of the auto exposure mode.

3. The apparatus of claim 1, further comprising:

a sensor controller operative to change an exposure value condition of an image sensor coupled to the image signal processing unit;

a sensor data sheet comprising a mapping between a plurality of second exposure value conditions and a plurality of first exposure value conditions; and

an exposure mode converting unit operative to;

determine the second exposure value condition based on the first exposure value condition by referencing the sensor data sheet, and

control the sensor controller to change the exposure value condition of the image sensor to the second exposure value condition.

4. The apparatus of claim 1, wherein the object labeling unit includes:

an image binarizer operative to convert the reference image and each comparative image into binary images;

an image reverser operative to convert the binary images of the reference image and each comparative image into reverse binary images;

an object labeler operative to label objects in each binary image and each reverse binary image; and

an image selector operative to;

compare the number of objects included in the binary image of the reference image with the number of objects included in the reverse binary image of the reference image, and select an image having more objects as an object tracking image.

5. The apparatus of claim 1 ,wherein the object labeling unit further includes:

an image filter operative to filter the reference image and each comparative image to sharpen boundaries of the images; and

an image synthesizer operative to;

synthesize the boundary-sharpened images received from the image filter with the binary images received from the image binarizer, synthesize the boundary-sharpened images received from the image filter with the reverse binary images received from the image reverser, and output the synthesized binary images and the synthesized reverse binary images, wherein the object labeler labels objects in the synthesized binary images and the synthesized reverse binary images.

6. The apparatus of claim 1, further comprising an image resizer coupled before the object labeling unit and operative to resize the plurality of second digital images by either scaling sizes of the second digital images to a predetermined size or clipping boundaries of a predetermined width from the second digital images.

7. The apparatus of claim 1, further comprising an image selecting unit coupled before the object labeling unit and operative to exclude a comparative image from images for object labeling when the magnitude of a motion vector of the comparative image relative to the reference image exceeds a first critical value.

8. The apparatus of claim 1, further comprising an image selecting unit coupled before the object labeling unit and operative to exclude a comparative image from images for object labeling when a difference in brightness between the reference image and the comparative image does not exceed a second critical value.

9. The apparatus of claim 1, wherein the image property correcting unit includes a color correcting unit operative to correct a color of the shaky hand compensated image by:

replacing pixel data of the shaky hand compensated image with pixel data of the first digital image;

or

interpolating pixel data of the shaky hand compensated image based on pixel data of the first digital image in consideration of a size ratio between the first digital image and the shaky hand compensated image.

10. The apparatus of claim 1, wherein the image property correcting unit includes a brightness correcting unit operative to:

compute quantitatively a difference in brightness between the first digital image and the shaky hand compensated image;

select a brightness correcting intensity; and

correct the brightness of the shaky hand compensated image by the selected intensity.

11. A method for digital image stabilization comprising:

receiving a first digital image obtained under a first exposure value condition and a plurality of second digital images obtained under a second exposure value condition;

selecting an object tracking image where the object tracking image is an image having the most objects among a binary image and a reverse binary image of one of the plurality of second digital images, the plurality of second digital images including a reference image and at least one comparative image;

tracking an object in the object tracking image;

labeling the object;

matching an object of each comparative image with an object of the reference image;

calculating an object parameter of each comparative image;

computing a motion value for each comparative image using the calculated object parameter;

compensating for motion of each comparative image by applying each computed motion value to a corresponding comparative image;

generating a shaky hand compensated image by overlapping each motion-compensated comparative image with the reference image; and

correcting a property of the shaky hand compensated image based on a property of the first digital image.

12. The method of claim 11, further comprising before the receiving of first and second digital images:

loading a sensor data sheet mapping a set of second exposure value conditions to a set of first exposure value conditions;

sensing the first exposure value condition under which the first digital image is taken;

determining the second exposure value condition by referencing the sensor data sheet; and

changing an exposure property of an image sensor from the first exposure value condition to the second exposure value condition.

13. The method of claim 11, wherein:

the first digital image is a preview image taken under an exposure value condition of an auto exposure mode, and

the plurality of second digital images is a respective plurality of captured still images taken continually at a regular interval under an exposure value condition that is different than the exposure value condition of the auto exposure mode.

14. The method of claim 11, wherein the selecting an object tracking image includes:

classifying the plurality of second digital images into a reference image and comparative images;

converting the reference image and each comparative image into binary images and reverse binary images;

labeling objects in each binary image and each reverse binary image;

comparing the number of objects included in the binary image of the reference image with the number of objects included in the reverse binary image of the reference image; and

selecting an image having more objects as an object tracking image.

15. The method of claim 11, wherein the selecting an object tracking image includes:

classifying the plurality of second digital images into a reference image and comparative images;

converting the reference image and each comparative image into boundary-sharpened images;

converting the reference image and each comparative image into binary images and reverse binary images;

synthesizing each binary image with each corresponding boundary-sharpened image;

synthesizing each reverse binary image with each corresponding boundary-sharpened image;

labeling objects in each synthesized binary image and in each synthesized reverse binary image;

comparing the number of objects included in the synthesized binary image of the reference image with the number of objects included in the synthesized reverse binary image of the reference image; and

selecting an image having more objects as an object tracking image.

16. The method of claims 14, further comprising before converting the reference and comparative images:

resizing each second digital image by either scaling the second digital image to a predetermined size or clipping a boundary of a predetermined width from the second digital image.

17. The method of claim 14, further comprising before converting the reference and comparative images:

excluding a comparative image from images for motion compensation when the magnitude of a motion vector of the comparative image relative to the reference image exceeds a first critical value.

18. The method of claim 14, further comprising before converting the reference and comparative images:

excluding a comparative image from images for motion compensation when a difference in brightness between the reference image and the comparative image does not exceed a second critical value.

19. The method of claim 11, wherein the correcting of a property includes correcting the color of the shaky hand compensated image by:

replacing a pixel data of the shaky hand compensated image with pixel data of the first digital image;

or

interpolating pixel data of the shaky hand compensated image based on pixel data of the first digital image in consideration of a size ratio between the first digital image and the shaky hand compensated image.

20. The method of claim 11, wherein the correcting of a property includes correcting the brightness of the shaky hand compensated image by:

computing quantitatively a difference in brightness between the first digital image and the shaky hand compensated image;

selecting a brightness correcting intensity; and

correcting the brightness of the shaky hand compensated image by the selected intensity.