Image feature identification and motion compensation apparatus, systems, and methods

• US 20080225125A1
• Filed: 03/14/2007
• Published: 09/18/2008
• Est. Priority Date: 03/14/2007
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Inventors: Lin, Sheng, Li, Dong, Silverstein, Amnon
• Granted Patent: US 7,924,316 B2
• Time in Patent Office: Days
• US Class Current: 348/208.4
• CPC Class Codes: H04N 23/73 by influencing the exposure time

Abstract

Apparatus, systems, and methods disclosed herein may estimate the magnitude of relative motion between a scene and an image capture device used to capture the scene. Some embodiments may utilize discrete cosine transform and/or Sobel gradient techniques to identify one or more blocks of pixels in an originating calibration image frame. Matching blocks of pixels may be located in a successive calibration image frame. Motion vectors originating at one calibration frame and terminating at the other calibration frame may be calculated. The magnitude of relative motion derived thereby may be used to adjust image capture parameters associated with the image capture device, including exposure settings.

Claims

1. An apparatus, comprising:

motion priority logic to estimate a magnitude of relative motion between a scene and an image capture device used to capture the scene, the relative motion occurring during a time period between a time of capturing a first calibration image of the scene and a time of capturing a second calibration image of the scene; and

final image exposure logic to calculate at least one exposure setting associated with the image capture device using the magnitude of relative motion.

2. The apparatus of claim 1, wherein the image capture device comprises a digital camera.

3. The apparatus of claim 1, further comprising:

an exposure control module coupled to the final image exposure logic to adjust the at least one exposure setting; and

an image sensor array (ISA) coupled to the exposure control module to capture the first calibration image of the scene and the second calibration image of the scene.

4. The apparatus of claim 3, further comprising:

calibration image capture logic coupled to the exposure control module to calculate a calibration exposure time used to capture the first calibration image and the second calibration image, the calibration exposure time calculated to limit blur due to image motion to a selected maximum amount and to provide image feature detail in at least highlighted areas of the first and second calibration images.

5. The apparatus of claim 1, further comprising:

an image feature selection module coupled to the motion priority logic to identify a first image feature block of pixels in the first calibration image and to store the first image feature block of pixels in an image feature buffer.

6. The apparatus of claim 5, further comprising:

a discrete cosine transform (DCT) module coupled to the image feature selection module to perform a DCT operation on the first image feature block of pixels selected from a sub-region of the first calibration image to derive an image feature strength metric value for the first image feature block of pixels.

7. The apparatus of claim 5, further comprising:

a multiple edge module (MEM) coupled to the image feature selection module to perform a gradient vector analysis on the first image feature block of pixels selected from a sub-region of the first calibration image to derive an image feature strength metric value for the first image feature block of pixels.

8. The apparatus of claim 5, further comprising:

a search window buffer coupled to the ISA to store a set of pixels from an area of the second calibration image for image feature block localization analysis, wherein the area of the second calibration image corresponds to an area of the first calibration image containing the first image feature block of pixels.

9. The apparatus of claim 5, further comprising:

target matching logic coupled to the motion priority logic to search an area of the second calibration image for a second image feature block, wherein the second image feature block matches the first image feature block better than any other image feature block located within the area of the second calibration image, and wherein the area of the second calibration image corresponds to an area of the first calibration image containing the first image feature block.

10. The apparatus of claim 9, wherein the target matching logic is configured such that a set of spatially arranged luminance values associated with the second image feature block matches a set of spatially arranged luminance values associated with the first image feature block better than any other set of spatially arranged luminance values located within the area of the second calibration image.

11. The apparatus of claim 9 or claim 10, wherein the target matching logic is configured to evaluate image feature block mismatch as at least one of a sum of absolute differences between luminance values associated with each of two image feature blocks or a sum of squares of differences between luminance values associated with each of the two image feature blocks.

12. The apparatus of claim 9, further comprising:

motion vector logic coupled to the motion priority logic to calculate a magnitude of a vector associated with the first image feature block as a relative distance between a location of the first image feature block and a location of the second image feature block, the relative distance divided by the time period between capturing the first calibration image and capturing the second calibration image.

13. A system, comprising:

motion priority logic associated with an image capture device to estimate a magnitude of relative motion between a scene and an image capture device used to capture the scene, the relative motion occurring during a period between a time of capturing a first calibration image of the scene and a time of capturing a second calibration image of the scene;

final image exposure logic to calculate an adjustment of at least one exposure setting associated with the image capture device using the magnitude of the relative motion between the scene and the image capture device used to capture the scene; and

a frame rate adjustment module to adjust a frame rate according to the adjustment of the at least one exposure setting.

14. The system of claim 13, wherein the frame rate adjustment module is configured to adjust a frame rate associated with a multi-shot still-frame camera sequence.

15. The system of claim 13, wherein the frame rate adjustment module is incorporated into a video camera to adjust a frame rate associated with a video sequence.

17. The method of claim 15, wherein the image capture device comprises a digital camera.

18. The method of claim 15, wherein the at least one exposure setting comprises at least one of an exposure time, an aperture size, an amplification factor associated with a plurality of image sensor array signals, or a flash duration.

19. The method of claim 15, further comprising:

calculating an amount of adjustment of the at least one exposure setting to limit image blur due to the relative motion between the scene and the image capture device to a desired maximum amount of blur.

20. The method of claim 15, further comprising:

setting an exposure time to a value corresponding to a desired maximum amount of blur expressed as a distance divided by the magnitude of relative motion between the scene and the image capture device.

21. The method of claim 15, further comprising:

setting a calibration exposure time used to capture the first calibration image and the second calibration image, the calibration exposure time calculated to limit blur due to image motion to a selected maximum amount and to provide image feature detail in at least highlighted areas of the first and second calibration images.

22. The method of claim 15, further comprising:

evaluating pixels in a plurality of rows in a horizontal strip in the second calibration image for sharpness; and

storing a sharpness metric value associated with the horizontal strip if a sharpness metric value associated with each row of the plurality of rows in the horizontal strip falls within a selected range.

23. The method of claim 22, further comprising:

evaluating pixels from a selected number of rows of the final image for final sharpness;

comparing the final sharpness to the sharpness metric value associated with the horizontal strip of the second calibration image; and

re-capturing the final image using a shorter exposure time than previously used to capture the final image if the final sharpness is lower than the sharpness metric value associated with the horizontal strip by greater than a minimum amount.

24. The method of claim 15, further comprising:

identifying a first image feature block of pixels in the first calibration image;

searching the second calibration image to find a second image feature block of pixels, wherein the second image feature block of pixels matches the first image feature block of pixels within a selected match tolerance; and

calculating a magnitude of a motion vector associated with the first image feature block of pixels as a relative distance between the first image feature block of pixels and the second image feature block of pixels divided by a difference between T1 and T2.

25. The method of claim 24, further comprising:

utilizing a nominal exposure to capture the final image if a match between the first image feature block of pixels and the second image feature block of pixels is not within the selected match tolerance.

26. The method of claim 24, further comprising:

utilizing a decreased exposure time to capture the final image if a match between the first image feature block of pixels and the second image feature block of pixels is within the selected match tolerance and the magnitude of the motion vector is above a selected motion threshold.

27. The method of claim 24, further comprising:

using an increased exposure time to capture the final image if a match between the first image feature block of pixels and the second image feature block of pixels is within the selected match tolerance and the magnitude of the motion vector is below a selected motion threshold.

28. The method of claim 24, further including:

measuring a position of the first image feature block of pixels relative to a horizontal axis corresponding to a horizontal frame edge of the first calibration image and a vertical axis corresponding to a vertical frame edge of the first calibration image; and

measuring a position of the second image feature block of pixels relative to a horizontal axis corresponding to a horizontal frame edge of the second calibration image and a vertical axis corresponding to a vertical frame edge of the second calibration image.

29. The method of claim 24, further comprising:

discarding the first calibration image if an image feature strength metric value associated with the first image feature block of pixels is below a selected feature strength threshold value; and

performing at least one of re-capturing the first calibration image or capturing the final image using at least one of a set of user-selected exposure settings or a pre-programmed automatic exposure setting.

30. The method of claim 24, further comprising:

segmenting an image frame associated with the first calibration image into a set of sub-regions; and

selecting an image feature block of pixels from each sub-region of the set of sub-regions, the image feature block of pixels having a highest image feature strength metric value of any block of pixels in the sub-region.

31. The method of claim 30, further comprising:

selecting the set of sub-regions such that sub-regions are distributed within the image frame according to a selected design.

32. The method of claim 30, further comprising:

dividing the image frame into a set of horizontal bands;

selecting the set of sub-regions such that a single sub-region is located in each horizontal band of the set of horizontal bands.

33. The method of claim 24, further comprising:

buffering a set of pixels from an area of the second calibration image in a search window buffer while searching the area for the second image feature block of pixels, wherein the area of the second calibration image is located at a position in the second calibration image corresponding to a position of an area in the first calibration image containing the first image feature block of pixels; and

releasing the set of pixels from the search window buffer prior to buffering a next set of pixels to search.

34. The method of claim 15, further comprising:

performing a discrete cosine transform (DCT) operation on a block of pixel luminance values associated with the first calibration image; and

performing a mathematical calculation on a plurality of DCT coefficients selected from low-frequency bands of a DCT coefficient matrix resulting from the DCT operation to derive an image feature strength metric value.

35. The method of claim 15, further comprising:

performing a gradient vector analysis on a block of pixel luminance values associated with the first calibration image to derive a value of an image feature strength metric associated with the block of pixels.

16. A method, comprising:

capturing a first calibration image of a scene at a first time (T1) using an image capture device;

capturing a second calibration image of the scene at a second time (T2);

estimating a magnitude of relative motion between the scene and the image capture device during a period between the first time and the second time using the first calibration image and the second calibration image;

adjusting an exposure setting using the magnitude of relative motion between the scene and the image capture device as estimated; and

capturing a final image using the adjusted exposure setting.