Reference frame selection for still image stabilization

• US 9,491,360 B2
• Filed: 06/06/2013
• Issued: 11/08/2016
• Est. Priority Date: 06/06/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Schulze, Anita Nariani, Toft, Rolf, Hubel, Paul M., Tico, Marius, Zhou, Jianping, Brunner, Ralph, Molgaard, Claus
• Primary Examiner(s): Harandi, Siamak
• Assistant Examiner(s): PATEL, PINALBEN V
• Application Number:
  US13/911,873
• Publication Number:
  US 20140362256A1
• Time in Patent Office: 1,251 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken

Abstract

Systems, methods, and computer readable media to improve image stabilization operations are described. A novel combination of image quality and commonality metrics are used to identify a reference frame from a set of commonly captured images which, when the set'"'"'s other images are combined with it, results in a quality stabilized image. The disclosed image quality and commonality metrics may also be used to optimize the use of a limited amount of image buffer memory during image capture sequences that return more images that the memory may accommodate at one time. Image quality and commonality metrics may also be used to effect the combination of multiple relatively long-exposure images which, when combined with a one or more final (relatively) short-exposure images, yields images exhibiting motion-induced blurring in interesting and visually pleasing ways.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more programmable control devices to:

obtain a temporal sequence of images of a scene;

determine, for each of the obtained images, a quality metric value;

identify, from the obtained images, a set of relevant images based at least in part on the obtained images'"'"' quality metric values;

generate, from the relevant set of images, a temporary image;

determine, for each image in the set of relevant images, a commonality metric value between the each image and the temporary image;

identify, from the set of relevant images, a reference image based at least in part on the commonality metric values; and

combine one or more of the set of relevant images with the reference image to generate an output image of the scene.

2. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a quality metric value comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image'"'"'s sharpness.

3. The non-transitory program storage device of claim 2, wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image'"'"'s sharpness comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on output from at least one of an auto-focus system and an auto-exposure system.

4. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a quality metric value comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a blur value based at least in part on output from one or more motion sensors.

5. The non-transitory program storage device of claim 2, wherein the instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image'"'"'s sharpness further comprise instructions to incorporate a secondary consideration.

6. The non-transitory program storage device of claim 5, wherein the secondary consideration comprises output from an object detection system.

7. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to identify, from the obtained images, a set of relevant images comprise instructions to cause the one or more processors to:

identify a highest quality metric value from the obtained images'"'"' quality metric values;

determine, for each of the obtained images, a relative quality metric value based on a ratio of the image'"'"'s quality metric value and the highest quality metric value; and

identify, as a relevant image, all those images from the obtained images having a relative quality metric value greater than a threshold value.

8. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to generate a temporary image comprise instructions to cause the one or more processors to determine an average image based, at least in part, on each of the relevant images.

9. The non-transitory program storage device of claim 8, wherein the average image comprises a pixel-by-pixel average of each of the relevant images.

10. The non-transitory program storage device of claim 8, wherein the instructions to cause the one or more processors to determine, for each of the relevant images, a commonality metric value comprise instructions to cause the one or more processors to determine a difference value between each of the relevant images and the average image, wherein the difference value represents a commonality metric value.

11. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain a temporal sequence of images of a scene comprise instructions to cause the one or more processors to:

obtain an initial temporal sequence of images that includes N images;

determine, for each of the N images, a quality metric value; and

retain, as a temporal sequence of images, M of the N images based on the determined quality metric value for each of the N images, where M is less than N.

12. The non-transitory program storage device of claim 11, wherein the instructions to cause the one or more processors to determine, for each of the N images, a quality metric value further comprise instructions to cause the one or more processors to determine, for each of the N images, a similarity metric value.

13. The non-transitory program storage device of claim 12, wherein the instructions to cause the one or more processors to retain, as a temporal sequence of images, M of the N images based on the determined quality metric value for each of the N images comprise instructions to cause the one or more processors to retain, as a temporal sequence of images, M of the N images based on the determined quality metric value and the determined similarity metric value for each of the N images.

14. The non-transitory program storage device of claim 13, wherein the similarity metric value between two images is based, at least in part, on a time between when the two images were obtained.

15. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to obtain a temporal sequence of images of a scene comprise instructions to cause the one or more programmable control devices to:

obtain an initial sequence of images, each image having an exposure time; and

obtain one or more final images after the initial sequence of images is obtained, wherein each of the one or more final images has an exposure time that is shorter than any exposure time of any image in the initial sequence of images.

16. The non-transitory program storage device of claim 15, wherein the instructions to cause the one or more programmable control devices to identify, from the obtained images, a set of relevant images comprise instructions to cause the one or more programmable control devices to:

determine a first set of relevant images from the initial sequence of images based, at least in part, on the initial sequence of images'"'"' quality metric values; and

determine a second set of relevant images from the one or more final images based, at least in part, on the one or more final images'"'"' quality metric values.

17. The non-transitory program storage device of claim 16, wherein the instructions to cause the one or more programmable control devices to determine, for each image in the set of relevant images, a commonality metric value comprise instructions to cause the one or more programmable control devices to:

determine, for each image in the first set of relevant images, a commonality metric value exclusive of the images in the second set of relevant images; and

determine, for each image in the second set of relevant images, a commonality metric value exclusive of the images in the first set of relevant images.

18. The non-transitory program storage device of claim 17, wherein the instructions to cause the one or more programmable control devices to identify, from the set of relevant images, a reference image comprise instructions to cause the one or more programmable control devices to:

identify, from the first set of relevant images, a first reference image based at least in part of the first set of relevant images'"'"' commonality metric values; and

identify, from the second set of relevant images, a second reference image based at least in part of the second set of relevant images'"'"' commonality metric values.

19. The non-transitory program storage device of claim 18, wherein the instructions to cause the one or more programmable control devices to combine one or more of the set of relevant images with the reference image to generate an output image of the scene comprise instructions to cause the one or more programmable control devices to:

combine zero or more images from the first set of relevant images with the first reference image to generate a first intermediate image;

combine zero or more images from the second set of relevant images with the second reference image to generate a second intermediate image; and

combine the first and second intermediate images to generate an output image of the scene.

20. A digital image capture device, comprising:

an imaging sensor;

a memory communicatively coupled to the imaging sensor;

a display communicatively coupled to the memory; and

one or more processors communicatively coupled to the imaging sensor, memory and display and configured to execute instructions stored in the memory to cause the one or more processors to—

obtain, from the imaging sensor, a temporal sequence of images of a scene,

determine, for each of the obtained images, a quality metric value,

identify, from the obtained images, a set of relevant images based at least in part on the obtained images'"'"' quality metric values,

generate, from the relevant images, a temporary image,

determine, for each image in the set of relevant images, a commonality metric value between the each image and the temporary image,

identify, from the set of relevant images, a reference image based at least in part on the commonality metric values,

combine one or more of the set of relevant images with the reference image to generate an output image of the scene, and

store the output image of the scene in the memory.