Automated Selection Of Keeper Images From A Burst Photo Captured Set

• US 20150071547A1
• Filed: 09/09/2013
• Published: 03/12/2015
• Est. Priority Date: 09/09/2013
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Keating, Brett, Sachs, Todd, Molgaard, Claus, Rousson, Michael, Harris, Elliott, Titi, Justin, Hsu, Karl, Zuliani, Marco, Wong, Vincent, Brasket, Jeff
• Application Number:
  US14/021,857
• Publication Number:
  US 20150071547A1
• Time in Patent Office: Days
• US Class Current: 382/195
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Systems and methods for improving automatic selection of keeper images from a commonly captured set of images are described. A combination of image type identification and image quality metrics may be used to identify one or more images in the set as keeper images. Image type identification may be used to categorize the captured images into, for example, three or more categories. The categories may include portrait, action, or “other.” Depending on the category identified, the images may be analyzed differently to identify keeper images. For portrait images, an operation may be used to identify the best set of faces. For action images, the set may be divided into sections such that keeper images selected from each section tell the story of the action. For the “other” category, the images may be analyzed such that those having higher quality metrics for an identified region of interest are selected.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more programmable control devices to:

obtain a temporal sequence of images of a scene;

detect, by an image processor, if each of the images contains primarily faces by calculating a total face size for each image and comparing the total face size of the image to a total size of the image;

process, by the image processor, each of the obtained images to obtain for each image at least one quality metric value;

select one or more images from the sequence of images as keeper images, wherein the selection is made at least in part based on whether each image contains primarily faces and on the at least one quality metric value for the image; and

retain the one or more keeper images in a memory.

2. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to process comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image'"'"'s sharpness.

3. The non-transitory program storage device of claim 2, wherein the value indicative of an image'"'"'s sharpness is based on output from at least one of an auto-focus system and an auto-exposure system.

4. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to process comprise instructions to cause the one or more programmable control devices to determine, for each of the obtained images, a blur value.

5. The non-transitory program storage device of claim 1, wherein the instructions further cause the one or more programmable control devices to:

detect, by the processor, if the scene in each image contains action when the image is detected as not containing primarily faces; and

to classify each image based on the detection of primarily faces or action.

6. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to detect further comprise instructions to cause the one or more programmable control devices to determine if each of the detected faces is smiling, when at least one face is detected.

7. The non-transitory program storage device of claim 5, wherein the instructions to cause the one or more programmable control devices to classify comprise instructions to cause the one or more programmable control devices to classify the images as portraits when one or more images in the image sequence contain primarily faces.

8. The non-transitory program storage device of claim 7, wherein when the images are classified as action, the instructions to cause the one or more programmable control devices to select one or more images comprise instructions to cause the one or more programmable control devices to:

divide the sequence of images into two or more sections; and

select a keeper image from each of the two or more sections based on the at least one quality metric value.

9. The non-transitory program storage device of claim 5, wherein the instructions to cause the one or more programmable control devices to select one or more images comprise instructions to cause the one or more programmable control devices to:

identify a region of interest in the one or more images;

obtain at least one quality metric for the region of interest; and

select one or more images from the sequence of images based at least on one or more quality metric values for the region of interest.

10. A digital image capture device, comprising:

a memory;

a display communicatively coupled to the memory; and

one or more processors communicatively coupled to the memory and display and configured to execute instructions stored in the memory comprising;

obtaining a temporal sequence of images of a scene;

detecting, by an image processor, if each of the images in the sequence of images contains primarily faces by calculating a total face size for each image and comparing the total face size of the image to a total size of the image;

detecting, by the image processor, if each image contains action when the image is detected as not containing primarily faces;

classifying each of the images based on the detection;

processing, by the image processor, each of the obtained images to obtain for each image at least one quality metric value;

selecting one or more images from the sequence of images as keeper images, wherein the selection is made at least in part based on the classification and on the at least one quality metric value; and

retaining the one or more keeper images in the memory.

11. The system of claim 10, wherein detecting if each of the images in the sequence of images contains action comprises constructing a feature vector from each of the images and applying the feature vectors to a classifier.

12. The system of claim 10, wherein classifying comprises classifying the images as other if the images are not detected to contain primarily faces and if the scene is not detected to contain action.

13. The system of claim 12, wherein when the images are classified as other, selecting one or more images from the sequence of images as keeper images comprises:

identifying a region of interest in the images;

expanding the region of interest to include one or more blocks that overlap the region of interest;

determining a quality metric value for the expanded region of interest; and

selecting one or more images from the sequence of images based on the at least one quality metric value for the expanded region of interest.

14. The system of claim 13, wherein identifying a region of interest comprises:

registering each two images in the sequence of images with respect to each other;

comparing the registered images with each other; and

identifying a region in the registered images where the differences between the registered images is larger than a specified threshold.

15. A method comprising:

obtaining a temporal sequence of images of a scene;

detecting, by an image processor, if each of the images in the sequence of images contains primarily faces by calculating a total face size for each image and comparing the total face size of the image to a total size of the image;

detecting, by the image processor, if the scene in each image contains action when the image is detected as not containing primarily faces;

classifying the images based on the detection;

processing, by the image processor, each of the obtained images to obtain for each image at least one quality metric value;

selecting one or more images from the sequence of images as keeper images, wherein the selection is made at least in part based on the classification and on the at least one quality metric value; and

retaining the one or more keeper images in a memory.

16. The method of claim 15, wherein classifying the images based on the detection comprises classifying the images as portraits when one or more images in the sequence of images contains primarily faces.

17. The method of claim 15, wherein detecting if each image in the sequence of images contains primarily faces comprises determining if each of the detected faces is blinking.

18. The method of claim 16, further comprising determining a face sharpness value for each detected face.

19. The method of claim 15, wherein selecting one or more images from the sequence of images as keeper images comprises:

identifying a region of interest in the images;

expanding the region of interest to include one or more blocks that overlap the region of interest;

determining, by the image processor, at least one quality metric value for the expanded region of interest; and

selecting one or more images from the sequence of images based at least in part on the at least one quality metric value for the expanded region of interest.

20. The method of claim 15, wherein processing each of the obtained images comprises determining, for each of the obtained images, a value based on at least a portion of the image, wherein the value is indicative of the image'"'"'s sharpness.