Automated selection of keeper images from a burst photo captured set

• US 10,523,894 B2
• Filed: 09/15/2016
• Issued: 12/31/2019
• Est. Priority Date: 09/09/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Keating, Brett, Wong, Vincent, Sachs, Todd, Molgaard, Claus, Rousson, Michael, Harris, Elliott, Titi, Justin, Hsu, Karl, Brasket, Jeff, Zuliani, Marco
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US15/266,460
• Publication Number:
  US 20170006251A1
• Time in Patent Office: 1,202 Days
• Field of Search: 382305
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Systems and methods for improving automatic selection of keeper images from a commonly captured set of images are described. A combination of image type identification and image quality metrics may be used to identify one or more images in the set as keeper images. Image type identification may be used to categorize the captured images into, for example, three or more categories. The categories may include portrait, action, or “other.” Depending on the category identified, the images may be analyzed differently to identify keeper images. For portrait images, an operation may be used to identify the best set of faces. For action images, the set may be divided into sections such that keeper images selected from each section tell the story of the action. For the “other” category, the images may be analyzed such that those having higher quality metrics for an identified region of interest are selected.

Claims

1. A non-transitory program storage device, readable by one or more processors and comprising instructions stored thereon to cause the one or more processors to:

obtain a temporal sequence of images of a scene from image data captured by a camera;

determine, for each of the obtained images, at least one quality metric value;

detect an indication of motion based on a comparison of the image data of two images of the temporal sequence of images;

determine, based on the indication of motion, whether the scene in the temporal sequence of images is an action scene;

select, based on the determination that the scene is an action scene, a keeper image based on a first technique, the first technique causing the one or more processors to;

divide the temporal sequence of images into two or more sections, in response to the determination that the scene contains action;

compare the at least one quality metric value of images within each section; and

select, for each of the two or more sections, at least one image as a keeper image based on the comparison of the at least one quality metric value; and

select, based on a determination that the scene is not an action scene, a keeper image based on a second technique.

2. The non-transitory program storage device of claim 1, wherein dividing the temporal sequence of images is based on detecting a scene change.

3. The non-transitory program storage device of claim 1, wherein the instructions further cause the one or more processors to:

calculate a distance between images of the temporal sequence of images, wherein the division of the temporal sequence of images is based on the calculated distance between the images.

4. The non-transitory program storage device of claim 3, wherein dividing the temporal sequence of images is based on a ratio of a first average distance between images within a section to a second average distance between images from different sections.

5. The non-transitory program storage device of claim 1, wherein the at least one quality metric value is indicative of at least one of sharpness and blurriness of the images.

6. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to determine, for each obtained image of the temporal sequence of images, the at least one quality metric value further comprise instructions to cause the one or more processors to create a color histogram of the images in an UV color space.

7. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to determine that the scene in the temporal sequence of images contains action comprise instructions to cause the one or more processors to:

construct a feature vector from the image of the temporal sequence of images; and

apply the feature vector from each of the images to a classifier to determine whether the scene contains action.

8. The non-transitory program storage device of claim 7, wherein the instructions to cause the one or more processors to determine that the scene in the temporal sequence of images contains action further comprise instructions to cause the one or more processors to:

register one or more first images from the temporal sequence of images with one or more second images from the temporal sequence of images; and

identify regions of interest between a first image and a second image,

wherein the first image and the second image form an image pair,

wherein a region of interest comprises a region in the first image and a corresponding region in the second image that exhibit a difference that is greater than a specified threshold, and

wherein the feature vector of the first image of the image pair is indicative of a similarity in a distribution of color between the first image'"'"'s region of interest and the second image'"'"'s corresponding region of interest.

9. The non-transitory program storage device of claim 1, wherein each section of the two or more sections corresponds to at least one determined action in the temporal sequence of images.

21. The non-transitory program storage device of claim 1, wherein the second technique comprises a face detection technique.

10. A digital image capture device, comprising:

a memory;

a display communicatively coupled to the memory; and

one or more processors communicatively coupled to the memory and display and configured to execute instructions stored in the memory comprising;

obtain a temporal sequence of images of a scene from image data captured by a camera;

determine, for each of the obtained images, at least one quality metric value;

detect an indication of motion based on a comparison of the image data of two images of the temporal sequence of images;

determine, based on the indication of motion, whether the scene in the temporal sequence of images is an action scene;

select, based on the determination that the scene is an action scene, a keeper image based on a first technique, the first technique causing the one or more processors to;

divide the temporal sequence of images into two or more sections, in response to the determination that the scene contains action;

compare the at least one quality metric value of images within each section; and

select, for each of the two or more sections, at least one image as a keeper image based on the comparison of the at least one quality metric value; and

select, based on a determination that the scene is not an action scene, a keeper image based on a second technique.

11. The device of claim 10, wherein dividing the temporal sequence of images is based on detecting a scene change.

12. The device of claim 10, wherein the instructions further comprising:

calculate a distance between images of the temporal sequence of images, wherein the division of the temporal sequence of images is based on the calculated distance between the images.

13. The device of claim 12, wherein dividing the temporal sequence of images is based on a ratio of a first average distance between images within a section to a second average distance between images from different sections.

14. The device of claim 10, wherein the at least one quality metric value is indicative of at least one of sharpness and blurriness of the images.

15. The device of claim 10, wherein determining, for each obtained image of the temporal sequence of images, the at least one quality metric value further comprises creating a color histogram of the images in an UV color space.

16. The device of claim 10, wherein determining that the scene in the temporal sequence of images contains action further comprises:

construct a feature vector from each image of the temporal sequence of images; and

apply the feature vector from each of the images to a classifier to determine whether the scene contains action.

22. The device of claim 10, wherein the second technique comprises a face detection technique.

17. A method for processing images, comprising:

obtaining a temporal sequence of images of a scene from image data captured by a camera;

determining, for each of the obtained images, at least one quality metric value;

detecting an indication of motion based on a comparison of the image data of two images of the temporal sequence of images;

determining, based on the indication of motion, whether the scene in the temporal sequence of images is an action scene;

selecting, based on the determination that the scene is an action scene, a keeper image based on a first technique, the first technique comprising;

dividing the temporal sequence of images into two or more sections, in response to the determination that the scene contains action;

comparing the at least one quality metric value of images within each section; and

selecting, for each of the two or more sections, at least one image as a keeper image based on the comparison of the at least one quality metric value; and

selecting, based on a determination that the scene is not an action scene, a keeper image based on a second technique.

18. The method of claim 17, wherein dividing the temporal sequence of images is based on detecting a scene change.

19. The method of claim 17, further comprises:

calculating a distance between images of the temporal sequence of images, wherein the division of the temporal sequence of images is based on the calculated distance between the images.

20. The method of claim 19, wherein dividing the temporal sequence of images is based on a ratio of a first average distance between images within a section to a second average distance between images from different sections.

23. The method of claim 17, wherein the second technique comprises a face detection technique.