Time-lapse video capture with optimal image stabilization

• US 9,426,409 B2
• Filed: 02/03/2015
• Issued: 08/23/2016
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Beysserie, Sebastien X., Klivington, Jason, Toft, Rolf, Doepke, Frank
• Primary Examiner(s): LAM, HUNG H
• Application Number:
  US14/613,133
• Publication Number:
  US 20160094801A1
• Time in Patent Office: 567 Days
• Field of Search: 348/222.1, 348/259
• US Class Current: 1/1
• CPC Class Codes: G11B 15/1875 adaptations for special effects or editing signal processing or indexing therefor G11B27/00
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 5/915 for field- or frame-skip recording or reproducing

Abstract

Traditionally, time-lapse videos are constructed from images captured at time intervals called “temporal points of interests” or “temporal POIs.” Disclosed herein are systems and methods of constructing improved, motion-stabilized time-lapse videos using temporal points of interest and image similarity comparisons. According to some embodiments, a “burst” of images may be captured, centered around the aforementioned temporal points of interest. Then, each burst sequence of images may be analyzed, e.g., by performing an image similarity comparison between each image in the burst sequence and the image selected at the previous temporal point of interest. Selecting the image from a given burst that is most similar to the previous selected image, while minimizing the amount of motion with the previous selected image, allows the system to improve the quality of the resultant time-lapse video by discarding “outlier” or other undesirable images captured in the burst sequence and motion stabilizing the selected image.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more processing units to:

acquire a first digital image from an image sensor at a first time;

acquire a plurality of second digital images from the image sensor, wherein acquisition times of the plurality of second digital images are temporally centered around a determined second time;

determine a similarity value for each of the plurality of second digital images, wherein the similarity values comprise an assessment of the similarity of a respective image from the plurality of second digital images and the first digital image;

discard, from the plurality of second digital images, those images having a similarity value below a threshold value;

select, from among the non-discarded images of the plurality of second digital images, a second digital image that minimizes a motion difference with the first digital image; and

combine the first digital image and the selected second digital image into a time lapse video.

2. The non-transitory program storage device of claim 1, wherein the similarity values are determined based, at least in part, on an evaluation of at least one of the following characteristics of a second digital image:

a color histogram, an exposure level, a focus level, a feature detection process, an image registration process, a pixel sum process, a Δ

E process, an auto focus oscillation level, and an average luminance level.

3. The non-transitory program storage device of claim 1, wherein the instructions to discard, from the plurality of second digital images, those images having a similarity value below a threshold value further comprise instructions to:

delete each discarded digital image from the plurality of second digital images before causing the one or more processing units to combine the first digital image and the selected second digital image into a time lapse video.

4. The non-transitory program storage device of claim 1, further comprising instructions stored thereon to cause the one or more processing units to:

apply image stabilization to the selected second digital image.

5. The non-transitory program storage device of claim 4, wherein a strength of the image stabilization applied to the selected second digital image is based, at least in part, on an estimated amount of motion blur in the selected second digital image.

6. The non-transitory program storage device of claim 1, further comprising instructions stored thereon to cause the one or more processing units to:

weight at least one of the determined similarity values for a respective second digital image based, at least in part, on a content of the respective second digital image.

7. The non-transitory program storage device of claim 1, further comprising instructions stored thereon to cause the one or more processing units to:

weight at least one of the determined similarity values for a respective second digital image based, at least in part, on a time of capture of the respective second digital image with respect to the determined second time.

8. A computer-implemented method, comprising:

acquiring a first digital image from an image sensor at a first time;

acquiring a plurality of second digital images from the image sensor, wherein acquisition times of the plurality of second digital images are temporally centered around a determined second time;

determining a similarity value for each of the plurality of second digital images, wherein the similarity values comprise an assessment of the similarity of a respective image from the plurality of second digital images and the first digital image;

discarding, from the plurality of second digital images, those images having a similarity value below a threshold value;

selecting, from among the non-discarded images of the plurality of second digital images, a second digital image that minimizes a motion difference with the first digital image; and

combining the first digital image and the selected second digital image into a time lapse video.

9. The method of claim 8, wherein the similarity values are determined based, at least in part, on an evaluation of at least one of the following characteristics of a second digital image:

a color histogram, an exposure level, a focus level, a feature detection process, an image registration process, a pixel sum process, a Δ

E process, an auto focus oscillation level, and an average luminance level.

10. The method of claim 8, wherein the act of discarding, from the plurality of second digital images, those images having a similarity value below a threshold value further comprises:

deleting each discarded digital image from the plurality of second digital images.

11. The method of claim 8, further comprising applying image stabilization to the selected second digital image, wherein a strength of the image stabilization applied to the selected second digital image is based, at least in part, on an estimated amount of motion blur in the selected second digital image.

12. The method of claim 8, further comprising weighting at least one of the determined similarity values for a respective second digital image based, at least in part, on a content of the respective second digital image.

13. The method of claim 8, further comprising weighting at least one of the determined similarity values for a respective second digital image based, at least in part, on a time of capture of the respective second digital image with respect to the determined second time.

14. An electronic device comprising:

an image sensor,

a memory operatively coupled to the image sensor, and

at least one processor operatively coupled to the memory and the image sensor, wherein the at least one processor is programmed to;

cause the image sensor to acquire a first digital image at a first time;

cause the image sensor to acquire a plurality of second digital images, wherein acquisition times of the plurality of second digital images are temporally centered around a determined second time;

determine a similarity value for each of the plurality of second digital images, wherein the similarity values comprise an assessment of the similarity of a respective image from the plurality of second digital images and the first digital image;

discard, from the plurality of second digital images, those images having a similarity value below a threshold value;

select, from among the non-discarded images of the plurality of second digital images, a second digital image that minimizes a motion difference with the first digital image; and

combine the first digital image and the selected second digital image into a time lapse video.

15. The electronic device of claim 14, wherein the similarity values are determined based, at least in part, on an evaluation of at least one of the following characteristics of a second digital image:

a color histogram, an exposure level, a focus level, a feature detection process, an image registration process, a pixel sum process, a Δ

E process, an auto focus oscillation level, and an average luminance level.

16. The electronic device of claim 14, wherein the instructions to discard, from the plurality of second digital images, those images having a similarity value below a threshold value that the at least one processor is programmed to perform further comprise instructions to:

delete each discarded digital image from the plurality of second digital images before the at least one processor combines the first digital image and the selected second digital image into a time lapse video.

17. The electronic device of claim 14, wherein the at least one processor is further programmed to:

apply image stabilization to the selected second digital image.

18. The electronic device of claim 17, wherein a strength of the image stabilization applied to the selected second digital image is based, at least in part, on an estimated amount of motion blur in the selected second digital image.

19. The electronic device of claim 14, wherein the at least one processor is further programmed to:

weight at least one of the determined similarity values for a respective second digital image based, at least in part, on a content of the respective second digital image.

20. The electronic device of claim 14, wherein the at least one processor is further programmed to:

weight at least one of the determined similarity values for a respective second digital image based, at least in part, on a time of capture of the respective second digital image with respect to the determined second time.