Method And Apparatus For Finding And Using Video Portions That Are Relevant To Adjacent Still Images

• US 20160358634A1
• Filed: 09/25/2015
• Published: 12/08/2016
• Est. Priority Date: 06/07/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tira-Thompson, Ethan J., Molgaard, Claus, Keating, Brett M., Williams, George E., Zuliani, Marco, Wong, Vincent Y., Doepke, Frank
• Granted Patent: US 10,170,157 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

The invention relates to systems, methods, and computer readable media for responding to a user snapshot request by capturing anticipatory pre-snapshot image data as well as post-snapshot image data. The captured information may be used, depending upon the embodiment, to create archival image information and image presentation information that is both useful and pleasing to a user. The captured information may automatically be trimmed or edited to facilitate creating an enhanced image, such as a moving still image. Varying embodiments of the invention offer techniques for trimming and editing based upon the following: exposure, brightness, focus, white balance, detected motion of the camera, substantive image analysis, detected sound, image metadata, and/or any combination of the foregoing.

Claims

1. ) A method of producing an enhanced image comprising:

capturing an archive of image data comprising a plurality of image frames segmented as a pre-capture image sequence, a snapshot, and a post-capture image sequence, wherein the archival image data is captured by;

capturing the pre-capture image sequence,

receiving a snapshot indication through a user interface, the snapshot indication received temporally after capturing at least a portion of the pre-capture image sequence,

in response to the snapshot indication, capturing the snapshot image,

capturing the post-capture image sequence, wherein at least a portion of the post-capture image sequence is captured temporally after receiving the snapshot indication, and

capturing contextual information associated with the pre-capture image sequence, the snapshot, and the post-capture image sequence,

trimming or flagging the image frames of the pre-capture image sequence and the post-capture image sequence based upon an analysis of the contextual information and combined image frames of the archival image data, wherein the analysis comprises;

developing a target brightness level and an average brightness level,

for each image frame, computing a ratio of the target brightness level with the average brightness level,

trimming or flagging each pre-capture image frame and each post-capture image frame based upon a comparison of a ratio threshold to the image frame'"'"'s ratio of target brightness level to average brightness level,

determining a velocity or acceleration associated with each image frame,

setting a threshold for velocity or acceleration based upon the velocity or acceleration of the snapshot image frame, and

trimming or flagging each pre-capture image frame based upon a comparison of its velocity or acceleration to the threshold for velocity or acceleration, and

creating an enhanced image based upon the archival image data and the analysis.

2. ) The method of claim 1, wherein the target brightness level, and the average brightness level are metadata received from an auto exposure system.

3. ) The method of claim 1, wherein the ratio threshold is set independently for each image frame.

4. ) The method of claim 1, wherein the pre-capture image data begins to be captured upon the activation of a camera application.

5. ) The method of claim 1, wherein the contextual information includes one or more of the following:

sensor output data;

application status data;

data indicating the presence of a face in an image frame; and

camera metadata.

6. ) The method of claim 1, wherein the analysis includes, for each image frame, determining whether a loud noise is associated with the image frame and flagging or trimming the image frame based upon the determination, and wherein flagging the image frame includes scoring the image frame.

7. ) The method of claim 1, wherein the analysis comprises:

determining a multi-factor indicator that includes a camera travel distance indicator derived from image frame registration techniques, a blurriness indicator, and an image quality assessment indicator derived based upon image frame registration techniques;

normalizing each of the camera travel distance indicator, the blurriness indicator, and the image quality assessment indicator;

combining the normalized indicators to compute a value for the multi-factor indicator; and

flagging or trimming each image frame based upon the value of the multi-factor indicator.

8. ) The method of claim 7, wherein the archival image data comprises a video file and an associated still image file, and wherein the method further comprises displaying the still image, and upon receipt of a user indication, finding the associated video file and playing the associated video file.

9. ) A computer readable medium comprising one or more instructions that, when executed, configure a processor to:

capture an archive of image data comprising a plurality of image frames segmented as a pre-capture image sequence, a snapshot, and a post-capture image sequence, wherein the instructions that configure the processor to capture the archival image data comprise instructions that configure the processor to;

capture the pre-capture image sequence,

receive a snapshot indication through a user interface, the snapshot indication received temporally after capturing at least a portion of the pre-capture image sequence,

in response to the snapshot indication, capture the snapshot image,

capture the post-capture image sequence, wherein at least a portion of the post capture image sequence is captured temporally after receiving the snapshot indication, and

capture contextual information associated with the pre-capture image sequence, the snapshot, and the post-capture image sequence,

automatically trim or flag the image frames of the pre-capture image sequence and the post-capture image sequence based at least in part upon the contextual information; and

create an enhanced image based upon the trimmed archival image data.

10. ) The computer readable medium of claim 9 wherein, the enhanced image is stored as a video file and an associated still image file.

11. ) The computer readable medium of claim 10, further comprising instructions that configure the processor to play the enhanced image by displaying the snapshot, finding the associated video file, and playing the associated video file.

12. ) The computer readable medium of claim 9, wherein the pre-capture image data begins to be captured upon the activation of a camera application.

13. ) The computer readable medium of claim 9, wherein the contextual information includes one or more of the following:

sensor output data;

application status data;

data indicating the presence of a face in an image frame;

camera metadata;

data indicating the presence of animals;

scene classification information; and

information accessed over the Internet.

14. ) The computer readable medium of claim 9, further comprising instructions that configure the processor to, for a plurality of image frames, determine whether a loud noise is associated with the image frame and flag or trim the image frame based upon the determination.

15. ) The computer readable medium of claim 9, further comprising instructions that configure the processor to:

determine a multi-factor indicator that includes a camera travel distance indicator derived from image frame registration techniques, a blurriness indicator, and an image quality assessment indicator derived based upon image frame registration techniques;

normalize each of the camera travel distance indicator, the blurriness indicator, and the image quality assessment indicator;

combine the normalized indicators to compute a value for the multi-factor indicator; and

flag or trim a plurality of image frames based upon the value of the multi-factor indicator.

16. ) A system comprising:

one or more CPUs;

one or more cameras for capturing an archive of image data comprising a plurality of image frames segmented as a pre-capture image sequence, a snapshot, and a post-capture image sequence;

a memory for storing program instructions for the one or more CPUs, where the instructions, when executed, cause the one or more CPUs to;

capture the pre-capture image sequence,

receive a snapshot indication, the snapshot indication received temporally after capturing at least a portion of the pre-capture image sequence,

in response to the snapshot indication, capture the snapshot image,

capture the post-capture image sequence, wherein at least a portion of the post capture image sequence is captured temporally after receiving the snapshot indication, and

capture contextual information associated with the pre-capture image sequence, the snapshot, and the post-capture image sequence,

trim or flag a plurality of the image frames of the pre-capture image sequence and the post-capture image sequence, wherein the instructions that cause the one or more CPUs to trim or flag the pre-capture image sequence and the post-capture image sequence comprise instructions that cause the one or more CPUs to trim or flag a plurality of pre-capture image frames and a plurality of post-capture image frames based upon the contextual information.

17. ) The system of claim 16, wherein the instructions, when executed, further cause the one or more CPUs to create an enhanced image.

18. ) The system of claim 17, wherein the instructions, when executed, further cause the one or more CPUs to store the enhanced image as an image file and a video file.

19. ) The system of claim 18, wherein the instructions, when executed, further cause the one or more CPUs to play the enhanced image by displaying the contents of the image file, followed by transitioning to playing the video file, followed by transitioning back to displaying the contents to the image file.

20. ) The system of claim 16, wherein the instructions, when executed, further cause the one or more CPUs to:

determine a multi-factor indicator that includes a camera travel distance indicator derived from image frame registration techniques, a blurriness indicator, and an image quality assessment indicator derived based upon image frame registration techniques;

normalize each of the camera travel distance indicator, the blurriness indicator, and the image quality assessment indicator;

combine the normalized indicators to compute a value for the multi-factor indicator; and

flag or trim a plurality of image frames based upon the value of the multi-factor indicator.