Method and apparatus for finding and using video portions that are relevant to adjacent still images

• US 10,170,157 B2
• Filed: 09/25/2015
• Issued: 01/01/2019
• Est. Priority Date: 06/07/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Molgaard, Claus, Keating, Brett M., Williams, George E., Zuliani, Marco, Wong, Vincent Y., Doepke, Frank, Tira-Thompson, Ethan J.
• Primary Examiner(s): Flohre, Jason A
• Application Number:
  US14/865,752
• Publication Number:
  US 20160358634A1
• Time in Patent Office: 1,194 Days
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

The invention relates to systems, methods, and computer readable media for responding to a user snapshot request by capturing anticipatory pre-snapshot image data as well as post-snapshot image data. The captured information may be used, depending upon the embodiment, to create archival image information and image presentation information that is both useful and pleasing to a user. The captured information may automatically be trimmed or edited to facilitate creating an enhanced image, such as a moving still image. Varying embodiments of the invention offer techniques for trimming and editing based upon the following: exposure, brightness, focus, white balance, detected motion of the camera, substantive image analysis, detected sound, image metadata, and/or any combination of the foregoing.

Claims

1. A method of producing an enhanced image comprising:

capturing a first plurality of images, wherein the first plurality of images comprises;

a snapshot image, wherein the snapshot image is captured in response to receiving a snapshot indication through a user interface;

a pre-capture image sequence, wherein the pre-capture image sequence comprises a plurality of images captured prior to capturing the snapshot image, and wherein the snapshot indication is received after capturing at least one of the plurality of images of the pre-capture image sequence; and

a post-capture image sequence, wherein the post-capture image sequence comprises a plurality of images captured after capturing the snapshot image;

obtaining motion information for each of the first plurality of images;

determining, for each image of the pre-capture image sequence and the post-capture image sequence, whether the motion information of a respective image meets a first motion criterion, wherein the first motion criterion comprises a comparison between the motion information of the respective image and the motion information of the snapshot image;

identifying a first subset of images from among the pre-capture image sequence that meet the first motion criterion;

identifying a second subset of images from among the post-capture image sequence that meet the first motion criterion; and

creating an enhanced image comprising the snapshot image, the first subset of images, and the second subset of images.

2. The method of claim 1, wherein the first subset of images comprises a consecutive set of images, beginning with an image in the pre-capture image sequence and ending with the image captured immediately prior to the snapshot image.

3. The method of claim 1, wherein the second subset of images comprises a consecutive set of images, beginning with the image captured immediately after the snapshot image and ending with an image in the post-capture image sequence.

4. The method of claim 1, wherein the motion information for at least one of the first plurality of images comprises information obtained from an accelerometer or information obtained from a gyroscope.

5. The method of claim 1, wherein the motion information for at least one of the first plurality of images comprises information associated with a direction of motion for the at least one of the first plurality of images.

6. The method of claim 1, wherein the first motion criterion further comprises determining whether a direction of the motion information for an image in the first plurality of images is consistent with a direction of the motion information for the snapshot image.

7. The method of claim 1, wherein the first motion criterion further comprises determining whether a direction of the motion information for an image in the first plurality of images is consistent with a direction of the motion information for one or more other images in the first plurality of images.

8. The method of claim 1, wherein the first motion criterion further comprises a difference between the motion information of the respective image and the motion information of the snapshot image being less than a predetermined threshold amount.

9. A non-transitory computer readable medium having stored thereon a program, the program comprising one or more instructions that, when executed by a processor, cause the processor to:

capture a first plurality of images, wherein the first plurality of images comprises;

a snapshot image, wherein the snapshot image is captured in response to receiving a snapshot indication through a user interface;

a pre-capture image sequence, wherein the pre-capture image sequence comprises a plurality of images captured prior to capturing the snapshot image, and wherein the snapshot indication is received after capturing at least one of the plurality of images of the pre-capture image sequence; and

a post-capture image sequence, wherein the post-capture image sequence comprises a plurality of images captured after capturing the snapshot image;

obtain motion information for each of the first plurality of images;

determine, for each image of the pre-capture image sequence and the post-capture image sequence, whether the motion information of a respective image meets a first motion criterion, wherein the first motion criterion comprises a comparison between the motion information of the respective image and the motion information of the snapshot image;

identify a first subset of images from among the pre-capture image sequence that meet the first motion criterion;

identify a second subset of images from among the post-capture image sequence that meet the first motion criterion; and

create an enhanced image comprising the snapshot image, the first subset of images, and the second subset of images.

10. The non-transitory computer readable medium of claim 9, wherein the first subset of images comprises a consecutive set of images, beginning with an image in the pre-capture image sequence and ending with the image captured immediately prior to the snapshot image.

11. The non-transitory computer readable medium of claim 9, wherein the second subset of images comprises a consecutive set of images, beginning with the image captured immediately after the snapshot image and ending with an image in the post-capture image sequence.

12. The non-transitory computer readable medium of claim 9, wherein the motion information for at least one of the first plurality of images comprises information obtained from an accelerometer or information obtained from a gyroscope.

13. The non-transitory computer readable medium of claim 9, wherein the motion information for at least one of the first plurality of images comprises information associated with a direction of motion for the at least one of the first plurality of images.

14. The non-transitory computer readable medium of claim 9, wherein the first motion criterion further comprises a difference between the motion information of the respective image and the motion information of the snapshot image being less than a predetermined threshold amount.

15. A system comprising:

one or more CPUs;

one or more cameras for capturing images; and

a memory for storing program instructions for the one or more CPUs, wherein the instructions, when executed, cause the one or more CPUs to;

capture a first plurality of images, wherein the first plurality of images comprises;

a snapshot image, wherein the snapshot image is captured in response to receiving a snapshot indication through a user interface;

a pre-capture image sequence, wherein the pre-capture image sequence comprises a plurality of images captured prior to capturing the snapshot image, and wherein the snapshot indication is received after capturing at least one of the plurality of images of the pre-capture image sequence; and

a post-capture image sequence, wherein the post-capture image sequence comprises a plurality of images captured after capturing the snapshot image;

obtain motion information for each of the first plurality of images;

determine, for each image of the pre-capture image sequence and the post-capture image sequence, whether the motion information of a respective image meets a first motion criterion, wherein the first motion criterion comprises a comparison between the motion information of the respective image and the motion information of the snapshot image;

identify a first subset of images from among the pre-capture image sequence that meet the first motion criterion;

identify a second subset of images from among the post-capture image sequence that meet the first motion criterion; and

create an enhanced image comprising the snapshot image, the first subset of images, and the second subset of images.

16. The system of claim 15, wherein the first subset of images comprises a consecutive set of images, beginning with an image in the pre-capture image sequence and ending with the image captured immediately prior to the snapshot image.

17. The system of claim 15, wherein the second subset of images comprises a consecutive set of images, beginning with the image captured immediately after the snapshot image and ending with an image in the post-capture image sequence.

18. The system of claim 15, wherein the motion information for at least one of the first plurality of images comprises information obtained from an accelerometer or information obtained from a gyroscope.

19. The system of claim 15, wherein the motion information for at least one of the first plurality of images comprises information associated with a direction of motion for the at least one of the first plurality of images.

20. The system of claim 15, wherein the first motion criterion further comprises a difference between the motion information of the respective image and the motion information of the snapshot image being less than a predetermined threshold amount.