Systems and methods for generating a time lapse video

• US 9,953,679 B1
• Filed: 05/24/2016
• Issued: 04/24/2018
• Est. Priority Date: 05/24/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Tse, Daniel, Chik, Desmond, Wills, Jonathan, Getachew, Mahlet, Singh, Rajvikram
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Tekle, Daniel
• Application Number:
  US15/163,637
• Time in Patent Office: 700 Days
• Field of Search: 386241
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/002 Programmed access in sequence to a plurality of record carriers or indexed parts, e.g. tracks, thereof, e.g. for editing; transfer of record carriers from magazine G11B15/68, G11B17/10; G11B17/22
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

Images may be extracted from a video. The images may be grouped into image groups. Numbers and types of classified visuals within the images may be detected. Individual types of classified visuals may correspond to individual classification weights. Image classification weights for the images may be determined based on the numbers and the types of classified visuals and the individual classification weights. Interest weights for the images may be determined based on the image classification weights and the sizes of the image groups to which the individual images belong. An interest curve may be generated based on the interest weights. A retime curve may be generated based on the interest curve. Time lapse images to be included in the time lapse video may be determined based on the images and the retime curve. The time lapse video may be generated based on the time lapse images.

Claims

1. A system for generating a time lapse video, the system comprising:

one or more physical processors configured by machine readable instructions to;

access a video;

extract images from the video;

group the images into image groups, individual image groups having individual sizes defined by numbers of the images in the individual image groups and including similar and sequential images such that a first image group having a first size includes a first image and a second image, the first image similar to the second image and sequential with the second image;

detect numbers and types of classified visuals within the images, individual types of classified visuals corresponding to individual classification weights, such that a first number of a first type of classified visual within the first image is detected, the first type of classified visual corresponding to a first classification weight;

determine image classification weights for the images based on the numbers and the types of classified visuals detected within the individual images and the individual classification weights such that a first image classification weight is determined for the first image based on the first number of the first type of classified visual and the first classification weight;

determine interest weights for the images based on the image classification weights for the individual images and the sizes of the image groups to which the individual images belong, such that a first interest weight is determined for the first image based on the first image classification weight and the first size;

generate an interest curve for the images based on the interest weights such that a value of the interest curve at a point corresponding to the first image is based on the first interest weight;

generate a retime curve for the images based on the interest curve, the retime curve defining perceived speeds at which the time lapse video is displayed during playback such that the retime curve defines a first perceived speed at which a portion of the time lapse video corresponding to the first image is displayed during playback;

determine time lapse images to be included in the time lapse video based on the images and the retime curve; and

generate the time lapse video based on the time lapse images.

2. The system of claim 1, wherein the video is characterized by an encoded framerate and the images are extracted from the video at the encoded framerate.

3. The system of claim 1, wherein the one or more physical processors are further configured by machine readable instructions to downsample the images.

4. The system of claim 1, wherein the one or more physical processors are further configured by machine readable instructions to stabilize the images based on a feature point detection.

5. The system of claim 1, wherein grouping the images into the image groups includes determining image similarity based on comparisons of feature vectors of sequential pairs of the individual images, the feature vectors including a first feature vector characterizing a local binary pattern distribution of the individual images and a second feature vector characterizing a color distribution of the individual images.

6. The system of claim 1, wherein the numbers and the types of classified visuals within the images are detected based on a convolutional neural network based scene and/or object classifier.

7. The system of claim 1, wherein an inverse relationship exists between the interest weights for the images and the sizes of the image groups to which the individual images belong.

8. A method for generating a time lapse video, the method performed by a computing system including one or more physical processors, the method comprising:

accessing, by the computing system, a video;

extracting, by the computing system, images from the video;

grouping, by the computing system, the images into image groups, individual image groups having individual sizes defined by numbers of the images in the individual image groups and including similar and sequential images such that a first image group having a first size includes a first image and a second image, the first image similar to the second image and sequential with the second image;

detecting, by the computing system, numbers and types of classified visuals within the images, individual types of classified visuals corresponding to individual classification weights, such that a first number of a first type of classified visual within the first image is detected, the first type of classified visual corresponding to a first classification weight;

determining, by the computing system, image classification weights for the images based on the numbers and the types of classified visuals detected within the individual images and the individual classification weights such that a first image classification weight is determined for the first image based on the first number of the first type of classified visual and the first classification weight;

determining, by the computing system, interest weights for the images based on the image classification weights for the individual images and the sizes of the image groups to which the individual images belong, such that a first interest weight is determined for the first image based on the first image classification weight and the first size;

generating, by the computing system, an interest curve for the images based on the interest weights such that a value of the interest curve at a point corresponding to the first image is based on the first interest weight;

generating, by the computing system, a retime curve for the images based on the interest curve, the retime curve defining perceived speeds at which the time lapse video is displayed during playback such that the retime curve defines a first perceived speed at which a portion of the time lapse video corresponding to the first image is displayed during playback;

determining, by the computing system, time lapse images to be included in the time lapse video based on the images and the retime curve; and

generating, by the computing system, the time lapse video based on the time lapse images.

9. The method of claim 8, wherein the video is characterized by an encoded framerate and the images are extracted from the video at the encoded framerate.

10. The method of claim 8, further comprising downsampling, by the computing system, the images.

11. The method of claim 8, further comprising stabilizing, by the computing system, the images based on a feature point detection.

12. The method of claim 8, wherein grouping the images into the image groups includes determining image similarity based on comparisons of feature vectors of sequential pairs of the individual images, the feature vectors including a first feature vector characterizing a local binary pattern distribution of the individual images and a second feature vector characterizing a color distribution of the individual images.

13. The method of claim 8, wherein the numbers and the types of classified visuals within the images are detected based on a convolutional neural network based scene and/or object classifier.

14. The method of claim 8, wherein an inverse relationship exists between the interest weights for the images and the sizes of the image groups to which the individual images belong.

15. A system for generating a time lapse video, the system comprising:

one or more physical processors configured by machine readable instructions to;

access a video, the video characterized by an encoded framerate;

extract images from the video at the encoded framerate;

downsample the images;

stabilize the images based on a feature point detection;

group the images into image groups, individual image groups having individual sizes defined by numbers of the images in the individual image groups and including similar and sequential images such that a first image group having a first size includes a first image and a second image, the first image similar to the second image and sequential with the second image;

detect numbers and types of classified visuals within the images, individual types of classified visuals corresponding to individual classification weights, such that a first number of a first type of classified visual within the first image is detected, the first type of classified visual corresponding to a first classification weight;

determine image classification weights for the images based on the numbers and the types of classified visuals detected within the individual images and the individual classification weights such that a first image classification weight is determined for the first image based on the first number of the first type of classified visual and the first classification weight;

determine interest weights for the images based on the image classification weights for the individual images and the sizes of the image groups to which the individual images belong, such that a first interest weight is determined for the first image based on the first image classification weight and the first size;

generate an interest curve for the images based on the interest weights such that a value of the interest curve at a point corresponding to the first image is based on the first interest weight;

generate a retime curve for the images based on the interest curve, the retime curve defining perceived speeds at which the time lapse video is displayed during playback such that the retime curve defines a first perceived speed at which a portion of the time lapse video corresponding to the first image is displayed during playback;

determine time lapse images to be included in the time lapse video based on the images and the retime curve, determining the time lapse images to be included in the time lapse video including modifying at least some of the images to include motion blur based on the interest curve; and

generate the time lapse video based on the time lapse images.

16. The system of claim 15, wherein:

grouping the images into the image groups includes determining image similarity based on comparisons of feature vectors of sequential pairs of the individual images, the feature vectors including a first feature vector characterizing a local binary pattern distribution of the individual images and a second feature vector characterizing a color distribution of the individual images; and

the numbers and the types of classified visuals within the images are detected based on a convolutional neural network based scene and/or object classifier.