Systems and methods for generating a video summary

• US 9,953,224 B1
• Filed: 08/23/2016
• Issued: 04/24/2018
• Est. Priority Date: 08/23/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Wills, Jonathan, Tse, Daniel, Chik, Desmond, Schunck, Brian
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Wendmagegn, Girumsew
• Application Number:
  US15/244,690
• Time in Patent Office: 609 Days
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

Systems and method of generating video summaries are presented herein. Information defining a video may be obtained. The video may include a set of frame images. Parameter values for parameters of individual frame images of the video may be determined. Interest weights for the frame images may be determined. An interest curve for the video that characterizes the video by interest weights as a function of progress through the set of frame images may be generated. One or more curve attributes of the interest curve may be identified and one or more interest curve values of the interest curve that correspond to individual curve attributes may be determined. Interest curve values of the interest curve may be compared to threshold curve values. A subset of frame images of the video to include within a video summary of the video may be identified based on the comparison.

Claims

1. A system configured to generate a video summary, the system comprising:

one or more physical computer processors configured by computer readable instructions to;

obtain information defining a video, the video including a set of frame images;

determine individual parameter values for one or more parameters of individual frame images of the video, the parameter values including a first set of parameter values for a first parameter of the individual frame images in the set of frame images;

determine individual interest weights for the individual frame images based on the individual parameter values of individual parameters of the individual frame images, the interest weights including a first set of interest weights for the set of frame images determined from the first set of parameter values;

wherein individual interest weights of individual frame images are determined from multiple parameter values of multiple parameters of the individual frame images; and

wherein determining an individual interest weight of an individual frame image comprises determining impact of individual ones of the one or more parameter values on the individual interest weight;

generate an interest curve for the video based on the individual interest weights of the individual frame images, wherein the interest curve characterizes the video by the interest weights as a function of progress through the set of frame images, such that the interest curve is comprised of a set of interest curve values that individually correspond to individual ones of the interest weights in the first set of interest weights and individual ones of the frame images in the set of frame images;

identify one or more curve attributes of the interest curve, one or more curve attributes including a first curve attribute;

determine individual interest curve values of the interest curve that correspond to the individual curve attributes, a first set of interest curve values corresponding to the first curve attribute;

compare the individual interest curve values of the interest curve that correspond to individual curve attributes to individual threshold curve values, individual threshold curve values corresponding to individual curve attributes, such that the first set of interest curve values is compared to one or more threshold values corresponding to the first curve attribute;

identify a subset of frame images of the video to include within a video summary of the video based on the comparison; and

generate the video summary using the subset of frame images.

2. The system of claim 1, wherein individual frame images of the video are included within the video summary based on individual interest curve values of the interest curve that correspond to the individual frame images being greater than individual threshold curve values, such that a first subset of frame images corresponding to the first set of interest curve values is include in the video summary based on the first set of interest curve values being greater than the one or more threshold curve values corresponding to the first curve attribute.

3. The system of claim 1, wherein the one or more curve attributes include one or more of an overall shape, an infection point, a maximum, a minimum, a rate of change, or a rate of the rage of change.

4. The system of claim 1, wherein individual threshold curve values for individual curve attributes specific to content of the video.

5. The system of claim 1, wherein the parameters include one or more of a scene parameter, a sharpness parameter, a motion parameter, an exposure parameter, a saturation parameter, or an audio parameter.

6. The system of claim 1, wherein determining the impact of individual ones of the parameter values on the individual interest weight is based on a convolutional neural network.

7. The system of claim 1, wherein the information defining the video includes metadata associated with the video, such that determining individual parameter values for one or more image parameters of individual frame images of the video is further based on the metadata.

8. The system of claim 7, wherein metadata associated with the video includes one or more of capture settings of a capture device, sensor output of one or more sensors coupled to the capture device, or user-provided information.

9. A method of generating a video summary, the method being implemented in a computer system comprising one or more physical processors and storage media storing machine-readable instructions, the method comprising:

obtaining information defining a video, the video including a set of frame images;

determining individual parameter values for one or more parameters of individual frame images of the video, the parameter values including a first set of parameter values for a first parameter of the individual frame images in the set of frame images;

determining interest weights for the individual frame images based on individual parameter values of individual parameters of the individual frame images, including determining a first set of interest weights for the set of frame images from the first set of parameter values;

wherein individual interest weights of individual frame images are determined from multiple parameter values of multiple parameters of the individual frame images; and

wherein determining an individual interest weight of an individual frame image comprises determining impact of individual ones of the one or more parameter values on the individual interest weight;

generating an interest curve for the video based on the individual interest weights of the individual frame images, wherein the interest curve characterizes the video by interest weights as a function of progress through the set of frame images of the video, such that the interest curve is comprised of a set of interest curve values that individually correspond to individual ones of the interest weights in the first set of interest weights and individual ones of the frame images in the set of frame images;

identifying one or more curve attributes of the interest curve, one or more curve attributes including a first curve attribute;

determining individual interest curve values of the interest curve that correspond to individual curve attributes, including determining a first set of interest curve values corresponding to the first curve attribute;

comparing the individual interest curve values of the interest curve that correspond to individual curve attributes to individual threshold curve values, individual threshold curve values corresponding to individual curve attributes, including comparing the first set of interest curve values to one or more threshold values for the first curve attribute;

identifying a subset of frame images of the video to include within a video summary of the video based on the comparison; and

generating the video summary using the subset of frame images.

10. The method of claim 9, wherein individual frame images of the video are included within the video summary based on individual interest curve values of the interest curve that correspond to the individual frame images being greater than individual threshold curve values, such that a first subset of frame images corresponding to the first set of interest curve values is include in the video summary based on the first set of interest curve values being greater than the one or more threshold curve values for the first curve attribute.

11. The method of claim 9, wherein the one or more curve attributes include one or more of an overall shape, an infection point, a maximum, a minimum, a rate of change, or a rate of the rage of change.

12. The method of claim 9, wherein individual threshold curve values for individual curve attributes specific to content of the video.

13. The method of claim 9, wherein the parameters include one or more of a scene parameter, a sharpness parameter, a motion parameter, an exposure parameter, a saturation parameter, or an audio parameter.

14. The method of claim 9, wherein determining the impact of individual ones of the parameter values on the individual interest weight is based on a convolutional neural network.

15. The method of claim 9, wherein the information defining the video includes metadata associated with the video, such that determining individual parameter values for one or more image parameters of individual frame images of the video is further based on the metadata.

16. The method of claim 15, wherein metadata associated with the video includes one or more of capture settings of a capture device, sensor output of one or more sensors coupled to the capture device, or user-provided information.