Summarizing video content based on memorability of the video content

• US 10,311,913 B1
• Filed: 02/22/2018
• Issued: 06/04/2019
• Est. Priority Date: 02/22/2018
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Shekhar, Sumit, Singh, Harvineet, Singal, Dhruv, Sinha, Atanu R.
• Primary Examiner(s): Topgyal, Gelek W
• Application Number:
  US15/902,046
• Time in Patent Office: 467 Days
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Certain embodiments involve generating summarized versions of video content based on memorability of the video content. For example, a video summarization system accesses segments of an input video. The video summarization system identifies memorability scores for the respective segments. The video summarization system selects a subset of segments from the segments based on each computed memorability score in the subset having a threshold memorability score. The video summarization system generates visual summary content from the subset of the segments.

Claims

1. A method for summarizing video content based on memorability of the video content, the method performed by one or more processing devices and comprising:

accessing segments of an input video;

computing memorability scores for the segments, respectively, wherein computing a memorability score for a segment comprises;

generating (i) a semantic feature computed from an auto-captioning operation applied to the segment and (ii) a visual feature computed from one or more of a saliency analysis operation applied to the segment, a color analysis operation applied to the segment, and a spatio-temporal analysis operation applied to the segment,

computing a first component score by applying a first predictor to the semantic feature, where the first predictor is trained to determine first component memorability scores by comparing user-generated memorability values with training semantic features generated by the auto-captioning operation,

computing a second component score by applying a second predictor to the semantic feature, where the second predictor is trained to determine second component memorability scores by comparing the user-generated memorability values with training visual features generated by the one or more of the saliency analysis operation, the color analysis operation, and the spatio-temporal analysis operation, and

computing the memorability score from an averaging operation applied to the first component score and the second component score;

selecting a subset of segments from the segments based on each computed memorability score in the subset having a threshold memorability score; and

generating visual summary content from the subset of the segments.

2. The method of claim 1, further comprising partitioning the input video into the segments prior to accessing the input video.

3. The method of claim 1, wherein selecting the subset of the segments based on each computed memorability score in the subset having the threshold memorability score comprises:

ranking the segments according to the computed memorability scores, wherein the ranking applies a first rank to a first segment and a second rank to a second segment;

determining that the first rank is greater than the second rank;

including the first segment in the subset of the segments; and

excluding the second segment from the subset of the segments.

4. The method of claim 1, wherein generating the visual summary content comprises:

identifying a summary length for the visual summary content;

selecting, from the subset of segments, a summary subset of segments having a combined length that is less than or equal to the summary length, wherein the summary subset includes a smaller number of segments than the subset of segments.

5. The method of claim 4, wherein selecting the summary subset comprises:

determining that the summary subset (i) is less than or equal to the summary length and (ii) maximizes a sum of criteria scores for respective segments in the summary subset, wherein a criteria score comprises a memorability score for a segment weighted by a memorability weight and an additional video metric weighted by an additional video metric weight, wherein the additional video metric comprises one or more of video uniformity and video representativeness; and

selecting the summary subset based on determining that the summary subset maximizes the sum of criteria scores and is less than or equal to the summary length.

6. The method of claim 1, wherein generating the visual summary content comprises combining the subset of the segments into a preview video that is included in the visual summary content.

7. The method of claim 1, wherein the visual summary content comprises a set of thumbnail images, wherein generating the set of thumbnail images comprises:

computing visual quality scores for the subset of the segments;

extracting thumbnail images from the subset of the segments based on the visual quality scores; and

selecting the extracted thumbnail images as the set of thumbnail images.

8. A system comprising:

a processing device; and

a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program stored in the non-transitory computer-readable medium and thereby perform operations comprising;

identifying a summary length for a visual summary content to be generated using input video segments;

determining that a summary subset of the input video segments (i) has a combined length that is less than or equal to the summary length and (ii) maximizes a sum of criteria scores for respective segments in the summary subset, wherein at least one criteria score comprises a memorability score for an input video segment weighted by a memorability weight and an additional video metric weighted by an additional video metric weight, wherein the additional video metric comprises one or more of video uniformity and video representativeness,

selecting the summary-a subset of the input video segments based on determining that the summary subset maximizes the sum of criteria scores and is less than or equal to the summary length, and

generating the visual summary content from the summary subset of the input video segments.

9. The system of claim 8, the operations further comprising selecting the summary subset of the input video segments based on each memorability score in the summary subset having a threshold memorability score.

10. The system of claim 9, wherein selecting the summary subset of the input video segments based on each memorability score in the summary subset having the threshold memorability score comprises:

ranking the input video segments according to the memorability scores, wherein the ranking applies a first rank to a first segment and a second rank to a second segment;

determining that the first rank is greater than the second rank;

including the first segment in the summary subset of the input video segments; and

excluding the second segment from the summary subset of the input video segments.

11. The system of claim 8, wherein generating the visual summary content comprises combining the summary subset of the input video segments into a preview video that is included in the visual summary content.

12. A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising:

accessing segments of an input video;

computing memorability scores for the segments, respectively, wherein computing a memorability score for a segment comprises;

generating (i) a semantic feature computed from an auto-captioning operation applied to the segment and (ii) a visual feature computed from one or more of a saliency analysis operation applied to the segment, a color analysis operation applied to the segment, and a spatio-temporal analysis operation applied to the segment,

computing a first component score by applying a first predictor to the semantic feature, where the first predictor is trained to determine first component memorability scores by comparing user-generated memorability values with training semantic features generated by the auto-captioning operation,

computing a second component score by applying a second predictor to the semantic feature, where the second predictor is trained to determine second component memorability scores by comparing the user-generated memorability values with training visual features generated by the one or more of the saliency analysis operation, the color analysis operation, and the spatio-temporal analysis operation, and

computing the memorability score from an averaging operation applied to the first component score and the second component score;

a step for selecting a subset of segments from the segments based on each computed memorability score in the subset having a threshold memorability score; and

generating visual summary content from the subset of the segments.

13. The non-transitory computer-readable medium of claim 12, the operations further comprising partitioning the input video into the segments prior to accessing the input video.

14. The non-transitory computer-readable medium of claim 12, wherein selecting the subset of the segments based on each computed memorability score in the subset having the threshold memorability score comprises:

ranking the segments according to the computed memorability scores, wherein the ranking applies a first rank to a first segment and a second rank to a second segment;

determining that the first rank is greater than the second rank;

including the first segment in the subset of the segments; and

excluding the second segment from the subset of the segments.

15. The non-transitory computer-readable medium of claim 12, wherein generating the visual summary content comprises:

identifying a summary length for the visual summary content;

selecting, from the subset of segments, a summary subset of segments having a combined length that is less than or equal to the summary length, wherein the summary subset includes a smaller number of segments than the subset of segments.

16. The non-transitory computer-readable medium of claim 15, wherein selecting the summary subset comprises:

determining that the summary subset (i) is less than or equal to the summary length and (ii) maximizes a sum of criteria scores for respective segments in the summary subset, wherein a criteria score comprises a memorability score for a segment weighted by a memorability weight and an additional video metric weighted by an additional video metric weight, wherein the additional video metric comprises one or more of video uniformity and video representativeness;

selecting the summary subset based on determining that the summary subset maximizes the sum of criteria scores and is less than or equal to the summary length.

17. The non-transitory computer-readable medium of claim 12, wherein the visual summary content comprises a set of thumbnail images, wherein generating the set of thumbnail images comprises:

computing visual quality scores for the subset of the segments;

extracting thumbnail images from the subset of the segments based on the visual quality scores; and

selecting the extracted thumbnail images as the set of thumbnail images.