Techniques for enhancing content memorability of user generated video content

• US 10,380,428 B2
• Filed: 09/26/2017
• Issued: 08/13/2019
• Est. Priority Date: 11/20/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Shekhar, Sumit, Angara, Srinivasa Madhava Phaneendra, Kedia, Manav, Singal, Dhruv, Shetty, Akhil Sathyaprakash
• Primary Examiner(s): Dulaney, Kathleen Y
• Application Number:
  US15/715,401
• Publication Number:
  US 20180018523A1
• Time in Patent Office: 686 Days
• Field of Search: 382218, 386241, 707748
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Techniques are described for analyzing a video for memorability, identifying content features of the video that are likely to be memorable, and scoring specific content features within the video for memorability. The techniques can be optionally applied to selected features in the video, thus improving the memorability of the selected features. The features may be organic features of the originally captured video or add-in features provided using an editing tool. The memorability of video features, text features, or both can be improved by analyzing the effects of applying different styles or edits (e.g., sepia tone, image sharpen, image blur, annotation, addition of object) to the content features or to the video in general. Recommendations can then be provided regarding memorability score caused by application of the image styles to the video features.

Claims

1. A computer-implemented method for quantifying memorability of video content, the method comprising:

receiving a video that comprises a plurality of video frames, wherein the video includes a video feature that is associated with a text feature, and wherein the text feature comprises textual content that is visually shown in at least one of the video frames;

identifying the video feature and the associated text feature in the received video;

determining a video feature score corresponding to the video feature, the video feature score indicating memorability of the video feature;

determining a text feature score corresponding to the text feature, the text feature score indicating memorability of the text feature; and

determining a content memorability score that is based on the video feature score and the text feature score.

2. The computer-implemented method of claim 1, further comprising causing display of a memorability map that:

indicates a particular time period associated with a subset of the video frames in which the video feature and the text feature are visible; and

includes a visual indicator corresponding to the content memorability score, wherein the visual indicator links the content memorability score with the particular time period.

3. The computer-implemented method of claim 1, wherein determining the text feature score comprises:

extracting the textual content from the received video;

using a recursive auto-encoder to extract a first semantic meaning from the textual content;

using a deep neural network to identify a semantic description of the video feature;

comparing the first semantic meaning with the identified semantic description of the video feature to determine a text/image meaning similarity; and

normalizing the text/image meaning similarity using a sigmoid function to produce the text feature score.

4. The computer-implemented method of claim 1, wherein determining the video feature score comprises:

determining at least one preliminary video feature score based on one or more of a spatiotemporal analysis of the video feature, a salience analysis of the video feature, and a deep neural network learning analysis of the video feature; and

processing the at least one preliminary video feature score using a regression function to produce the video feature score.

5. The computer-implemented method of claim 1, wherein determining the text feature score comprises:

determining a first semantic meaning associated with the video feature;

determining a second semantic meaning associated with the text feature;

determining a degree of similarity between the first and second semantic meanings; and

determining the text feature score based on the degree of similarity.

6. The computer-implemented method of claim 1, wherein determining the video feature score comprises:

determining a first semantic meaning associated with the video feature; and

determining the video feature score based on the first semantic meaning, wherein the video feature score is determined using a trained deep neural network learning algorithm.

7. The computer-implemented method of claim 1, wherein identifying the video feature comprises:

performing a spatiotemporal analysis of the received video, the spatiotemporal analysis comprising evaluating relative movement between two of the video frames; and

making a determination that the identified video feature has relative movement between two video frames that exceeds a threshold value.

8. The computer-implemented method of claim 1, wherein identifying the video feature comprises:

performing a salience analysis of the received video, the salience analysis comprising identifying a plurality of objects appearing in the received video, and for each of the identified objects, evaluating an object color; and

making a determination that the identified video feature includes a particular object having a particular object color that is associated with a brightness that exceeds a threshold brightness value.

9. The computer-implemented method of claim 1, wherein identifying the video feature comprises:

performing a salience analysis of the received video, the salience analysis comprising identifying a plurality of objects appearing in the received video, and for each of the identified objects, evaluating a frame occupancy proportion; and

making a determination that the identified video feature includes a particular object having a particular frame occupancy proportion that exceeds a threshold frame occupancy proportion.

10. A computer program product for quantifying memorability of video content, the computer program product comprising a non-transitory computer readable medium containing computer program code that, when executed by one or more processors, performs a video content memorability quantification process that comprises:

receiving a video comprising video frames, wherein the video includes a video feature that is associated with a text feature, wherein the video feature and the text feature are displayed during a particular time period in the video, and wherein the text feature comprises textural content that is visually shown in at least one of the video frames;

identifying the video feature and the associated text feature in the video;

determining a video feature score corresponding to the video feature, the video feature score indicating memorability of the video feature;

determining a text feature score corresponding to the text feature, the text feature score indicating memorability of the text feature;

determining a content memorability score that is based on the video feature score and the text feature score; and

causing display of a memorability map that indicates the particular time period and includes a visual indicator corresponding to the content memorability score.

11. The computer program product of claim 10, wherein the content memorability score is determined by multiplying the video and text feature scores.

12. The computer program product of claim 10, wherein:

determining the video feature score comprises determining a first semantic meaning associated with the video feature and determining the video feature score based on the first semantic meaning, wherein the video feature score is determined using a trained deep neural network learning algorithm; and

determining the text feature score comprises determining a second semantic meaning associated with the text feature, determining a degree of similarity between the first and second semantic meanings, and determining the text feature score based on the degree of similarity.

13. The computer program product of claim 10, wherein the video content memorability quantification process further comprises:

modifying the video feature; and

in response to modifying the video feature, determining a modified video feature score and a modified content memorability score.

14. The computer program product of claim 10, wherein the video content memorability quantification process further comprises:

modifying the video feature;

in response to modifying the video feature, determining a modified video feature score and a modified content memorability score; and

updating the memorability map to reflect the modified content memorability score.

15. A system for quantifying memorability of video content, the system comprising:

a server configured to receive a video that comprises a plurality of video frames, wherein the video includes a video feature that is associated with a text feature, and wherein the text feature comprises textural content that is visually shown in at least one of the video frames; and

a non-transitory computer-readable medium to perform functions of a scoring module, the scoring module configured to;

determine a first semantic meaning associated with the video feature, determine a second semantic meaning associated with the text feature, determine a degree of similarity between the first and second semantic meanings, and determine a text feature score based on the degree of similarity,

determine a video feature score based on the first semantic meaning, wherein the video feature score is determined using a trained deep neural network learning algorithm, and

determine a content memorability score based on the text feature score and the video feature score.

16. The system of claim 15, further comprising a user interface module configured to cause display of a memorability map that:

indicates a particular time period associated with a subset of the video frames that include the video feature and the text feature; and

includes a visual indicator corresponding to the content memorability score, wherein the visual indicator links the content memorability score with the particular time period.

17. The system of claim 15, wherein the content memorability score is determined by multiplying the video and text feature scores.

18. The system of claim 15, further comprising a content feature identifier configured to identify the video feature in the video by:

performing a spatiotemporal analysis of the video, the spatiotemporal analysis comprising evaluating relative movement between two of the video frames; and

making a determination that the identified video feature has relative movement between two of the video frames that exceeds a threshold value.

19. The system of claim 15, further comprising a content feature identifier configured to identify the video feature in the video by:

performing a salience analysis of the video, the salience analysis comprising identifying a plurality of objects appearing in the video, and for each of the identified objects, evaluating a contrast between an object color and a background color;

making a determination that the identified video feature includes a particular object having a particular contrast between the object color and the background color that exceeds a threshold contrast value.

20. The system of claim 15, further comprising a content feature identifier configured to identify the video feature in the video by:

performing a salience analysis of the video, the salience analysis comprising identifying a plurality of objects appearing in the video, and for each of the identified objects, evaluating a frame position; and

making a determination that the identified video feature includes a particular object having a particular frame position that is within a threshold distance from a center of a frame containing the particular object.