VIDEO SUMMARIZATION USING SEMANTIC INFORMATION

• US 20170185846A1
• Filed: 12/24/2015
• Published: 06/29/2017
• Est. Priority Date: 12/24/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Hwangbo, Myung, Singh, Krishna Kumar, Lee, Teahyung, Tickoo, Omesh
• Granted Patent: US 10,229,324 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

An apparatus for video summarization using sematic information is described herein. The apparatus includes a controller, a scoring mechanism, and a summarizer. The controller is to segment an incoming video stream into a plurality of activity segments, wherein each frame is associated with an activity. The scoring mechanism is to calculate a score for each frame of each activity, wherein the score is based on a plurality of objects in each frame. The summarizer is to summarize the activity segments based on the score for each frame.

Claims

1. An apparatus, comprising:

a controller to segment an incoming video stream into a plurality of activity segments, wherein each frame is associated with an activity;

a scoring mechanism to calculate a score for each frame of each activity, wherein the score is based, at least partially, on a classification probability of each frame; and

a summarizer to summarize the activity segments based on the score for each frame.

2. The apparatus of claim 1, wherein the score is based on, at least partially, an activity-to-object co-occurrence.

3. The apparatus of claim 1, wherein the activities are segmented by one or more shot boundaries and each frame is labeled according to an activity class.

4. The apparatus of claim 1, wherein a convolutional neural network is used to classify each segment into an activity.

5. The apparatus of claim 4, wherein frames with mislabeled activities are relabeled according to the activities of surrounding frames.

6. The apparatus of claim 1, wherein a fast, regional convolutional neural network is used to classify a plurality of objects of each frame.

7. The apparatus of claim 1, wherein segments lower than a predefined threshold in length are discarded.

8. The apparatus of claim 1, wherein the scoring mechanism determines a score that is the probability that a frame belongs to an activity based on objects in the frame.

9. The apparatus of claim 1, wherein the scoring mechanism determines a score that is the probability that an object of a frame belongs to a class of objects combined with the importance of the object for the activity assigned to the frame.

10. The apparatus of claim 1, wherein the summarizer is to create a summary by adding frames to the summary with scores above a predefined threshold.

11. The apparatus of claim 1, wherein the summary is generated by selecting key image frames that correspond to the highest score for each segment, or key clips of N-seconds are selected for each segment.

13. The method of claim 11, wherein object to activity correlation is obtained by executing both an activity classifier and object detector on a set of training data, by human annotation, or by any combination thereof.

14. The method of claim 11, wherein the importance of an object for an activity is directly proportional to the value of the number of frames in activity that contains an object divided by the total number of frames for that activity.

15. The method of claim 11, wherein a convolutional neural network is used to label each frame according to an activity class.

16. The method of claim 11, wherein a fast, regional convolutional neural network is used to classify a plurality of objects of each frame.

17. The method of claim 11, wherein a probability than the object belongs to a particular activity is used to determine, at least partially, the object to activity correlation.

12. A method for video summarization, comprising:

labeling each frame of a plurality of frames according to an activity class;

determining an object-to-activity correlation for an object within each frame; and

rendering a video summary that comprises the frames with object-to-activity correlations above a predetermined threshold for each frame in a shot boundary.

18. A system, comprising:

a display;

an image capture mechanism;

a memory that is to store instructions and that is communicatively coupled to the image capture mechanism and the display; and

a processor communicatively coupled to the image capture mechanism, the display, and the memory, wherein when the processor is to execute the instructions, the processor is to;

label each frame of a plurality of frames according to an activity class;

determine a score corresponding to each frame; and

render a video summary that comprises the frames with scores above a predetermined threshold for each frame in a shot boundary.

19. The system of claim 18, wherein the score is based on, at least partially, an activity-to-object co-occurrence of an object within each frame.

20. The system of claim 18, wherein the score is based on, at least partially, a classification probability of each frame.

21. The system of claim 18, wherein a convolutional neural network is used to label each frame of the plurality of frames according to an activity class.

22. The system of claim 18, wherein frames with mislabeled activity classes are relabeled according to the activities of surrounding frames.

23. A tangible, non-transitory, computer-readable medium comprising instructions that, when executed by a processor, direct the processor to:

label each frame of a plurality of frames according to an activity class;

determine an object-to-activity correlation for an object within each frame; and

render a video summary that comprises the frames with object-to-activity correlations above a predetermined threshold for each frame in a shot boundary.

24. The computer readable medium of claim 23, wherein object to activity correlation is obtained by executing both an activity classifier and object detector on a set of training data, by human annotation, or by any combination thereof.

25. The computer readable medium of claim 23, wherein the importance of an object for an activity is directly proportional to the value of the number of frames in activity that contains an object divided by the total number of frames for that activity.