AUTOMATIC VIDEO SUMMARIZATION

• US 20170124400A1
• Filed: 10/28/2015
• Published: 05/04/2017
• Est. Priority Date: 10/28/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Yehezkel Rohekar, Raanan Y., Koren, Guy
• Granted Patent: US 9,818,032 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

System and techniques for automatic video summarization are described herein. A video may be obtained and a semantic model of the video may be generated from frames of the video. Respective relevancy scores may be assigned to the frames. The semantic model may be initialized with the respective relevancy scores. The semantic model may then be iteratively processed to produce sub-scenes of the video, the collection of sub-scenes being the video summarization.

Claims

1. A device for automatic video summarization, the method comprising:

a storage device to hold a video;

a semantic classifier to generate a semantic model of the video from frames of the video;

a relevancy classifier to assign respective relevancy scores to the frames;

a multiplexer to;

initialize the semantic model with the respective relevancy scores; and

process, iteratively, the semantic model to produce a set of sub-scenes, each iteration comprising the multiplexer to;

converge the semantic model following initialization;

select a sequence of frames with a highest relevancy score after converging; and

re-initialize the semantic model by fixing the relevancy scores for the selected sequence of frames.

2. The device of claim 1, wherein generating the semantic model includes extracting features of the frames.

3. The device of claim 2, wherein generating the semantic model includes generating a pseudo-semantic domain from the extracted features, wherein the pseudo-semantic domain is an n-dimensional space derived from the features.

4. The device of claim 3, wherein generating the semantic model includes mapping the frames to the pseudo-semantic domain by, for each frame, deriving a coordinate in the pseudo-semantic domain, each element of the coordinate corresponding to a dimensionality of the pseudo-semantic domain and derived from a presence of a feature of the extracted features that is specific to the frame.

5. The device of claim 4, wherein generating the semantic model includes creating a generative model from the frames mapped to the pseudo-semantic domain.

6. The device of claim 5, wherein creating the generative model includes recursively identifying a set of key frames in the frames by adding a frame with a highest score to the set of frames, the score of a frame being the inverse of the sum of a square norm of the coordinate of the frame multiplied by a constant and divided by the square of the norm of the distance between the frame and a frame in the set of key frames for all members of the set of key frames.

7. The device of claim 1, wherein assigning respective relevancy scores to the frames includes extracting behavioral indicators of a person who took the video from the video.

8. The device of claim 1, wherein initializing the semantic model with the respective relevancy scores includes constructing a graph in which nodes correspond to the frames, nodes values correspond to respective relevancy scores of the frames, and edges are weighted by the inverse distance between frames established in the semantic model.

9. The device of claim 1, comprising controller to:

present a user interface including the sub-scenes in an order in which they were produced;

receive a discard selection from a user for a sub-scene; and

discard the sub-scene from a video summarization.

10. A machine-implemented method for automatic video summarization, the method comprising:

obtaining a video;

generating a semantic model of the video from frames of the video;

assigning respective relevancy scores to the frames;

initializing the semantic model with the respective relevancy scores; and

iteratively processing the semantic model to produce a set of sub-scenes, each iteration comprising;

converging the semantic model following initialization;

selecting a sequence of frames with a highest relevancy score after converging; and

re-initializing the semantic model by fixing the relevancy scores for the selected sequence of frames.

11. The method of claim 10, wherein generating the semantic model includes extracting features of the frames.

12. The method of claim 11, wherein generating the semantic model includes generating a pseudo-semantic domain from the extracted features, wherein the pseudo-semantic domain is an n-dimensional space derived from the features.

13. The method of claim 12, wherein generating the semantic model includes mapping the frames to the pseudo-semantic domain by, for each frame, deriving a coordinate in the pseudo-semantic domain, each element of the coordinate corresponding to a dimensionality of the pseudo-semantic domain and derived from a presence of a feature of the extracted features that is specific to the frame.

14. The method of claim 10, wherein assigning respective relevancy scores to the frames includes extracting behavioral indicators of a person who took the video from the video.

15. The method of claim 10, wherein initializing the semantic model with the respective relevancy scores includes constructing a graph in which nodes correspond to the frames, nodes values correspond to respective relevancy scores of the frames, and edges are weighted by the inverse distance between frames established in the semantic model.

16. The method of claim 10, comprising:

presenting a user interface including the sub-scenes in an order in which they were produced;

receiving a discard selection from a user for a sub-scene; and

discarding the sub-scene from a video summarization.

17. At least one machine readable medium including instructions that, when executed by a machine, cause the machine to perform operations for automatic video summarization, the operations comprising:

obtaining a video;

generating a semantic model of the video from frames of the video;

assigning respective relevancy scores to the frames;

initializing the semantic model with the respective relevancy scores; and

iteratively processing the semantic model to produce a set of sub-scenes, each iteration comprising;

converging the semantic model following initialization;

selecting a sequence of frames with a highest relevancy score after converging; and

re-initializing the semantic model by fixing the relevancy scores for the selected sequence of frames.

18. The at least one machine readable medium of claim 17, wherein generating the semantic model includes extracting features of the frames.

19. The at least one machine readable medium of claim 18, wherein generating the semantic model includes generating a pseudo-semantic domain from the extracted features, wherein the pseudo-semantic domain is an n-dimensional space derived from the features.

20. The at least one machine readable medium of claim 19, wherein generating the semantic model includes mapping the frames to the pseudo-semantic domain by, for each frame, deriving a coordinate in the pseudo-semantic domain, each element of the coordinate corresponding to a dimensionality of the pseudo-semantic domain and derived from a presence of a feature of the extracted features that is specific to the frame.

21. The at least one machine readable medium of claim 20, wherein generating the semantic model includes creating a generative model from the frames mapped to the pseudo-semantic domain.

22. The at least one machine readable medium of claim 21, wherein creating the generative model includes recursively identifying a set of key frames in the frames by adding a frame with a highest score to the set of frames, the score of a frame being the inverse of the sum of a square norm of the coordinate of the frame multiplied by a constant and divided by the square of the norm of the distance between the frame and a frame in the set of key frames for all members of the set of key frames.

23. The at least one machine readable medium of claim 17, wherein assigning respective relevancy scores to the frames includes extracting behavioral indicators of a person who took the video from the video.

24. The at least one machine readable medium of claim 17, wherein initializing the semantic model with the respective relevancy scores includes constructing a graph in which nodes correspond to the frames, nodes values correspond to respective relevancy scores of the frames, and edges are weighted by the inverse distance between frames established in the semantic model.

25. The at least one machine readable medium of claim 17, where the instructions comprise:

presenting a user interface including the sub-scenes in an order in which they were produced;

receiving a discard selection from a user for a sub-scene; and

discarding the sub-scene from a video summarization.