Video Highlight Detection with Pairwise Deep Ranking

• US 20170109584A1
• Filed: 10/20/2015
• Published: 04/20/2017
• Est. Priority Date: 10/20/2015
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Yao, Ting, Mei, Tao, Rui, Yong
• Application Number:
  US14/887,629
• Publication Number:
  US 20170109584A1
• Time in Patent Office: Days
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 21/45457 applied to a time segment
• CPC Class Codes: H04N 21/4666 using neural networks, e.g. processing the feedback provided by the user
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Video highlight detection using pairwise deep ranking neural network training is described. In some examples, highlights in a video are discovered, then used for generating summarization of videos, such as first-person videos. A pairwise deep ranking model is employed to learn the relationship between previously identified highlight and non-highlight video segments. This relationship is encapsulated in a neural network. An example two stream process generates highlight scores for each segment of a user'"'"'s video. The obtained highlight scores are used to summarize highlights of the user'"'"'s video.

Claims

1. An apparatus comprising:

a processor; and

a computer-readable medium storing modules of instructions that, when executed by the processor, configure the apparatus to perform video highlight detection, the modules comprising;

a training module to configure the processor to train a neural network based at least on a previously identified highlight segment and a previously identified non-highlight segment, wherein the highlight and non-highlight segments are from a same video;

a highlight detection module to configure the processor to generate a highlight score for a video segment of a plurality of video segments from an input video based at least in part on a set of information associated with the video segment and the neural network; and

an output module to configure the processor to generate an output based at least in part on the highlight scores for the plurality of video segments.

2. The apparatus of claim 1, wherein the training module is further to configure the processor to:

generate a highlight segment score by inserting first information associated with the previously identified highlight video segment into a first neural network, the inserted first information having a format similar to the set of information associated with the video segment;

generate a non-highlight segment score by inserting second information associated with the previously identified non-highlight video segment into a second neural network, the inserted second information having a format similar to the set of information associated with the video segment,

compare the highlight segment score to the non-highlight segment score; and

adjust one or more parameters for at least one of the neural networks based at least in part on the comparing.

3. The apparatus of claim 1, wherein the highlight detection module is further to configure the processor to:

identify the set of information by selecting spatial information samples of the video segment;

determine a plurality of classification values for the spatial information samples;

determine an average of the plurality of classification values; and

insert the average of the plurality of classification values into the neural network.

4. The apparatus of claim 1, wherein the highlight detection module is further to configure the processor to:

identify the set of information by selecting temporal information samples of the video segment;

determine a plurality of classification values for the temporal information samples;

determine an average of the plurality of classification values; and

insert the average of the plurality of classification values into the neural network.

5. The apparatus of claim 1, wherein the output module is further to configure the processor to:

determine a first playback speed for frames of one of the video segments in response to the highlight score of the one of the video segments being greater than a threshold value; and

determine a second playback speed for frames of the one of the video segments in response to the highlight score of the one of the video segments being less than the threshold value.

6. The apparatus of claim 1, wherein the output module is further to configure the processor to:

identify video segments having a highlight score greater than a threshold; and

combine at least a portion of the frames of the video segments identified as having the highlight score greater than a threshold value.

7. A system comprising:

a processor; and

a computer-readable media including instructions that, when executed by the processor, configure the processor to;

generate a first highlight score for a video segment of a plurality of video segments of an input video based at least in part on a first set of information associated with the video segment and a first neural network;

generate a second highlight score for the video segment based at least in part on a second set of information associated with the video segment and a second neural network;

generate a third highlight score for the video segment by merging the first highlight score and the second highlight score for the video segment; and

generate an output based at least on the third highlight scores for the plurality of video segments.

8. The system of claim 7, wherein the computer-readable media includes further instructions that, when executed by the processor, further configure the processor to:

generate a first highlight segment score by inserting first information associated with a previously identified highlight video segment from another video into the first neural network;

generate a first non-highlight segment score by inserting second information associated with a previously identified non-highlight video segment from the other video into the first neural network, wherein the first and second information have a format similar to the first set of information;

compare the first highlight segment score to the first non-highlight segment score;

adjust one or more parameters for the first neural network based at least in part on the comparing;

generate a second highlight segment score by inserting third information associated with a previously identified highlight video segment from the other video into the second neural network;

generate a second non-highlight segment score by inserting fourth information associated with a previously identified non-highlight video segment from the other video into the second neural network, wherein the third and fourth information have a format similar to the second set of information;

compare the second highlight segment score to the second non-highlight segment score; and

adjust one or more parameters for the second neural network based at least in part on the comparing.

9. The system of claim 7, wherein the computer-readable media includes further instructions that, when executed by the processor, further configure the processor to:

identify the first set of information by selecting spatial information samples of the video segment;

determine a plurality of classification values for the spatial information samples;

determine an average of the plurality of classification values;

insert the average of the plurality of classification values into the first neural network;

identify the second set of information by selecting temporal information samples of the video segment;

determine a plurality of classification values for the temporal information samples;

determine an average of the plurality of classification values for the temporal information samples; and

insert the average of the plurality of classification values for the temporal information samples into the second neural network.

10. The system of claim 7, wherein the computer-readable media includes further instructions that, when executed by the processor, further configure the processor to

determine a first playback speed for frames of one of the video segments in response to the third highlight score of the one of the video segments being greater than a first threshold value;

and

determine a second playback speed for frames of the one of the video segments in response to the third highlight score of the one of the video segments being less than the first threshold value;

or

identify video segments having a third highlight score greater than a second threshold value; and

combine at least a portion of the frames of the video segments identified as having the third highlight score greater than the second threshold value.

11. A method comprising:

generating, at a computing device, a first highlight score for a video segment of a plurality of video segments of an input video based at least in part on a first set of information associated with the video segment and a first neural network;

generating a second highlight score for the video segment based at least in part on a second set of information associated with the video segment and a second neural network;

generating a third highlight score for the video segment by merging the first highlight score and the second highlight score for the video segment; and

generating an output based at least on the third highlight scores for the plurality of video segments.

12. The method of claim 11, further comprising:

training the first neural network comprising;

generating a highlight segment score by inserting first information associated with a previously identified highlight video segment from another video into a first version of the first neural network;

generating a non-highlight segment score by inserting second information associated with a previously identified non-highlight video segment from the other video into a second version of the first neural network, wherein the first and second information have a format similar to the first set of information;

comparing the highlight segment score to the non-highlight segment score; and

adjusting one or more parameters for the first neural network based at least in part on the comparing.

13. The method of claim 11, further comprising:

training the second neural network comprising;

generating a highlight segment score by inserting first information associated with a previously identified highlight video segment from another video into a first version of the second neural network to generate a highlight segment score;

generating a non-highlight segment score by inserting second information associated with a previously identified non-highlight video segment from the other video into a second version of the second neural network, wherein the first and second information have a format similar to the second set of information;

comparing the highlight segment score to the non-highlight segment score; and

adjusting one or more parameters for the second neural network based at least in part on the comparing.

14. The method of claim 11, further comprising:

identifying the first set of information by selecting spatial information samples of the video segment;

determining a plurality of classification values for the spatial information samples;

determining an average of the plurality of classification values; and

inserting the average of the plurality of classification values into the first neural network.

15. The method of claim 11, further comprising:

identifying the second set of information by selecting temporal information samples of the video segment;

determining a plurality of classification values for the temporal information samples;

determining an average of the plurality of classification values; and

inserting the average of the plurality of classification values into the second neural network.

16. The method of claim 11, further comprising:

identifying the first set of information by selecting spatial information samples of the video segment;

determining a plurality of classification values for the spatial information samples;

determining an average of the plurality of classification values;

inserting the average of the plurality of classification values into the first neural network;

identifying the second set of information by selecting temporal information samples of the video segment;

determining a plurality of classification values for the temporal information samples;

determining an average of the plurality of classification values for the temporal information samples; and

inserting the average of the plurality of classification values for the temporal information samples into the second neural network.

17. The method of claim 11, further comprising:

determining a first playback speed for frames of one of the video segments in response to the third highlight score of the one of the video segments being greater than a threshold value; and

determining a second playback speed for frames of the one of the video segments in response to the third highlight score of the one of the video segments being less than the threshold value.

18. The method of claim 11, further comprising:

determining a playback speed for frames of one of the video segments based at least on the third highlight score of one of the video segments.

19. The method of claim 11, further comprising:

identifying video segments having a third highlight score greater than a threshold value; and

combining at least a portion of the frames of the video segments identified as having the third highlight score greater than the threshold value.

20. The method of claim 11, further comprising:

ordering at least a portion of the frames of at least a portion of the video segments based at least on the third highlight scores of the portion of the video segments.