Learning highlights using event detection

• US 9,715,641 B1
• Filed: 12/29/2014
• Issued: 07/25/2017
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Tang, Henry Hao, Kwatra, Vivek, Gargi, Ullas, Sargin, Mehmet Emre
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/585,075
• Time in Patent Office: 939 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/44 Event detection

Abstract

A highlight learning technique is provided to detect and identify highlights in sports videos. A set of event models are calculated from low-level frame information of the sports videos to identify recurring events within the videos. The event models are used to characterize videos by detecting events within the videos and using the detected events to generate an event vector. The event vector is used to train a classifier to identify the videos as highlight or non-highlight.

Claims

1. A computer-implemented method for identifying a portion of a video as a highlight, comprising:

accessing a video;

extracting, from the video, a feature set associated with frames of the video;

identifying a plurality of video portions of the video, each video portion associated with a subset of frames of the video, and each video portion also associated with a subset of the feature set, wherein the frames of at least one of the video portions overlap with the frames of another one of the video portions;

classifying each video portion using a plurality of highlight classifiers and based on the subset of the feature set associated with that video portion to generate a classification result; and

identifying one or more video portions of the plurality of video portions as highlights according to the classification results generated by the highlight classifiers.

2. The computer-implemented method of claim 1, further comprising:

detecting a set of events for each subset of the feature set using a set of event models;

calculating an event vector for each of the set of events detected for each video portion; and

wherein classifying each video portion using the plurality of highlight classifiers comprises;

classifying the set of events detected for the video portion using the highlight classifiers and the calculated event vector for the video portion; and

wherein identifying one or more video portions of the plurality of video portions as highlights according to the classification result of the highlight classifiers comprises;

identifying one or more sets of events detected for one or more video portions as highlights according to the classification result, the one or more video portions identified as highlights.

3. The computer-implemented method of claim 2, wherein the event models include at least one from a group consisting of hidden Markov models and Gaussian mixture models.

4. The computer-implemented method of claim 2, wherein each event vector is calculated based at least in part on histograms generated for the corresponding detected set of events.

5. The computer-implemented method of claim 4, wherein the histograms used for each event vector include a unigram of event frequency and a bigram of event transitions.

6. The computer-implemented method of claim 1, wherein the feature set extracted from the video includes at least one of:

a color histogram, a histogram of oriented gradients, and a histogram of oriented optical flow.

18. The method of claim 1, wherein a highlight classifier uses one of a linear support vector machine, logistic regression, or boosting algorithm, and wherein the highlight classifier outputs a score representing a degree to which a video portion indicates a particular highlight.

19. The method of claim 2, wherein events represent coherent patterns of feature data in the video portions.

7. A non-transitory computer-readable storage medium having executable program instructions embodied therein, the computer program instructions controlling a computer system to perform a method comprising:

accessing a video;

extracting, from the video, a feature set associated with frames of the video;

identifying a plurality of video portions of the video, each video portion associated with a subset of frames of the video, and each video portion also associated with a subset of the feature set, wherein frames of at least one video portion overlap with frames of another video portion;

classifying each video portion using a plurality of highlight classifiers based on the subset of the feature set identified for each video portion to generate a classification result; and

identifying one or more video portions of the plurality of video portions as highlights according to the classification result of the highlight classifiers.

8. The non-transitory computer-readable storage medium of claim 7, the method further comprising:

detecting a set of events for each subset of the feature set using a set of event models;

calculating an event vector for each of the set of events detected for each video portion; and

wherein classifying each video portion using the plurality of highlight classifiers comprises;

classifying the set of events detected for the video portion using the highlight classifiers and the calculated event vector for the video portion; and

wherein identifying one or more video portions of the plurality of video portions as highlights according to the classification result of the highlight classifiers comprises;

identifying one or more sets of events detected for one or more video portions as highlights according to the classification result, the one or more video portions identified as highlights.

9. The non-transitory computer-readable storage medium of claim 8, wherein the event models include at least one from a group consisting of hidden Markov models and Gaussian mixture models.

10. The non-transitory computer-readable storage medium of claim 8, wherein each event vector is calculated based at least in part on histograms generated for the corresponding detected set of events.

11. The non-transitory computer-readable storage medium of claim 10, wherein the histograms used for each event vector include a unigram of event frequency and a bigram of event transitions.

12. The non-transitory computer-readable storage medium of claim 7, wherein the feature set extracted from the video includes at least one of:

a color histogram, a histogram of oriented gradients, and a histogram of oriented optical flow.

13. A computer system comprising:

a computer processor;

a computer-readable storage medium storing data including a computer program which when executed by the computer processor performs the operations of;

accessing a video;

extracting, from the video, a feature set associated with frames of the video;

identifying a plurality of video portions of the video, each video portion associated with a subset of frames of the video, and each video portion also associated with a subset of the feature set, wherein frames of at least one video portion overlap with frames of another video portion;

classifying each video portion using a plurality of highlight classifiers based on the subset of the feature set identified for each video portion to generate a classification result; and

identifying one or more video portions of the plurality of video portions as highlights according to the classification result of the highlight classifiers.

14. The system of claim 13, wherein the computer program which when executed by the computer processor further performs the steps of:

detecting a set of events for each subset of the feature set using a set of event models;

calculating an event vector for each of the set of events detected for each video portion; and

wherein classifying each video portion using the plurality of highlight classifiers comprises;

classifying the set of events detected for the video portion using the highlight classifiers and the calculated event vector for the video portion; and

wherein identifying one or more video portions of the plurality of video portions as highlights according to the classification result of the highlight classifiers comprises;

identifying one or more sets of events detected for one or more video portions as highlights according to the classification result, the one or more video portions identified as highlights.

15. The system of claim 14, wherein the event models include at least one from a group consisting of hidden Markov models and Gaussian mixture models.

16. The system of claim 14, wherein each event vector is calculated based at least in part on histograms generated for the corresponding detected set of events.

17. The system of claim 16, wherein the histograms used for each event vector include a unigram of event frequency and a bigram of event transitions.