Accurate video concept recognition via classifier combination

• US 9,087,297 B1
• Filed: 12/09/2011
• Issued: 07/21/2015
• Est. Priority Date: 12/17/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Filippova, Katja, Hall, Keith B.
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Brown, Jr., Nathan
• Application Number:
  US13/316,337
• Time in Patent Office: 1,320 Days
• Field of Search: 706/12, 706/16
• US Class Current: 1/1
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06V 10/87 using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

A classifier learning module trains video classifiers associated with a stored set of concepts derived from textual metadata of a plurality of videos. Specifically, a first type of classifier (e.g., a content-based classifier) and a second type of classifier (e.g., a text-based classifier) are trained, the classifiers when applied to a video indicating a likelihood that the video represents one or more concepts corresponding to the classifier. The first type of classifier can be used to determine the training set for the second type of classifier. The learning process does not require any concepts to be known a priori, nor does it require a training set of videos having training labels manually applied by human experts. Scores from the first type of classifier are combined with scores from the second type of classifier to obtain video classification of enhanced accuracy.

Claims

1. A computer-implemented method comprising:

accessing a set of concepts derived from textual metadata of videos;

accessing, for each of a plurality of the concepts, a corresponding content-based classifier that inputs content features of a video and outputs a score indicating a likelihood that the video represents the corresponding concept;

obtaining a set of concept scores by applying, to content features of each video of a plurality of the videos, the content-based classifiers;

identifying, for each concept of a plurality of the concepts, a training set comprising ones of the videos for which the obtained concept scores corresponding to the concept indicate at least a threshold level of likelihood that the videos represent the concept;

deriving, from textual metadata of the videos of the identified training sets, textual features;

training text-based classifiers for the plurality of the concepts, the text-based classifiers taking as input the textual features and outputting concept scores indicating a likelihood that the videos represent the corresponding concepts;

after training the text-based classifiers, receiving a new video to which the text-based classifiers have not previously been applied;

obtaining a first score for a first one of the concepts by applying the content-based classifier to the new video;

obtaining a second score for the first concept by applying the text-based classifier to the new video;

forming a combined score, the forming comprising forming a product of the first score and the second score; and

determining whether the new video represents the first concept based on the combined score.

2. The computer-implemented method of claim 1, further comprising selecting, as the textual metadata for a video from which textual features are derived, a given maximum number of a set of most recent user comments on the video.

3. The computer-implemented method of claim 1, wherein deriving the textual features for a video comprises:

identifying, for each word of some set of words, occurrences of the word in the video; and

forming a vector representing the occurrences, each element of the vector corresponding to one of the words and representing the occurrences of the word.

4. A computer-readable storage medium storing executable computer program instructions, comprising:

instructions for obtaining a set of concept scores by applying, to each video of a set of videos, a first classifier for a concept, the first classifier taking as input a first type of feature of the video and outputting a concept score indicating a likelihood that the video represents the concept;

instructions for identifying ones of the videos for which the corresponding obtained concept score has at least a threshold value;

instructions for deriving, from the identified ones of the videos, features of a second type different from the first type of features; and

instructions for training a second classifier for the concept based on the second type of features, the second classifier taking as input the second type of features of a video and outputting a concept score indicating a likelihood that the video represents the corresponding concept,

wherein the first classifier is one of a content-based classifier and a text-based classifier, and wherein the second classifier is one of a content-based classifier and a text-based classifier, the first classifier and the second classifier taking as input different types of features.

5. The computer-readable storage medium of claim 4, further comprising instructions for:

after training the second classifier, receiving a new video to which the second classifier has not previously been applied;

obtaining a first score for the concept by applying the first classifier to the new video;

obtaining a second score for the concept by applying the second classifier to the new video;

forming a combined score as a function of the first score and the second score; and

determining whether the new video represents the first concept based on the combined score.

6. The computer-readable storage medium of claim 5, wherein forming the combined score comprises forming the product of the first score and the second score.

7. The computer-readable storage medium of claim 6, further comprising instructions for, prior to forming the product:

reducing the first score by some value representing a score of minimum reliability; and

responsive to the reduced first score being less than 0.5, converting the reduced first score to 0.0.

8. The computer-readable storage medium of claim 4, further comprising instructions for selecting, as textual metadata for a video from which textual features are derived, a given maximum number of a set of most recent user comments on the video.

9. The computer-implemented method of claim 8, further comprising, prior to forming the product:

reducing the first score by some value representing a score of minimum reliability; and

responsive to the reduced first score being less than 0.5, converting the reduced first score to 0.0.

10. The computer-readable storage medium of claim 4, further comprising instructions for deriving textual features for a video, comprising:

identifying, for each word of some set of words, occurrences of the word in the video; and

forming a vector representing the occurrences, each element of the vector corresponding to one of the words and representing the occurrences of the word.

11. A computer system comprising:

a computer processor; and

a computer-readable storage medium having executable computer program instructions embodied therein that when executed by the computer processor perform actions comprising;

accessing a set of concepts derived from textual metadata of videos;

accessing, for each of a plurality of the concepts, a corresponding content-based classifier that inputs content features of a video and outputs a score indicating a likelihood that the video represents the corresponding concept;

obtaining a set of concept scores by applying, to content features of each video of a plurality of the videos, the content-based classifiers;

identifying, for each concept of a plurality of the concepts, a training set comprising ones of the videos for which the obtained concept scores corresponding to the concept indicate at least a threshold level of likelihood that the videos represent the concept;

deriving, from textual metadata of the videos of the identified training sets, textual features;

training text-based classifiers for the plurality of the concepts, the text-based classifiers taking as input the textual features and outputting concept scores indicating a likelihood that the videos represent the corresponding concepts,

after training the text-based classifiers, receiving a new video to which the text-based classifiers have not previously been applied;

obtaining a first score for a first one of the concepts by applying the content-based classifier to the new video;

obtaining a second score for the first concept by applying the text-based classifier to the new video;

forming a combined score, the forming comprising forming a product of the first score and the second score; and

determining whether the new video represents the first concept based on the combined score.

12. The computer system of claim 11, the actions further comprising selecting, as the textual metadata for a video from which textual features are derived, a given maximum number of a set of most recent user comments on the video.

17. The computer system of claim 11, wherein deriving the textual features for a video comprises:

identifying, for each word of some set of words, occurrences of the word in the video; and

forming a vector representing the occurrences, each element of the vector corresponding to one of the words and representing the occurrences of the word.

13. A computer-implemented method comprising:

accessing a set of concepts derived from textual metadata of videos;

identifying, for each of a plurality of the concepts, a first training set of videos based on a presence of the concept within textual metadata of the videos;

training one or more content-based classifiers for the concepts based on content of the videos in the training sets that correspond to the concepts;

obtaining a set of scores for each video of a plurality of the videos by applying the trained one or more content-based classifiers to content of the video, each score of the set of scores representing a likelihood that the video represents a corresponding one of the concepts;

identifying, for each of a plurality of the concepts, a second training set of videos based on ones of the scores that correspond to the concept;

selecting, as textual metadata for the videos in the training sets that correspond to the concepts, a given maximum number of a set of most recent user comments on the videos in the training sets that correspond to the concepts; and

training one or more text-based classifiers for the concepts based on the selected textual metadata.

14. The computer-implemented method of claim 13, further comprising:

after training the classifiers, receiving a new video to which the classifiers have not previously been applied;

obtaining a first score for a first one of the concepts by applying the one or more content-based classifiers to the new video;

obtaining a second score for the first concept by applying the one or more text-based classifiers to the new video;

forming a combined score as a function of the first score and the second score; and

determining whether the new video represents the first concept based on the combined score.

15. The computer-implemented method of claim 14, wherein forming the combined score comprises forming the product of the first score and the second score.

16. The computer-implemented method of claim 15, further comprising, prior to forming the product:

reducing the first score by some value representing a score of minimum reliability; and

responsive to the reduced first score being less than 0.5, converting the reduced first score to 0.0.