Machine learned classifiers for rating the content quality in videos using panels of human viewers

• US 8,706,655 B1
• Filed: 06/03/2011
• Issued: 04/22/2014
• Est. Priority Date: 06/03/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Rangarajan, Anand, DuHadway, Charles, Aradhye, Hrishikesh Balkrishna
• Primary Examiner(s): VINCENT, DAVID ROBERT
• Application Number:
  US13/152,978
• Time in Patent Office: 1,054 Days
• Field of Search: 706/12, 706/45
• US Class Current: 706/12
• CPC Class Codes: G06Q 30/0282 Rating or review of business operators or products
• CPC Class Codes: H04N 21/4756 for rating content, e.g. scoring a recommended movie

Abstract

A multi-phase process first trains a machine learned rating classifier, and then uses the rating classifier to automatically rate videos in a selected category in a way which mimics human rating. Panels of human viewers rate videos in tuples, and these tuples along with human preference data distilled from the ratings are used to create a training set that is used to train the machine learned rating classifier. The rating classifier becomes capable of predicting human preferences with regards to videos in the selected category. Optionally a second machine learned classifier can be trained to automatically identify videos in the selected category for the panels of human viewers to rate. The output of the multi-phase process can be used to highlight content that is predicted to be higher quality.

Claims

1. A computer-implemented method for training a machine learned rating classifier for rating the content quality of videos, the method comprising:

receiving from a plurality of panels of human viewers, ratings associated with a plurality of tuples of videos of a selected category, the ratings indicating preferences of the human viewers for videos in the tuples;

determining from the ratings, preferred videos in the tuples of videos, wherein each tuple has at least one preferred video selected responsive to the indicated preferences of the human viewers;

creating a training set of training videos from the preferred videos in the tuples of videos; and

training the machine learned rating classifier using the training set, the rating classifier configured to determine a rating of content quality of an unrated video of the selected category based on a correlation of the unrated video with the ratings associated with the plurality of tuples of videos of the selected category.

2. The method of claim 1, wherein the videos of the selected category are automatically identified using a machine learned category classifier.

3. The method of claim 2, wherein the category classifier uses an Adaptive Boosting algorithm.

4. The method of claim 2, wherein the category classifier is trained to identify videos of the selected category by using a training set of videos from user playlists that are identified as being associated with the selected category.

5. The method of claim 1, wherein the videos of the selected category are identified by providing a search query, comprising the selected category, to a search engine, and including videos in search results responsive to the query in the selected category.

6. The method of claim 1, wherein the rating classifier comprises a Kernel PAMIR machine learned ranker.

7. The method of claim 1, wherein the ratings associated with the plurality of tuples of videos are produced by providing the same tuple to no more than one panel.

8. The method of claim 1, wherein the ratings associated with the plurality of tuples of videos are produced by providing at least one tuple of videos to more than one panel.

9. The method of claim 1, wherein the ratings associated with the plurality of tuples of videos further comprise a ranking of the videos in each tuple.

10. The method of claim 1, wherein the ratings associated with the plurality of tuples of videos further comprise one or more votes for each human viewer on a panel of the plurality of panels of human viewers.

11. The method of claim 1, wherein the ratings associated with the plurality of tuples of videos further comprise a numerical score for one or more videos in each tuple.

12. The method of claim 1, wherein the ratings associated with the plurality of tuples of videos further comprise a categorical value for one or more videos in each tuple.

35. The method of claim 1, wherein the rating classifier is further configured to learn content features from each tuple of videos that are correlated positively or negatively with the preferences of the human viewers.

36. The method of claim 35, wherein the rating classifier determines the rating of content quality of the unrated video based on a correlation of content features of the unrated video with the learned content features that are correlated positively or negatively with the preferences of the human viewers.

13. A computer implemented method of rating the content quality of content in videos comprising:

submitting a plurality of unrated videos of a selected category to a machine learned rating classifier, the machine learned rating classifier trained using a training set of videos in the selected category, the training set generated from a plurality of tuples of videos of the selected category and ratings associated with the plurality of tuples of videos of the selected category provided by a plurality of panels of human viewers, the ratings indicating preferences of the human viewers for videos in the tuples;

using the rating classifier to automatically generate a rating of content quality for each of the unrated videos of the selected category, the rating of content quality of an unrated video based on a correlation of the unrated video with the ratings associated with the plurality of tuples of videos in the selected category;

making a list of the plurality of videos of the selected category that were rated by the machine learned rating classifier;

using the generated ratings of the videos in the list to order the list; and

using the ordered list to provide users of a video hosting service with a selection of videos to view.

14. A computer system for automatically rating videos of a selected category, the system comprising:

one or more computer processors;

rating software instructions executing on the one or more processors, for receiving ratings associated with a plurality of tuples of videos of the selected category from a plurality of panels of human viewers, wherein the ratings indicate preferences of the human viewers for videos in the tuples;

scoring software instructions executing on the one or more processors, for (1) training a machine learned rating classifier using a training set of training videos of the selected category, the training set created from preferred videos in the tuples of videos selected responsive to the indicated preferences of the human viewers, (2) submitting a set of one or more unrated videos of the selected category to the machine learned rating classifier, (3) automatically generating a rating for each of the one or more unrated videos in the set to produce a set of rated videos, the rating of a rated video based on a correlation of the rated video with the ratings associated with the plurality of tuples of videos of the selected category, (4) ordering the videos in the set of rated videos using the ratings, (5) providing users of a video hosting service with a selection of videos to view based on the ordering of the videos in the set of rated videos.

15. The system of claim 14, wherein the videos of the selected category are automatically identified using a machine learned category classifier.

16. The system of claim 15, wherein the category classifier uses an Adaptive Boosting algorithm.

17. The system of claim 15, wherein the category classifier is trained to identify videos of the selected category by using a training set of videos from user playlists that are identified as being associated with the selected category.

18. The system of claim 14, wherein the videos of the selected category are identified by providing a search query, comprising the selected category, to a search engine, and including videos in search results responsive to the query in the selected category.

19. The system of claim 14, wherein the rating classifier comprises a Kernel PAMIR machine learned ranker.

20. The system of claim 14, wherein the ratings of the tuples of videos are produced by providing the same tuple of videos to no more than one panel.

21. The system of claim 14, wherein the ratings associated with the plurality of tuples of videos are produced by providing at least one tuple videos to more than one panel.

22. The system of claim 14, wherein the ratings associated with the plurality of tuples of videos further comprise a ranking of the videos in each tuple.

23. The system of claim 14, wherein the ratings associated with the plurality of tuples of videos further comprise one or more votes for each human viewer on a panel of the plurality of panels of human viewers.

24. The system of claim 14, wherein the ratings associated with the plurality of tuples of videos further comprise a numerical score for one or more videos in each tuple.

25. The system of claim 14, wherein the ratings associated with the plurality of tuples of videos further comprise a categorical value for one or more videos in each tuple.

26. An apparatus comprising a non-transitory computer-readable medium having executable instructions encoded thereon that, in response to execution by a computing device, cause the computing device to perform operations comprising:

receiving from a plurality of panels of human viewers, ratings associated with a plurality of tuples of videos of a selected category, the ratings indicating preferences of the human viewers for videos in the tuples;

determining from the ratings, preferred videos in the tuples of videos, wherein each tuple has at least one preferred video selected responsive to the indicated preferences of the human viewers;

creating a training set of training videos from the preferred videos in the tuples of videos; and

training the machine learned rating classifier using the training set, the rating classifier configured to determine a rating of content quality of an unrated video of the selected category based on a correlation of the unrated video with the ratings associated with the plurality of tuples of videos of the selected category.

27. The apparatus of claim 26, wherein the videos of the selected category are automatically identified using a machine learned category classifier.

28. The apparatus of claim 27, wherein the category classifier uses an Adaptive Boosting algorithm.

29. The apparatus of claim 27, wherein the category classifier is trained to identify videos of the selected category by using a training set of videos from user playlists that are identified as being associated with the selected category.

30. The apparatus of claim 26, wherein the videos of the selected category are identified by providing a search query, comprising the selected category, to a search engine, and including videos in search results responsive to the query in the selected category.

31. The apparatus of claim 26, wherein the rating classifier comprises a Kernel PAMIR machine learned ranker.

32. The apparatus of claim 26, wherein the ratings associated with the plurality of tuples of videos are produced by providing the same tuple to no more than one panel.

33. The apparatus of claim 26, wherein the ratings associated with the plurality of tuples of videos are produced by providing at least one tuple of videos to more than one panel.

34. The apparatus of claim 26, wherein the ratings associated with the plurality of tuples of videos further comprise a ranking of the videos in each tuple.