RANKING AND SELECTING REPRESENTATIVE VIDEO IMAGES

• US 20130142418A1
• Filed: 12/06/2011
• Published: 06/06/2013
• Est. Priority Date: 12/06/2011
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: van Zwol, Roelof, Pueyo, Lluis Garcia
• Granted Patent: US 9,020,244 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

Techniques are described herein for selecting representative images for video items using a trained machine learning engine. A training set is fed to a machine learning engine. The training set includes, for each image in the training set, input parameter values and an externally-generated score. Once a machine learning model has been generated based on the training set, input parameters for unscored images are fed to the trained machine learning engine. Based on the machine learning model, the trained machine learning engine generates scores for the images. To select a representative image for a particular video item, candidate images for that particular video item may be ranked based on their scores, and the candidate image with the top score may be selected as the representative image for the video item.

Claims

1. A method comprising:

generating a plurality of model-generated scores;

wherein each model-generated score of the plurality of model-generated scores corresponds to a candidate image from a plurality of candidate images for a particular video item;

wherein generating the plurality of model-generated scores includes, for each candidate image of the plurality of candidate images, feeding a set of input parameter values into a trained machine learning engine to produce the model-generated score that corresponds to the candidate image;

establishing a ranking of the candidate images for the particular video item based, at least in part, on the model-generated scores that correspond to the candidate images;

selecting a candidate image, from the plurality of candidate images, as a representative image for the video item based, at least in part, on the ranking;

wherein the method is performed by one or more computing devices.

2. The method of claim 1 further comprising automatically selecting the plurality of candidate images for which to generate model-generated scores, by performing the steps of:

dividing the particular video item into a plurality of scenes by detecting scene boundaries within the particular video item; and

selecting a particular number of candidate videos from each of the plurality of scenes.

3. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine includes feeding into the trained machine learning engine at least one input parameter for a target-specific feature.

4. The method of claim 3 wherein the target-specific feature is based on behavior information collected about a target user.

5. The method of claim 3 wherein the target-specific feature is based on behavior information collected about users that have been determined to be similar to a target user.

6. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine includes feeding into the trained machine learning engine at least one input parameter for a request-specific feature.

7. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine includes feeding into the trained machine learning engine at least one input parameter for a temporal feature.

8. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine includes feeding into the trained machine learning engine at least one input parameter for a naturalness feature.

9. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine for a particular candidate image includes feeding into the trained machine learning engine at least one input parameter for an activity feature that reflects actions that users have performed, during playback of the particular video, relative to the frame that corresponds to the particular candidate image.

10. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine for a particular candidate image includes feeding into the trained machine learning engine at least one input parameter for an audio feature.

11. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine for a particular candidate image includes feeding into the trained machine learning engine at least one input parameter for a feature that reflects how similar the particular candidate image is to images in other video items that belong to a collection to which the particular video item belongs.

12. The method of claim 1 wherein feeding a set of input parameter values into the trained machine learning engine for a particular candidate image includes feeding into the trained machine learning engine at least one input parameter for at least one of:

an orientation feature;

a contrast feature;

a saturation feature;

a brightness feature;

a sharpness feature;

a colorfulness feature;

or

a SIFT or SURF feature.

13. The method of claim 1 further comprising employing an explore/exploit technique to test actual user responses to different representative images of the particular video item.

14. The method of claim 13 further comprising training the machine learning engine based, at least in part, on results obtained by employing the explore/exploit technique.

15. One or more non-transitory computer-readable media storing instructions which, when executed, cause performance of a method that comprises the steps of:

generating a plurality of model-generated scores;

wherein each model-generated score of the plurality of model-generated scores corresponds to a candidate image from a plurality of candidate images for a particular video item;

wherein generating the plurality of model-generated scores includes, for each candidate image of the plurality of candidate images, feeding a set of input parameter values into a trained machine learning engine to produce the model-generated score that corresponds to the candidate image;

establishing a ranking of the candidate images for the particular video item based, at least in part, on the model-generated scores that correspond to the candidate images;

selecting a candidate image, from the plurality of candidate images, as a representative image for the video item based, at least in part, on the ranking;

wherein the method is performed by one or more computing devices.

16. The one or more non-transitory computer-readable media of claim 15 wherein the method further comprises automatically selecting the plurality of candidate images for which to generate model-generated scores, by performing the steps of:

dividing the particular video item into a plurality of scenes by detecting scene boundaries within the particular video item; and

selecting a particular number of candidate videos from each of the plurality of scenes.

17. The one or more non-transitory computer-readable media of claim 15 wherein feeding a set of input parameter values into the trained machine learning engine includes feeding into the trained machine learning engine at least one input parameter for a target-specific feature.

18. The one or more non-transitory computer-readable media of claim 15 wherein feeding a set of input parameter values into the trained machine learning engine includes feeding into the trained machine learning engine at least one input parameter for a request-specific feature.

19. The one or more non-transitory computer-readable media of claim 15 wherein the method further comprises employing an explore/exploit technique to test actual user responses to different representative images of the particular video item.

20. The one or more non-transitory computer-readable media of claim 19 wherein the method further comprises training the machine learning engine based, at least in part, on results obtained by employing the explore/exploit technique.