Video search engine using joint categorization of video clips and queries based on multiple modalities

• US 20070255755A1
• Filed: 05/01/2006
• Published: 11/01/2007
• Est. Priority Date: 05/01/2006
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Dai, Wei, Chow, Jyh-Herng, Zhang, Ruofei, Sarukkai, Ramesh
• Application Number:
  US11/415,838
• Publication Number:
  US 20070255755A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content

Abstract

A method comprises generating a first classification model, e.g., metadata-based, for determining whether a video belongs to a category; generating a second classification model, e.g., content-based, for determining whether the video belongs to a category, the first classification model and second classification model being based on different modalities; and generating a fusion model that blends the categorization results of the models. Each classification model may classify the video to multiple categories. During operation, a method obtains a video; uses the first classification model, the second classification model and the fusion model to determine whether the video belongs to a category; and indexes the video in a video index. The method may enable selection of a category corresponding to the video search results. The category may be identified based on a query profile, which may be learned from users'"'"' query logs or popular queries and click history.

Claims

1. A method comprising:

generating a first classification model for determining whether a video belongs to a category;

generating a second classification model for determining whether the video belongs to the category, the first classification model being based on a different modality than the second classification model; and

generating a fusion model that uses the results of the first classification model and the second classification model for determining whether the video belongs to the category.

2. The method of claim 1, wherein the first classification model includes a metadata-based classification model.

3. The method of claim 1, wherein the second classification model includes a content-based classification model.

4. The method of claim 3, wherein the generating the second classification model includes extracting a keyframe from the video clip and extracting visual features from the keyframe.

5. The method of claim 1, wherein each of the steps of generating a classification model uses statistical pattern learning.

6. The method of claim 1, wherein the step of generating a fusion model uses query profiles generated by a learning algorithm using users'"'"' query logs and click history data.

7. A system comprising:

a first learning engine for generating a first classification model for determining whether a video belongs to a category;

a second leaning engine for generating a second classification model for determining whether the video belongs to the category, the first classification model being based on a different modality than the second classification model; and

a third learning engine for generating a fusion model that uses the results of the first classification model and the second classification model for determining whether the video belongs to the category.

8. The system of claim 7, wherein the first classification model includes a metadata-based classification model.

9. The system of claim 7, wherein the second classification model includes a content-based classification model.

10. The system of claim 9, further comprising

a video analysis component for extracting a keyframe from the video clip;

and

a feature extraction component for extracting visual features from the keyframe.

11. The system of claim 7, wherein each of the first and second learning engines uses statistical pattern learning.

12. The system of claim 7, wherein the third learning engine uses query profiles generated by a learning algorithm using users'"'"' query logs and click history data.

13. A method comprising:

obtaining a video clip;

using a first classification model to determine whether the video belongs to a category;

using a second classification model to determine whether the video belongs to the category, the first classification model being based on a different modality than the second classification model;

using a fusion model that uses the results of the first classification model and the second classification model to determine whether the video clip belongs to the category; and

indexing the video based on the result of the fusion model in a video index.

14. The method of claim 13, wherein the first classification model includes a metadata-based classification model.

15. The method of claim 13, wherein the second classification model includes a content-based classification model.

16. The method of claim 13, wherein the step of generating a fusion model uses query profiles generated by a learning algorithm using users'"'"' query logs and click history data.

17. The method of claim 15, further comprising extracting a keyframe from the video clip and extracting visual features from the keyframe.

18. The method of claim 13, further comprising generating video search results in response to a query and enabling selection of a category corresponding to the query.

19. The method of claim 18, wherein the category is identified from the possible categories of a subset of the video search results.

20. The method of claim 18, wherein the category is identified based on a query profile associated with the query.

21. The method of claim 20, wherein the query profile is determined based on users'"'"' query logs and click history.

22. The method of claim 20, wherein the query profile is determined based on popular queries and click history.

23. A system comprising:

a first classification model for determining whether a video clip belongs to a category;

a second classification model for determining whether the video clip belongs to the category, the first classification model being based on a different modality than the second classification model;

a fusion model that uses the results of the first classification model and the second classification model for determining whether the video belongs to the category; and

an index building component for indexing the video based on the result of the fusion model in a video index.

24. The system of claim 23, wherein the first classification model includes a metadata-based classification model.

25. The system of claim 23, wherein the second classification model includes a content-based classification model.

26. The system of claim 25, further comprising a video analysis component for extracting a keyframe from the video;

and

a feature extraction component for extracting visual features from the keyframe.

27. The system of claim 23, further comprising a video search engine for generating video search results in response to a query and enabling selection of a category corresponding to the query.

28. The system of claim 27, wherein the video search engine identifies the category from the possible categories of a subset of the video search results.

29. The system of claim 27, wherein the video search engine identifies the category based on a query profile associated with the query.

30. The system of claim 29, wherein the video search engine determines the query profile based on users'"'"' query logs and click history.

31. The system of claim 29, wherein the video search engine determines the query profile based on popular queries and click history.