Media content search engine incorporating text content and user log mining

• US 20020161747A1
• Filed: 03/13/2001
• Published: 10/31/2002
• Est. Priority Date: 03/13/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Chen, Zheng, Zhang, Hong-Jiang, Liu, Wen-Yin
• Granted Patent: US 7,231,381 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: Y10S 707/913 Multimedia
• CPC Class Codes: Y10S 707/915 Image
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

Text features corresponding to pieces of media content (e.g., images, audio, multimedia content, etc.) are extracted from media content sources. One or more text features (e.g., one or more words) for a piece of media content are extracted from text associated with the piece of media content and text feature vectors generated therefrom and used during subsequent searching. Additional low-level feature vectors may also be extracted from the piece of media content and used during the subsequent searching. Relevance feedback can also be received from a user(s) identifying the relevance of pieces of media content rendered to the user in response to his or her search request. The relevance feedback is logged and can be used in determining how to respond to subsequent search requests, such as by modifying feature vectors (e.g., text feature vectors) corresponding to the pieces of media content for which relevance feedback is received.

Claims

1. One or more computer-readable media having stored thereon a plurality of instructions that, when executed by one or more processors of a computer, causes the one or more processors to perform the following acts:

receiving search criteria;

generating a query vector based on text features of the search criteria;

identifying media content pieces to be rendered by comparing the query vector to text feature vectors associated with a plurality of media content pieces;

receiving user feedback regarding the relevancy of the identified media content pieces;

modifying the query vector based on the user feedback;

modifying one or more of the text feature vectors associated with the plurality of media content pieces based on the user feedback; and

identifying new media content pieces to be rendered by comparing the modified query vector to the text feature vectors, including the one or more modified text feature vectors, associated with the plurality of media content pieces.

2. One or more computer readable media as recited in claim 1, further comprising:

generating another query vector based on one or more low-level features of the search criteria; and

wherein the identifying comprises,

comparing the query vector to text feature vectors associated with the plurality of media content pieces to generate first results,

comparing the other query vector to other low-level feature vectors associated with the plurality of media content pieces to generate second results, and

combining, for one of the plurality of media content pieces, the first and second results corresponding to the one media content piece.

3. One or more computer readable media as recited in claim 2, further comprising altering, based on the user feedback, a weighting of the results used in the combining.

4. One or more computer readable media as recited in claim 3, wherein the altering comprises:

determining, for the one of the plurality of media content pieces, whether the first result corresponding to the one media content piece is greater than the second result corresponding to the one media content piece; and

weighting the first result corresponding to the one media content piece more heavily if the first result corresponding to the one media content piece is less than the second result corresponding to the one media content piece, and otherwise weighting the second result corresponding to the one media content piece more heavily.

5. One or more computer readable media as recited in claim 1, wherein modifying one or more of the text feature vectors associated with the plurality of media content pieces based on the user feedback comprises altering a weighting of one or more elements in the feature vector based on the user feedback.

6. One or more computer readable media as recited in claim 1, wherein the search criteria comprises one or more words.

7. One or more computer readable media as recited in claim 1, wherein the piece of media content comprises an image.

8. One or more computer readable media as recited in claim 1, wherein the piece of media content comprises a portion of audio content.

9. One or more computer readable media as recited in claim 1, wherein the piece of media content comprises a portion of multimedia content.

10. A method comprising:

identifying a media content source;

collecting one or more pieces of media content and associated text from the media content source;

extracting, for a piece of media content, one or more text features from the associated text; and

making the one or more text features available for searching.

11. A method as recited in claim 10, further comprising:

generating one or more text feature vectors from the extracted one or more text features; and

wherein the making comprises making the one or more text feature vectors available for searching.

12. A method as recited in claim 10, further comprising:

extracting one or more low-level features from the media content piece; and

making the one or more low-level features available for searching.

13. A method as recited in claim 10, further comprising, for each collected piece of media content:

classifying the image as meaningful or not meaningful; and

wherein the extracting comprises extracting the one or more text features for the piece of media content only if the piece of media content is classified as meaningful.

14. A method as recited in claim 10, wherein the media content source comprises a web site including a plurality of web pages, each web page including a plurality of pieces of media content and text associated with one or more of the plurality of pieces of media content.

15. A method as recited in claim 10, wherein the associated text for a piece of media content comprises a filename and the one or more text features comprises one or more words in the filename.

16. A method as recited in claim 10, wherein the associated text for a piece of media content comprises a uniform resource locator (URL) and the one or more text features comprises one or more words in the URL.

17. A method as recited in claim 10, wherein the associated text for a piece of media content comprises alternate text that can be displayed in place of he media content, and the one or more text features comprises one or more words of the alternate text.

18. A method as recited in claim 10, wherein the associated text for a piece of media content comprises text surrounding the piece of media content on a web page, and the one or more text features comprises one or more words of the text surrounding the piece of media content.

19. A method as recited in claim 10, wherein the associated text for a piece of media content comprises a title of a web page that includes the piece of media content, and the one or more text features comprises one or more words in the title.

20. A method as recited in claim 10, wherein the associated text for a piece of media content comprises a link on a web page that includes the piece of media content, and the one or more text features comprises one or more words in the link.

21. A method as recited in claim 10, wherein the associated text for a piece of media content comprises anchor text corresponding to the piece of media content, and the one or more text features comprises one or more words in the anchor text.

22. A method as recited in claim 10, wherein the associated text for a piece of media content comprises an image annotation corresponding to the piece of media content, and the one or more text features comprises one or more words in the image annotation.

23. A method as recited in claim 10, wherein each of the one or more pieces of media content comprises an image.

24. A method as recited in claim 10, wherein each of the one or more pieces of media content comprises a piece of audio content.

25. A method as recited in claim 10, wherein each of the one or more pieces of media content comprises a piece of multimedia content.

26. One or more computer-readable memories containing a computer program that is executable by a processor to perform the method recited in claim 10.

28. A method as recited in claim 27, further comprising:

rendering a plurality of pieces of media content to the user;

receiving feedback from the user regarding one or more of the plurality of pieces of media content;

modifying the query vector based on the feedback; and

repeating the comparing and determining based on the modified query vector.

29. A method as recited in claim 27, further comprising:

generating another query vector based on one or more low-level features of the search criteria;

comparing the other query vector to another feature vector corresponding to the piece of media content and having been generated based on the one or more low-level features of the piece of media content;

combining a result of the other query vector to the other feature vector with the result of comparing the query vector to the feature vector; and

wherein the determining comprises determining, based at least in part on the combined result, whether to render the piece of media content to the user.

30. A method as recited in claim 29, further comprising:

accessing user feedback regarding the relevancy of one or more pieces of media content rendered to the user; and

altering, based on the user feedback, a weighting of the results during the combining.

31. A method as recited in claim 30, wherein the altering comprises:

determining whether a first distance between the other query vector and the other feature vector is greater than a second distance between the query vector and the feature vector; and

weighting the result of the comparing the other query vector to the other feature vector more heavily if the first distance is less than the second distance, and otherwise weighting the result of the comparing the query vector to the feature vector more heavily.

32. A method as recited in claim 27, further comprising:

accessing user feedback regarding the relevancy of one or more pieces of media content rendered to the user; and

modifying the feature vector corresponding to the piece of media content based on the user feedback.

33. A method as recited in claim 32, further comprising:

generating a user space vector corresponding to each of the one or more pieces of media content for which user feedback is accessed; and

using the user space vector corresponding to a particular piece of media content to modify the feature vector corresponding to the particular piece of media content.

34. A method as recited in claim 27, further comprising:

accessing user feedback regarding the relevancy of one or more pieces of media content rendered to the user; and

altering a weighting of one or more elements in the feature vector based on the user feedback.

35. A method as recited in claim 27, wherein the search criteria comprises one or more words.

36. A method as recited in claim 27, wherein the piece of media content comprises an image.

37. A method as recited in claim 27, wherein the piece of media content comprises a piece of audio content.

38. A method as recited in claim 27, wherein the piece of media content comprises a piece of multimedia content.

39. One or more computer-readable memories containing a computer program that is executable by a processor to perform the method recited in claim 27.

27. A method comprising:

receiving search criteria;

generating a query vector based on the search criteria;

comparing the query vector to a feature vector corresponding to a piece of media content and having been generated based on text associated with the piece of media content; and

determining, based at least in part on a result of the comparing, whether to render the piece of media content to a user.

40. A system comprising:

a crawler module coupled to access a media content source and collect a plurality of media content pieces and associated text from the media content source;

a feature extraction module coupled to extract one or more text features from one of the media content pieces; and

a media content indexing module coupled to generate a text feature vector, based on the extracted one or more text features, corresponding to the one media content piece.

41. A system comprising:

a query generator to generate a query vector based on received search criteria; and

a matching module coupled to,

receive the query vector and compare the query vector to a plurality of feature vectors corresponding to a plurality of pieces of media content, wherein each of the plurality of feature vectors has been generated based on text associated with one of the plurality of pieces of media content, and

identify one or more of the plurality of pieces of media content to return for rendering to a user based on the comparison of the query vector to the plurality of feature vectors.

42. A method comprising:

receiving search criteria;

identifying, based at least in part on the search criteria, a piece of media content to be rendered;

receiving user feedback regarding the relevancy of the rendered piece of media content;

weighting for another piece of media content, based on the user feedback, both a result of comparing the high-level query vector to a high-level feature vector of the other piece of media content and a result of comparing the low-level query vector to a low-level feature vector of the other piece of media content; and

combining the weighted result to determine whether to identify the other piece of media content for rendering.

43. A method as recited in claim 42, further comprising generating a new high-level query vector and a new low-level query vector based at least in part on the search criteria.

44. A method as recited in claim 42, further comprising:

generating a user space vector corresponding to the piece of media content; and

using the user space vector corresponding to the piece of media content to modify the high-level feature vector corresponding to the piece of media content.

45. A method as recited in claim 42, further comprising altering a weighting of one or more elements in the feature vector based on the user feedback.

46. A method as recited in claim 42, wherein the high-level feature vector of the other piece of media content is a text feature vector.

47. One or more computer-readable memories containing a computer program that is executable by a processor to perform the method recited in claim 42.

49. A method as recited in claim 48, wherein the feature vector is a text feature vector.

50. One or more computer-readable memories containing a computer program that is executable by a processor to perform the method recited in claim 48.

48. A method comprising:

receiving user feedback regarding the relevancy of one or more pieces of rendered media content; and

for each of the one or more pieces of media content,

modifying, based on the user feedback, a feature vector corresponding to the piece of media content, and

making the modified feature vector available for subsequent searching of the one or more pieces of media content.

51. One or more computer-readable media having stored thereon a plurality of instructions that, when executed by one or more processors of a computer, causes the one or more processors to perform acts including:

identifying a piece of media content to render to a user based at least in part on comparing a query vector corresponding to search criteria of the user and a feature vector corresponding to the piece of media content;

receiving user feedback regarding the relevancy of the piece of media content;

modifying the query vector based on the received user feedback; and

modifying the feature vector based on the received user feedback in an off-line log mining process.

52. One or more computer-readable media as recited in claim 51, wherein modifying the query vector comprises generating a vector U based on pieces of media content identified as relevant in the user feedback, and generating a new query vector D_new according to the following:

D_new=η

U+(1−

η

)D

where η

represents a confidence in the vector U.

53. One or more computer-readable media as recited in claim 52, wherein modifying the query vector comprises generating a vector V based on pieces of media content identified as irrelevant in the user feedback, and generating a new query vector D_final according to the following:

D_final=D_new*(1−

V).

54. One or more computer-readable media as recited in claim 51, wherein the piece of media content comprises one of:

audio content, visual content, and multimedia content.