RICH MEDIA CONTENT SEARCH ENGINE

• US 20080270344A1
• Filed: 04/30/2007
• Published: 10/30/2008
• Est. Priority Date: 04/30/2007
• Status: Abandoned Application

• Current Assignee: Sonic Foundry Incorporated
• Original Assignee: Sonic Foundry Incorporated
• Inventors: Hancock, John, Scott, Jonathan, Knight, Michael J., Yurick, Steven J.
• Application Number:
  US11/742,125
• Publication Number:
  US 20080270344A1
• Time in Patent Office: Days
• US Class Current: 707/2
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/61 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data

Abstract

A method of generating a set of search results. An audio content search results set including an individual audio content search result corresponding to a rich media time segment is generated. A visual content search results set including an individual visual content search result corresponding to the rich media time segment is also generated. A relevance of the rich media time segment is determined based at least in part on an individual search result count. The individual search result count is a sum of a number of individual audio content search results corresponding to the rich media time segment and a number of individual visual content search results corresponding to the rich media time segment. The rich media time segment is included in an ordered set of search results, wherein an order of the rich media time segment is based at least in part on the determined relevance.

Claims

1. A method of generating a set of search results, the method comprising:

(a) generating an audio content search results set comprising an individual audio content search result corresponding to a rich media time segment;

(b) generating a visual content search results set comprising an individual visual content search result corresponding to the rich media time segment;

(c) determining a relevance of the rich media time segment based at least in part on an individual search result count, wherein the individual search result count comprises a sum of a number of individual audio content search results corresponding to the rich media time segment and a number of individual visual content search results corresponding to the rich media time segment; and

(d) including the rich media time segment in an ordered set of search results, wherein an order of the rich media time segment is based at least in part on the determined relevance.

2. The method of claim 1, further comprising generating a textual metadata content search results set comprising an individual textual metadata search result corresponding to the rich media time segment.

3. The method of claim 2, wherein the individual search result count comprises a sum of the number of individual audio content search results corresponding to the rich media time segment, the number of individual visual content search results corresponding to the rich media time segment, and a number of individual textual metadata content search results corresponding to the rich media time segment.

4. The method of claim 2, wherein the individual textual metadata content search result is obtained from at least one of an abstract describing the rich media time segment, a presenter name, a title of the rich media time segment, and an annotation provided by a viewer of the rich media presentation.

5. The method of claim 1, wherein the individual visual content search result is generated by:

obtaining a representation of visual content corresponding to the rich media time segment;

creating a visual content index based on the representation of the visual content; and

comparing a received search query to the visual content index.

6. The method of claim 5, wherein the representation comprises a textual representation.

7. The method of claim 5, wherein obtaining the representation comprises performing an optical character recognition operation on the visual content.

8. The method of claim 7, further comprising performing a duplicate frame detection operation such that the representation is not duplicatively obtained from frames of the visual content which are substantially identical.

9. The method of claim 5, wherein obtaining the representation comprises extracting the representation from formatted text in the visual content.

10. The method of claim 5, wherein obtaining the representation comprises extracting the representation from a software application file.

11. The method of claim 5, wherein obtaining the representation comprises performing an object recognition operation on the visual content.

12. The method of claim 5, wherein obtaining the representation comprises performing a face recognition operation on the visual content.

13. The method of claim 5, further comprising performing a conditioning operation on the representation.

14. The method of claim 13, wherein the conditioning operation comprises a tokenization operation in which the representation is separated into words.

15. The method of claim 14, further comprising evaluating the words to determine whether the words are valid.

16. The method of claim 13, wherein the conditioning operation comprises a stemming operation in which word stems of the words are identified.

17. The method of claim 5, wherein the visual content index comprises an inverted index.

18. The method of claim 5, wherein the visual content index includes a timestamp corresponding to the representation and based on a playback time of the representation.

19. The method of claim 1, wherein the individual audio content search result is obtained by:

obtaining a representation of audio content corresponding to the rich media time segment;

creating an audio content index based on the representation of the audio content; and

comparing a received search query to the audio content index.

20. The method of claim 19, wherein the representation comprises a textual representation of words uttered in the audio content.

21. The method of claim 19, wherein obtaining the representation comprises performing an automatic speech recognition operation on the audio content.

22. The method of claim 19, wherein the audio content index comprises an inverted index.

23. The method of claim 19, wherein the audio content index includes a timestamp corresponding to the representation and based on a playback time of the representation.

24. The method of claim 19, wherein the representation comprises a phoneme sequence corresponding to the audio content.

25. The method of claim 19, wherein the audio content index comprises a k-phoneme index.

26. The method of claim 19, wherein the received search query is compared to the audio content index using a phoneme matching algorithm capable of matching phonemes based on a phonetic pronunciation of the received search query to phonemes in the audio content index.

27. The method of claim 1, wherein the relevance is further based at least in part on a completeness of the individual audio content search result, wherein the completeness of the individual audio content search result comprises an extent to which the individual audio content search result matches a received search query.

28. The method of claim 27, wherein the completeness of the individual audio content search result is based at least in part on whether at least a portion of the individual audio content search result is an exact match with at least a portion of the received search query.

29. The method of claim 27, wherein the completeness of the individual audio content search result is based at least in part on whether at least a portion of the individual audio content search result comprises a stem of a word in the received search query.

30. The method of claim 27, wherein the completeness of the individual audio content search result is based at least in part on whether an order of words within the individual audio content search result matches an order of words within the received search query.

31. The method of claim 27, wherein the completeness of the individual audio content search result is based at least in part on a number of distinct words from a received search query which occur in audio content corresponding to the rich media time segment.

32. The method of claim 1, wherein the relevance is further based at least in part on a confidence score of the individual audio content search result.

33. The method of claim 1, wherein the relevance is further based at least in part on a matching score of the individual audio content search result.

34. The method of claim 1, wherein the relevance is further based at least in part on a number of search results sets in which the rich media time segment appears.

35. The method of claim 1, wherein the relevance is further based at least in part on a reliability of a type of search results set in which the rich media time segment appears.

36. The method of claim 1, wherein the relevance is further based at least in part on a relevance of a type of search results set in which the rich media time segment appears.

37. The method of claim 1, wherein the relevance is further based at least in part on a temporal proximity within the rich media time segment of the individual audio content search result with respect to the individual visual content search result.

38. The method of claim 1, wherein the relevance is further based at least in part on a temporal proximity of the individual audio content search result to a second individual audio content search result within audio content corresponding to the rich media time segment.

39. The method of claim 1, wherein the relevance is further based at least in part on user feedback.

40. The method of claim 39, wherein the user feedback comprises information provided by a user and regarding the rich media time segment.

41. The method of claim 39, wherein the user feedback is based on a user'"'"'s interaction with the ordered set of search results.

42. The method of claim 39, wherein the user feedback is based on a frequency with which users experience the rich media time segment.

43. The method of claim 1, wherein the relevance is further based at least in part on a contextual analysis.

44. The method of claim 43, wherein the contextual analysis comprises analyzing a subset of audio content in temporal proximity to the individual audio content search result to determine whether the individual audio content search result is relevant to a received search query.

45. The method of claim 43, wherein the contextual analysis comprises analyzing a subset of visual content in temporal proximity to the individual visual content search result to determine whether the individual visual content search result is relevant to a received search query.

46. The method of claim 43, wherein the contextual analysis comprises analyzing all audio content associated with the rich media time segment to determine whether the individual audio content search result is relevant to a received search query.

47. The method of claim 43, wherein the contextual analysis comprises analyzing all visual content associated the rich media time segment to determine whether the individual visual content search result is relevant to a received search query.

48. The method of claim 43, wherein the contextual analysis is implemented using at least one of a lexical database and a semantic similarity measurement.

49. The method of claim 1, further comprising:

receiving a search query; and

performing a search query expansion on the search query to determine a word related to the search query.

50. The method of claim 49, wherein the individual audio content search result is generated at least in part by comparing the word related to the search query to audio content corresponding to the rich media time segment.

51. The method of claim 49, wherein the relevance is further based at least in part on whether the individual audio content search result is based on the search query or the word related to the search query.

52. The method of claim 49, wherein the individual visual content search result is generated at least in part by comparing the word related to the search query to visual content corresponding to the rich media time segment.

53. The method of claim 49, wherein the relevance is further based at least in part on whether the individual visual content search result is based on the search query or the word related to the search query.

54. The method of claim 1, wherein the rich media time segment comprises an entire rich media presentation.

55. The method of claim 1, wherein the rich media time segment comprises a portion of a rich media presentation.

56. The method of claim 1, wherein the relevance is further based at least in part on a completeness of the individual visual content search result, wherein the completeness of the individual visual content search result comprises an extent to which the individual visual content search result matches a received search query.

57. The method of claim 56, wherein the completeness of the individual visual content search result is based at least in part on whether at least a portion of the individual visual content search result is an exact match with at least a portion of the received search query.

58. The method of claim 56, wherein the completeness of the visual content search result is based at least in part on whether at least a portion of the individual visual content search result comprises a stem of a word in the received search query.

59. The method of claim 56, wherein the completeness of the individual visual content search result is based at least in part on whether an order of words within the individual visual content search result matches an order of words within the received search query.

60. The method of claim 56, wherein the completeness of the individual visual content search result is based at least in part on a number of distinct words from a received search query which occur in visual content corresponding to the rich media time segment.

61. The method of claim 1, wherein the relevance is further based at least in part on a confidence score of the individual visual content search result.

62. The method of claim 1, wherein the relevance is further based at least in part on a matching score of the individual visual content search result.

63. A computer-readable medium having computer-readable instructions stored thereon that, upon execution by a processor, cause the processor to generate a set of search results, the instructions configured to:

(a) generate an audio content search results set comprising an individual audio content search result corresponding to a rich media time segment;

(b) generate a visual content search results set comprising an individual visual content search result corresponding to the rich media time segment;

(c) determine a relevance of the rich media time segment based at least in part on an individual search result count, wherein the individual search result count comprises a sum of a number of individual audio content search results corresponding to the rich media time segment and a number of individual visual content search results corresponding to the rich media time segment; and

(d) include the rich media time segment in an ordered set of search results, wherein an order of the rich media time segment is based at least in part on the determined relevance.

64. A system for generating a set of search results, the system comprising:

(a) a search results fusion application, wherein the search results fusion application comprises computer code configured to

receive an audio content search results set comprising an individual audio content search result associated with a rich media time segment;

receive a visual content search results set comprising an individual visual content search result associated with the rich media time segment;

determine a relevance of the rich media time segment; and

include the rich media time segment in a set of search results based on the determined relevance;

(b) a memory configured to store the search results fusion application; and

(c) a processor coupled to the memory and configured to execute the search results fusion application.