AUDIO CONTENT SEARCH ENGINE

• US 20080270138A1
• Filed: 04/30/2007
• Published: 10/30/2008
• Est. Priority Date: 04/30/2007
• Status: Active Grant

• Current Assignee: Sonic Foundry Incorporated
• Original Assignee: Sonic Foundry Incorporated
• Inventors: Hancock, John, Scott, Jonathan, Knight, Michael J., Yurick, Steven J.
• Granted Patent: US 7,983,915 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/61 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

A method of generating an audio content index for use by a search engine includes determining a phoneme sequence based on recognized speech from an audio content time segment. The method also includes identifying k-phonemes which occur within the phoneme sequence. The identified k-phonemes are stored within a data structure such that the identified k-phonemes are capable of being compared with k-phonemes from a search query.

Claims

1. A method of generating an audio content index for use by a search engine, the method comprising:

(a) determining a phoneme sequence based on recognized speech from an audio content time segment;

(b) identifying k-phonemes which occur within the phoneme sequence; and

(c) storing the identified k-phonemes within a data structure such that the identified k-phonemes are capable of being compared with k-phonemes from a search query.

2. The method of claim 1, wherein determining the phoneme sequence comprises looking up words in the recognized speech in a phonetic pronunciation dictionary.

3. The method of claim 1, wherein the identified k-phonemes comprise at least one of phonemes, bi-phonemes, tri-phonemes, quad-phonemes, and penta-phonemes.

4. The method of claim 1, wherein the identified k-phonemes and the k-phonemes from the search query comprise tri-phonemes.

5. The method of claim 1, wherein the identified k-phonemes comprise order independent k-phonemes.

6. The method of claim 1, wherein the identified k-phonemes comprise order dependent k-phonemes.

7. The method of claim 1, wherein the data structure comprises a multi-array lookup table.

8. The method of claim 7, wherein an array in the multi-array lookup table comprises entries of a plurality of distinct audio content time segments.

9. The method of claim 7, wherein an array in the multi-array lookup table comprises a mapped sequence of the identified k-phonemes.

10. The method of claim 7, wherein an array in the multi-array lookup table comprises starting locations within the phoneme sequence of the identified k-phonemes.

11. The method of claim 1, further comprising using an ordinal function to sequentially map the k-phonemes into the data structure.

12. The method of claim 1, further comprising recognizing the recognized speech with an automatic speech recognition algorithm.

13. The method of claim 1, further comprising storing timestamps corresponding to the identified k-phonemes in the data structure.

14. A method of locating information within an audio content time segment comprising:

(a) identifying k-phonemes of a search query based on a phonetic pronunciation of the search query;

(b) identifying k-phonemes within an audio content index which match the identified k-phonemes of the search query to determine an audio content index starting location, wherein the audio content index comprises phoneme data associated with an audio content time segment; and

(c) performing a comparison of the phonetic pronunciation to the phoneme data at the audio content index starting location to identify an audio content search result.

15. The method of claim 14, further comprising receiving the search query.

16. The method of claim 14, further comprising determining the phonetic pronunciation of the search query.

17. The method of claim 16, wherein the phonetic pronunciation of the search query is determined using a phonetic pronunciation dictionary.

18. The method of claim 16, wherein the phonetic pronunciation of the search query is determined using a rules-based guessing algorithm.

19. The method of claim 16, wherein the phonetic pronunciation of the search query is determined using a phonetic pronunciation dictionary and a rules-based guessing algorithm.

20. The method of claim 14, wherein the comparison is scored based on at least one of an insertion, a substitution, and a deletion identified by a phoneme matching algorithm used to implement the comparison.

21. The method of claim 14, further comprising generating a phonetic lattice based on the phonetic pronunciation of the search query.

22. The method of claim 21, wherein the phonetic lattice comprises a compressed phonetic graph.

23. The method of claim 14, further comprising identifying bi-phonemes of the search query.

24. The method of claim 23, wherein identifying the k-phonemes within the audio content index comprises:

comparing the identified bi-phonemes of the search query to the audio content index to identify one or more matching bi-phonemes such that one or more initial starting locations are identified; and

comparing the identified k-phonemes of the search query to the audio content index at the one or more initial starting locations to identify matching k-phonemes within the audio content index.

25. The method of claim 14, wherein the identified k-phonemes within the audio content index and the identified k-phonemes of the search query comprise at least one of bi-phonemes, tri-phonemes, quad-phonemes, and penta-phonemes.

26. The method of claim 14, wherein the comparison of the phonetic pronunciation to the phoneme data comprises a phoneme comparison in a forward direction from the audio content index starting location and a phoneme comparison in a backward direction from the audio content index starting location.

27. The method of claim 14, wherein the comparison of the phonetic pronunciation to the phoneme data comprises at least one of a phoneme comparison in a forward direction from the audio content index starting location and a phoneme comparison in a backward direction from the audio content index starting location.

28. The method of claim 14, wherein the comparison comprises a depth first comparison such that a first comparison at a first audio content index starting location is completed prior to commencing a second comparison at a second audio content index starting location.

29. The method of claim 14, wherein the comparison comprises a heuristic comparison based on a comparison relevance such that a comparison with a higher comparison relevance is implemented prior to a comparison with a lower comparison relevance, wherein the comparison relevance is determined by a phoneme matching algorithm.

30. The method of claim 14, further comprising assigning a score to the audio content search result based on a quality of the comparison.

31. The method of claim 30, wherein the score is based at least in part on whether the comparison comprises an exact match, an insertion, a substitution, or a deletion.

32. The method of claim 30, wherein the comparison at the audio content starting location is terminated if the score exceeds a predetermined score threshold.

33. The method of claim 30, wherein the score comprises a first score based on a phoneme comparison in a forward direction from the audio content index starting location and a second score based on a phoneme comparison in a backward direction from the audio content index starting location.

34. The method of claim 14, further comprising performing (a)-(c) in parallel to locate audio content search results in a plurality of audio content time segments.

35. The method of claim 14, further comprising forming a valid sequence comprising one or more audio content search results from the audio content time segment.

36. The method of claim 35, wherein if the valid sequence comprises a plurality of audio content search results, the plurality of audio content search results occur in the audio content time segment within a specified time of one another.

37. The method of claim 35, wherein if the valid sequence comprises a plurality of audio content search results, the plurality of audio content search results occur in the same order as words in the search query.

38. The method of claim 35, further comprising determining a confidence score for the valid sequence, wherein the confidence score comprises a likelihood that the valid sequence is a true occurrence of the search query.

39. The method of claim 38, wherein the confidence score is determined using a curve fitting function and an observed trend from a language corpora.

40. The method of claim 38, wherein the confidence score is based at least in part on a total penalty score assigned to the valid sequence, wherein the total penalty score is based at least in part on one or more audio content search result scores assigned to the one or more audio content search results within the valid sequence and a number of words from the search query which are missing from the valid sequence.

41. The method of claim 40, wherein the confidence score is based at least in part on a match percentage score, wherein the match percentage score comprises the total penalty score divided by a maximum possible penalty score.

42. The method of claim 38, wherein the confidence score is based at least in part on a number of words in the search query.

43. The method of claim 38, wherein the confidence score is based at least in part on a phonetic uniqueness score of the search query.

44. The method of claim 14, further comprising including the audio content time segment in an ordered audio content search results set.

45. The method of claim 44, wherein an order of the audio content time segment is based at least in part on a confidence score of a valid sequence within the audio content time segment.

46. The method of claim 44, wherein an order of the audio content time segment is determined using a mathematical formula.

47. The method of claim 46, wherein the mathematical formula comprises a score summation formula capable of being used to sum at least one of match scores associated with the audio content time segment and confidence scores associated with the audio content time segment.

48. The method of claim 46, wherein the mathematical formula includes an exponential power factor corresponding to at least one of a match score associated with the audio content time segment and a confidence score associated with the audio content time segment.

49. The method of claim 44, wherein an order of the audio content time segment is based at least in part on a score associated with the audio content time segment and a number of individual audio content search results in the audio content time segment.

50. The method of claim 44, wherein an order of the audio content time segment in the ordered audio content search results set is based at least in part on whether the identified k-phonemes within the audio content index occur in the same order as the identified k-phonemes of the search query.

51. The method of claim 44, wherein an order of the audio content time segment in the ordered audio content search results set is based at least in part on identified k-phonemes within the audio content index which do not appear in at least a portion of the search query.

52. The method of claim 44, wherein an order of the audio content time segment is determined based on a relevance string corresponding to the audio content time segment.

53. The method of claim 52, wherein the relevance string is based at least in part on a number of conjunctive results within the audio content time segment.

54. The method of claim 52, wherein the relevance string is based at least in part on a confidence score of a valid sequence within the audio content time segment.

55. The method of claim 44, further comprising providing the ordered audio content search results set to a multi-modal search results fusion engine.

56. A system for locating information within an audio content time segment comprising:

(a) an audio content search application, wherein the audio content search application comprises computer code configured to

identify k-phonemes of a search query based on a phonetic pronunciation of the search query;

identify k-phonemes within an audio content index which match the identified k-phonemes of the search query to determine an audio content index starting location, wherein the audio content index comprises phoneme data associated with an audio content time segment; and

perform a comparison of the phonetic pronunciation to the phoneme data at the audio content index starting location to identify an audio content search result;

(b) a memory configured to store the audio content search application; and

(c) a processor coupled to the memory and configured to execute the audio content search application.