Methods and apparatus relating to searching of spoken audio data

• US 20060206324A1
• Filed: 02/06/2006
• Published: 09/14/2006
• Est. Priority Date: 02/05/2005
• Status: Active Grant

• Current Assignee: Arlington Technologies, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Aurix Ltd. (Avaya Incorporated)
• Inventors: Skilling, Adrian I., Wright, Howard A.K.
• Granted Patent: US 8,694,317 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

Methods for processing audio data containing speech to produce a searchable index file and for subsequently searching such an index file are provided. The processing method uses a phonetic approach and models each frame of the audio data with a set of reference phones. A score for each of the reference phones, representing the difference of the audio from the phone model, is stored in the searchable data file for each of the phones in the reference set. A consequence of storing information regarding each of the reference phones is that the accuracy of searches carried out on the index file is not compromised by the rejection of information about particular phones. A subsequent search method is also provided which uses a simple and efficient dynamic programming search to locate instances of a search term in the audio. The methods of the present invention have particular application to the field of audio data mining.

Claims

1. A method of processing audio data to provide a searchable data file comprising the step of analysing the audio data with a phonetic recogniser wherein the phonetic recogniser acts on frames of the audio data and determines, for each frame, a score for each of a set of reference phones, the score being indicative of the likelihood that said frame corresponds to said phone characterised in that the score for each of the reference set of phones for each of said audio frames is stored in the searchable data file.

3. A method according to claim 1 wherein the set of reference phones comprises the complete set of phones of a particular language or dialect.

4. A method according to claim 1 wherein the set of reference phones is a preselected sub-set of a larger database of phones.

5. A method according to claim 4 wherein the method comprises the step of using a language recogniser to identify the language/dialect being spoken in the audio data and select an appropriate sub-set of phones as the reference phone set.

6. A method according to claim 1 wherein the reference phone set comprises the totality of phones from a plurality of languages/dialects.

7. A method according to claim 1 wherein the phonetic recogniser determines an interim score for each phone in a particular audio frame and modifies the interim score for each phone based on the scores for the phones determined for one or more of the audio frames immediately preceding and/or following said particular frame.

8. A method according to claim 1 wherein the method includes the step of storing in the searchable data file the relative position of at least some of the audio frames in the audio data.

9. A method according to claim 1 wherein the method includes the step of processing the audio to calculate features relevant to speaker identification and storing these features in the searchable data file.

10. A method according to claim 9 wherein the method includes the step of storing time references along with the features relevant to speaker identification in the searchable data file.

20. A computer program for performing the method of claim 1.

2. A method of processing audio data to provide a searchable data file comprising the step of analysing the audio data with a phonetic recogniser wherein the phonetic recogniser acts on frames of the audio data and determines, for each frame, a score for a plurality of reference phones, the score being indicative of the likelihood that said frame corresponds to said phone characterised in that the searchable data file stores scores for each said audio frame in a simple matrix format.

11. A method of searching audio data for a phonetic search sequence comprising the steps of;

(i) taking a searchable data file having a score for each of a set of reference phones for each of a series of frames of the audio data, the scores being indicative of the likelihood that that particular frame corresponds to that particular phone,

(ii) searching said searchable data file to find likely matches to a phonetic search sequence wherein the scores for the reference phones for each audio frame are used to determine the likely matches.

12. A method as claimed in claim 11 wherein the step of searching for likely matches using the scores for the reference phones comprises the step of using a dynamic programming search.

13. A method as claimed in claim 11 comprising the step of determining at least one phonetic search sequence from a defined search term.

14. A method as claimed in claim 13 wherein the method comprises the step of converting a text search term into one or more phonetic search sequences using a processor.

15. A method as claimed in claim 14 wherein the processor uses letter-to-sound trees and/or phonetic dictionaries to create the one or more search phonetic sequences.

16. A method as claimed in claim 13 wherein the search term is supplied as audio data and wherein the method comprises the step of using a phonetic recogniser/speech recogniser to determine the phonetic search sequence.

17. A searchable data file comprising a phonetic index file, the phonetic index file corresponding to a series of frames of an audio data file and comprising for each frame a score for each of a set of reference phones.

18. A searchable data file as claimed in claim 17 wherein the searchable data file has a simple matrix structure.

19. A searchable data file as claimed in claim 17 wherein the searchable data file further comprises the audio file containing data allowing the reproduction of the audio.

21. An apparatus for acting on audio data to create a searchable data file comprising a reference set of phones, a phonetic recogniser adapted to compare a frame of audio data with the reference set of phones and to output a score indicative of the likelihood that said frame corresponds to each phone, and a data output for creating a searchable data file comprising, for each audio frame, the score for each of the set of reference phones.