Database annotation and retrieval including phoneme data

• US 6,990,448 B2
• Filed: 08/23/2001
• Issued: 01/24/2006
• Est. Priority Date: 03/05/1999
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Garner, Philip Neil, Charlesworth, Jason Peter Andrew, Rajan, Jebu Jacob
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Storm, Donald L.
• Application Number:
  US09/934,799
• Publication Number:
  US 20020052740A1
• Time in Patent Office: 1,615 Days
• Field of Search: 704/243, 704/240, 704/259, 704/255, 704/270, 704/256, 704/9, 707/5
• US Class Current: 704/243
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams

Abstract

The data structure is used in accessing a plurality of data files. The data stucture comprises a plurality of annotation storage areas adapted to correspond with the data files, each annotation storage area containing an annotation an annotation representing a time sequential signal and each annotation storage area comprising a plurality of block storage areas each containing phoneme and word data forming a respective temporal block of the annotation and each block having an associated time index identifying a timing of the block within the corresponding annotation. Each block storage area includes a plurality of node storage areas, each asociated with a node which represents a point in time at which a word and/or phoneme begins or ends within the corresponding annotation, and each node storage area having a time offset storage area containing a time offset defining the point in time represented by the node relative to the time index associated with the corresponding block. Each node storage area further has one or more phoneme link storage areas, each having a phoneme storage area containing data identifying a phoneme associated with corresponding node; or one or more word link storage areas, each having a word storage area containing data identifying a word associated with the corresponding node. One or more of the node storage areas has at least one of the phoneme link storage areas and at least one of the word link storage areas.

Claims

1. An apparatus for generating annotation data for annotating a data file comprising audio data, the apparatus comprising:

an automatic speech recognizer operable to generate phoneme data for audio data in the data file;

a word decoder operable to process the phoneme data generated by the automatic speech recognizer to identify words within the phoneme data;

an annotation generator operable to generate annotation data for the data file by combining the generated phoneme data and the words identified by said word decoder, the annotation data being for use in subsequent retrieval of the data file;

an associator operable to associate the generated annotation data with the data file; and

a storage device operable to store the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

2. An apparatus according to claim 1, wherein the annotation data defines a phoneme and word lattice and wherein said annotation generator comprises:

(i) a first generator operable to generate data defining a plurality of nodes within the lattice and a plurality of links connecting the nodes within the lattice;

(ii) a second generator operable to generate data associating a plurality of phonemes of the phoneme data with a respective plurality of links; and

(iii) a third generator operable to generate data associating at least one of the words with at least one of the links.

3. An apparatus according to claim 2, wherein said annotation generator is operable to generate the data defining the phoneme and word lattice in blocks of the nodes.

4. An apparatus according to claim 3, wherein said annotation generator is operable to generate data which defines each block'"'"'s location within the database.

5. An apparatus according to claim 2, wherein said annotation generator is operable to generate data defining time stamp information for each of the nodes.

6. An apparatus according to claim 5, wherein said annotation generator is operable to generate the phoneme and word lattice data in blocks of equal time duration.

7. An apparatus according to claim 5, wherein said data file includes a time sequential signal, and wherein said annotation generator is operable to generate time stamp data which is time synchronized with the time sequential signal.

8. An apparatus according to claim 2, wherein said first generator is operable to define at least one node which is connected to a plurality of other nodes by a plurality of links.

9. An apparatus according to claim 1, wherein the data file includes audio data which defines the speech of a plurality of speakers, and wherein said annotation generator is operable to generate data which defines separate phoneme and word annotation data for the speech of each speaker.

10. An apparatus according to claim 1, wherein said speech recognizer is operable to generate data defining a weighting for phonemes in the phoneme data.

11. An apparatus according to claim 1, wherein said word decoder is operable to generate data defining a weighting for the words identified within the phoneme data.

12. An apparatus according to claim 1, wherein said annotation generator is operable to generate annotation data that defines a phoneme and word lattice arranged in a time-ordered sequence of blocks.

13. An apparatus according to claim 12, wherein the phoneme and word lattice is associated with a time-sequential signal, and wherein said annotation generator is operable to generate the phoneme and word lattice so that:

(i) the time-ordered sequence of blocks of the phoneme and word lattice is time-synchronized with the time-sequential signal; and

(ii) each block of the phoneme and word lattice includes an associated time index identifying a timing of the block within the time-sequential signal.

14. An apparatus according to claim 13, wherein said annotation generator is operable to generate node data which defines a plurality of nodes within the lattice, each node representing a point in time at which a word and/or phoneme begins or ends within the associated time-sequential signal and wherein each node includes a time-offset value defining the point in time represented by the node relative to the time index associated with the corresponding block.

15. An apparatus according to claim 1, wherein said automatic speech recognizer is operable to generate phoneme data that includes a phoneme lattice which identifies a number of different possible phoneme strings which correspond to speech within the audio data, and wherein said annotation generator is operable to generate the annotation data by combining the generated phoneme lattice with the words identified by said word decoder.

16. An apparatus according to claim 1, wherein said annotation generator is operable to generate header data relating to the speech recognizer that generated the phoneme data for the annotation.

17. An apparatus according to claim 16, when said annotation generator is operable to generate header data including data identifying the language and the phoneme sets used by said automatic speech recognizer.

18. An apparatus for generating annotation data for annotating a data file, the apparatus comprising;

an input operable to receive an input voice signal;

a speech recognizer operable (i) to convert the input voice signal into phoneme data and (ii) to process the phoneme data to identify words within the phoneme data;

an annotation generator operable to generate annotation data for the data file by combining the phoneme data and the words obtained by said speech recognizer, the annotation data being for use in subsequent retrieval of the data file;

an associator operable to associate the generated annotation data with the data file; and

a storage device operable to store the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

19. An apparatus according to claim 18, wherein said speech recognizer is operable to generate data defining a weighting for phonemes in the phoneme data.

20. An apparatus according to claim 19, wherein said speech recognizer is operable to generate data defining a weighting for the words.

21. An apparatus according to claim 18, wherein said speech recognizer is operable to generate phoneme data that includes a phoneme lattice which identifies a number of different possible phoneme strings which correspond to speech within the input voice signal, and wherein said annotation generator is operable to generate the annotation data by combining the generated phoneme lattice with the words identified by said speech recognizer.

22. An apparatus according to claim 18, wherein said annotation generator is operable to generate header data relating to the speech recognizer that generated the phoneme data for the annotation.

23. An apparatus according to claim 22, wherein said annotation generator is operable to generate header data including data identifying the language and the phoneme sets used by said speech recognizer.

24. An apparatus for generating annotation data for annotating a data file, the apparatus comprising:

an input operable to receive image data representative of text;

a character recognizer operable to convert the image data into text data;

a converter operable to convert words in the text data into phoneme data; and

an annotation generator operable to generate annotation data for the data file by combining the phoneme data and words in the text data, the annotation data being for use in subsequent retrieval of the data file;

an associator operable to associate the generated annotation data with the data file; and

a storage device operable to store the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

25. An apparatus according to claim 24, wherein said converter comprises an automatic phonetic transcription unit which generates the phoneme data from words within the text data output by said character recognizer.

26. An apparatus according to claim 24, wherein said input comprises a document scanner or a facsimile machine.

27. A method of generating annotation data for annotating a data file comprising audio data, the method comprising the steps of:

using an automatic speech recognizer to generate phoneme data for audio data in the data file;

using a word decoder to process the phoneme data generated by the automatic speech recognizer to identify words within the phoneme data;

generating annotation data for the data file by combining the generated phoneme data and the words identified by the word decoder, the annotation data being for use in subsequent retrieval of the data file;

associating the generated annotation data with the data file; and

storing the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

28. A method according to claim 27, wherein the annotation data defines a phoneme and word lattice and wherein said step of generating annotation data comprises the steps of:

(i) generating data defining a plurality of nodes within the lattice and a plurality of links connecting the nodes within the lattice;

(ii) generating data associating a plurality of phonemes of the phoneme data with a respective plurality of links; and

(iii) generating data associating at least one of the words with at least one of the links.

29. A method according to claim 28, wherein said step of generating annotation data generates data defining time stamp information for each of the nodes.

30. A method according to claim 29, wherein said step of generating annotation data generates data which defines a block'"'"'s location within a database.

31. A method according to claim 29, wherein the data file includes a time sequential signal, and wherein said step of generating annotation data generates time stamp data which is time synchronized with the time sequential signal.

32. A method according to claim 28, wherein the speech recognizer generates data defining a weighting for the phonemes associated with the links.

33. A method according to claim 28, wherein the word decoder generates data defining a weighting for the words associated with the links.

34. A method according to claim 28, wherein said step of defining a plurality of nodes and a plurality of links defines at least one node which is connected to a plurality of other nodes by a plurality of links.

35. A method according to claim 28, wherein said step of generating annotation data generates data defining the phoneme and word lattice in blocks of the nodes.

36. A method according to claim 27, wherein the audio data includes audio data which defines the speech of a plurality of speakers, and wherein said generating step generates data which defines separate phoneme and word annotation data for the speech of each speaker.

37. A method of generating annotation data for annotating a data file, the method comprising the steps of:

receiving an input voice signal;

processing the input voice signal using a speech recognizer to generate phoneme data for the input voice signal;

processing the generated phoneme data to identify words within the phoneme data;

generating annotation data by combining the phoneme data and the word data generated for the input voice signal, the annotation data being for use in subsequent retrieval of the data file;

associating the generated annotation data with the data file; and

storing the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

38. A method according to claim 37, wherein the speech recognizer generates data defining a weighting for the generated phoneme data.

39. A method according to claim 37, wherein said step of processing the generated phoneme data generates data defining a weighting for the words identified within the phoneme data.

40. A method of generating annotation data for use in annotating a data file, the method comprising the steps of:

receiving image data representative of text;

converting the image data into text data using a character recognizer;

converting words in the text data into phoneme data;

generating annotation data for the data file by combining the phoneme data and words within the text data, the annotation data being for use in subsequent retrieval of the data file;

associating the generated annotation data with the data file; and

storing the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

41. A method according to claim 40, wherein said step of converting words into phoneme data uses an automatic phonetic transcription unit which generates the phoneme data for words within the text data output by the character recognizer.

42. A method according to claim 40, wherein said receiving step uses a document scanner or a facsimile machine.

43. A computer readable medium storing computer program code for causing a programmable processing apparatus to become operable to generate annotation data for annotating a data file comprising audio data, the computer readable medium comprising:

code for causing an automatic speech recognizer to generate phoneme data for audio data in the data file;

code for causing a word decoder to process the phoneme data generated by the automatic speech recognizer to identify words within the phoneme data;

code for generating annotation data for the data file by combining the generated phoneme data and the words identified by the word decoder, the annotation data being for use in subsequent retrieval of the data file;

code for associating the generated annotation data with the data file; and

code for storing the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

44. A computer readable medium storing computer executable instructions for causing a programmable processing apparatus to become operable to generate annotation data for annotating a data file, the medium comprising:

code for receiving an input voice signal;

code for causing an automatic speech recognizer to convert the input voice signal into phoneme data;

code for processing the phoneme data to identify words within the phoneme data;

code for generating annotation data by combining the generated phoneme data and the identified words, the annotation data being for use in subsequent retrieval of the data file;

code for associating the generated annotation data with the data file; and

code for storing the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

45. A computer readable medium storing computer executable instructions for causing a programmable processing apparatus to become operable to generate annotation data for annotating a data file, the medium comprising:

code for receiving image data representative of text;

code for converting the image data into text data using a character recognizer;

code for converting words in the text data into phoneme data;

code for generating annotation data for the data file by combining the phoneme data and words within the text data, the annotation data being for use in subsequent retrieval of the data file;

code for associating the generated annotation data with the data file; and

code for storing the annotation data in a database to allow the associated data file to be retrieved by searching for words and phonemes within the stored annotation data.

46. An apparatus for generating annotation data for use in annotating a data file comprising audio data, the apparatus comprising:

an automatic speech recognizer operable to generate phoneme data for audio data in the data file;

a word decoder operable to identify words within the phoneme data generated by said automatic speech recognizer; and

an annotation generator operable to generate annotation data by combining the generated phoneme data and the words identified by said word decoder and operable to generate header data relating to the speech recognizer that generated the phoneme data for the annotation, which header data includes data identifying the language and phoneme set used by said automatic speech recognizer.

47. An apparatus for generating annotation data for use in annotating a data file, the apparatus comprising:

an input operable to receive an input voice signal;

a speech recognizer operable to convert the input voice signal into phoneme data and words; and

an annotation generator operable to generate annotation data by combining the phoneme data and the words and operable to generate header data relating to the speech recognizer that generated the phoneme data for the annotation, which header data includes data identifying the language and the phoneme set used by the speech recognizer.