Attribute-based word modeling

• US 6,963,837 B1
• Filed: 10/06/2000
• Issued: 11/08/2005
• Est. Priority Date: 10/06/1999
• Status: Active Grant

• Current Assignee: Multimodal Technologies Incorporated (3M Company)
• Original Assignee: Multimodal Technologies Incorporated (3M Company)
• Inventors: Fritsch, Jurgen, Koll, Detleff, Finke, Michael, Waibel, Alex
• Primary Examiner(s): CHAWAN, VIJAY B
• Application Number:
  US09/680,925
• Time in Patent Office: 1,859 Days
• Field of Search: 704/256, 704/255, 704/230, 704240-244, 704/275, 704/270, 704/232, 704/245, 704/257, 704/270.1, 704/248
• US Class Current: 704/256
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams

Abstract

An attribute-based speech recognition system is described. A speech pre-processor receives input speech and produces a sequence of acoustic observations representative of the input speech. A database of context-dependent acoustic models characterize a probability of a given sequence of sounds producing the sequence of acoustic observations. Each acoustic model includes phonetic attributes and suprasegmental non-phonetic attributes. A finite state language model characterizes a probability of a given sequence of words being spoken. A one-pass decoder compares the sequence of acoustic observations to the acoustic models and the language model, and outputs at least one word sequence representative of the input speech.

Claims

1. An attribute-based speech recognition system comprising:

A speech pre-processor that receives input speech and produces a sequence of acoustic observations representative of the input speech;

A database of context-dependent acoustic models that characterize a probability of a given sequence of sounds producing the sequence acoustic observations, each acoustic model including phonetic attributes and suprasegmental non-phonetic attributes;

A finite state language model that characterizes a probability of a given sequence of words being spoken; and

A one-pass decoder that compares the sequence of acoustic observations to the acoustic models and the language model, and outputs at least one word sequence representative of the input speech.

2. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include speaking rate.

3. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include phone durations.

4. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include dialect.

5. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include gender.

6. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include fundamental frequencies.

7. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include hidden Markov model state.

8. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include word class.

9. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include articulatory features.

10. A speech recognition system according to claim 9, wherein the articulatory features include stress.

11. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes possess discrete values.

12. A speech recognition system according to claim 11, wherein the discrete values are binary.

13. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes possess continuous values.

14. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include syllabic structure.

15. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include syntactic structure.

16. A speech recognition system according to claim 1, wherein the suprasegmental non-phonetic attributes include semantic structure.

17. A speech recognition system according to claim 1, wherein the decoder further comprises:

A probabilistic pronunciation model that characterizes possible pronunciation effects,

Wherein an acoustic model induction algorithm augments the acoustic models with the pronunciation model.

18. A speech recognition system according to claim 1, wherein the decoder is a single-prefix-tree decoder.

19. A speech recognition system according to claim 18, wherein the prefix tree includes nodes and leaves, and for each node and leaf a priority heap represents alternative linguistic theories.

20. A speech recognition system according to claim 19, wherein the heap maintains all contexts within a selected threshold probability.

21. A speech recognition system according to claim 19, wherein lookahead cost trees are used to determine for every node a best language model score for all words with a given prefix.

22. A speech recognition system according to claim 21, wherein lookahead trees are saved in an aging cache to avoid recomputing for subsequent frames.

23. A speech recognition system according to claim 21, wherein each heap element has a current lookahead score attached.

24. A speech recognition system according to claim 19, wherein the decoder further comprises:

A multiplexer to represent fan-in of root nodes and fan-out of leaf nodes.

25. A speech recognition system according to claim 24, wherein the prefix tree includes root nodes to represent a first attribute instance of words for a given multiplexer.

26. A speech recognition system according to claim 24, wherein the prefix tree includes word internal nodes wherein the decoder collapses alternative hypotheses only if the alternative hypotheses are in the same finite machine state.

27. A speech recognition system according to claim 24, wherein each word has a leaf node, and the multiplexer describes the fan-out such that each heap element represents a complete fan-out for a given grammar state.

28. A speech recognition system according to claim 24, wherein a single phone word is represented by a multiplexer of multiplexers that, depending on a left context, returns a multiplexer representing a right context-dependent fan-out of the single phone word.

29. A speech recognition system according td claim 1, wherein the decoder is time synchronous.

30. A speech recognition system according to claim 1, wherein the decoder uses decision trees to induce a set of context-dependent duration and acoustic models.

31. A speech recognition system according to claim 1, wherein the language model uses n-gram models in a finite state machine wrapper.

32. A speech recognition system according to claim 1, wherein the decoder uses finite state recognition lattices that enable rescoring a word graph with alternative word models or language models.

33. A speech recognition system according to claim 1, wherein the decoder uses a bucket box intersection technique.

34. A speech recognition system according to claim 1, wherein the decoder uses a dynamic frame skipping technique.

35. An attribute-based method of speech recognition comprising:

Pre-processing input speech to produce a sequence of acoustic observations representative of the input speech;

Characterizing, with context-dependent acoustic models, a probability of a given sequence of sounds producing the sequence of acoustic observations, each acoustic model including phonetic attributes and suprasegmental non-phonetic attributes;

Characterizing, with a finite state language model, a probability of a given sequence of words being spoken; and

Comparing, with a one-pass decoder, the sequence of acoustic observations to the acoustic models and the language model, and outputs at least one word sequence representative of the input speech.

36. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include speaking rate.

37. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include phone durations.

38. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include dialect.

39. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include gender.

40. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include fundamental frequencies.

41. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include hidden Markov model state.

42. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include word class.

43. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include articulatory features.

44. A method according to claim 43, wherein the articulatory features include stress.

45. A method according to claim 35, wherein the suprasegmental non-phonetic attributes possess discrete values.

46. A method according to claim 45, wherein the discrete values are binary.

47. A method according to claim 35, wherein the suprasegmental non-phonetic attributes possess continuous values.

48. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include syllabic structure.

49. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include syntactic structure.

50. A method according to claim 35, wherein the suprasegmental non-phonetic attributes include semantic structure.

51. A method according to claim 35, wherein the comparing further comprises:

Characterizing, with a probabilistic pronunciation model, possible pronunciation effects, and

Augmenting the acoustic models with the pronunciation model using an acoustic model induction algorithm.

52. A method according to claim 35, wherein the comparing uses a single-prefix-tree decoder.

53. A method according to claim 52, wherein the prefix tree includes nodes and leaves, and for each node and leaf a priority heap represents alternative linguistic theories.

54. A method according to claim 53, wherein the priority heap maintains all contexts within a selected threshold probability.

55. A speech recognition system according to claim 53, wherein lookahead cost trees are used to determine for every node a best language model score for all words with a given prefix.

56. A method according to claim 55, wherein lookahead cost trees are saved in an aging cache to avoid recomputing for subsequent frames.

57. A speech recognition system according to claim 55, wherein each heap element has a current lookahead score attached.

58. A method according to claim 53, wherein the comparing further includes representing, with a multiplexer, fan-in of root nodes and fan-out of leaf nodes.

59. A method according to claim 58, wherein the prefix tree includes root nodes to represent a first attribute instance of words for a given multiplexer.

60. A method according to claim 58, wherein the prefix tree includes word internal nodes in which the decoder collapses alternative hypotheses only if the alternative hypotheses are in the same finite machine state.

61. A method according to claim 58, wherein each word has a leaf node and the multiplexer describes the fan-out such that each heap element represents a complete fan-out for a given grammar state.

62. A method according to claim 58, wherein a single phone word is represented by a multiplexer of multiplexers that, depending on a left context, returns a multiplexer representing a right context-dependent fan-out of the single phone word.

63. A method according to claim 35, wherein the comparing includes using a time synchronous decoder.

64. A method according to claim 35, wherein the comparing includes using decision trees to induce a set of context-dependent duration and acoustic models.

65. A method according to claim 35, wherein the language model uses n-gram models in a finite state machine wrapper.

66. A method according to claim 35, wherein the comparing includes using finite state recognition lattices that enable rescoring a word graph with alternative word models or language models.

67. A method according to claim 35, wherein the comparing includes using a bucket box intersection technique.

68. A method according to claim 35, wherein the comparing includes using a dynamic frame skipping technique.