Method and apparatus for correcting and repairing machine-transcribed input using independent or cross-modal secondary input

• US 5,855,000 A
• Filed: 10/01/1996
• Issued: 12/29/1998
• Est. Priority Date: 09/08/1995
• Status: Expired due to Term

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Waibel, Alexander, Suhm, Bernhard, McNair, Arthur
• Primary Examiner(s): Knepper, David D.
• Assistant Examiner(s): Smits, Talivaldis Ivars
• Application Number:
  US08/724,420
• Time in Patent Office: 819 Days
• Field of Search: 704/239, 704/240, 704/252, 704/257, 704/235, 704/236, 704/255, 382/186
• US Class Current: 704/235
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

The method of the present invention is comprised of the steps of locating an error within a recognition hypothesis. A repair hypothesis is generated for the located error from a secondary input signal. At least a portion of the recognition hypothesis is correlated with the repair hypothesis to produce a new hypothesis for the located error. The located error is replaced with the new hypothesis. It has been determined that when the primary and secondary input signals are cross-modal, the chances that the error will be repeated are so low that the correlation step becomes optional.

Claims

1. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal including an utterance, comprising:

locating an error within the recognition hypothesis;

generating a repair hypothesis for said located error from a secondary input signal including a paraphrasing of at least a portion of the utterance;

correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis; and

replacing said located error with the new hypothesis.

2. The method of claim 1 additionally comprising repeating the method in the event that the new hypothesis is incorrect.

3. The method of claim 1 wherein locating an error includes highlighting the error by using a pointing device.

4. The method of claim 1 wherein locating an error includes highlighting the error by using a keyboard.

5. The method of claim 1 wherein locating an error includes speaking the portion that is in error.

6. The method of claim 1 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

7. The method of claim 1 wherein correlating at least a portion of the recognition hypotheses includes correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

8. The method of claim 1 wherein correlating at least a portion of the recognition hypothesis includes correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

9. The method of claim 1 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

10. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal including a writing, comprising:

locating an error within the recognition hypothesis;

generating a repair hypothesis for said located error from a secondary input signal including a paraphrasing of at least a portion of the writing;

correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis; and

replacing said located error with the new hypothesis.

11. The method of claim 10 additionally comprising repeating the method in the event that the new hypothesis is incorrect.

12. The method of claim 10 wherein locating an error includes highlighting the error by using a pointing device.

13. The method of claim 10 wherein locating an error includes highlighting the error by using a keyboard.

14. The method of claim 10 wherein locating an error includes speaking the portion that is in error.

15. The method of claim 10 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

16. The method of claim 10 wherein correlating at least a portion of the recognition hypothesis includes correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

17. The method of claim 10 wherein correlating at least a portion of the recognition hypothesis includes correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

18. The method of claim 10 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

19. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal including a verbal spelling, comprising:

locating an error within the recognition hypothesis;

generating a repair hypothesis for said located error from a secondary input signal including a paraphrasing of at least a portion of the verbal spelling;

correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis; and

replacing said located error with the new hypothesis.

20. The method of claim 19 additionally comprising repeating the method in the event that the new hypothesis is incorrect.

21. The method of claim 19 wherein locating an error includes highlighting the error by using a pointing device.

22. The method of claim 19 wherein locating an error includes highlighting the error by using a keyboard.

23. The method of claim 19 wherein locating an error includes speaking the portion that is in error.

24. The method of claim 19 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

25. The method of claim 19 wherein correlating at least a portion of the recognition hypothesis includes correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

26. The method of claim 19 wherein correlating at least a portion of the recognition hypothesis includes correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

27. The method of claim 19 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

28. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal, comprising:

locating an error within the recognition hypothesis, wherein locating an error includes speaking the portion that is in error;

generating a repair hypothesis for said located error from a secondary input signal;

correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis; and

replacing said located error with the new hypothesis.

29. The method of claim 28 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a respeaking of at least a portion of the utterance.

30. The method of claim 28 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a verbal spelling of at least a portion of the utterance.

31. The method of claim 28 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a non-verbal spelling of at least a portion of the utterance.

32. The method of claim 28 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a writing of at least a portion of the utterance.

33. The method of claim 28 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a paraphrasing of at least a portion of the utterance.

34. The method of claim 28 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a speaking of at least a portion of the writing.

35. The method of claim 28 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a verbal spelling of at least a portion of the writing.

36. The method of claim 28 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a nonverbal spelling of at least a portion of the writing.

37. The method of claim 28 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a rewriting of at least a portion of the writing.

38. The method of claim 28 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a paraphrasing of at least a portion of the writing.

39. The method of claim 28 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a speaking of at least a portion of the verbal spelling.

40. The method of claim 28 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a verbal respelling of at least a portion of the verbal spelling.

41. The method of claim 28 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a non-verbal spelling of at least a portion of the verbal spelling.

42. The method of claim 28 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a writing of at least a portion of the verbal spelling.

43. The method of claim 28 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a paraphrasing of at least a portion of the verbal spelling.

44. The method of claim 28 additionally comprising repeating the method in the event that the new hypothesis is incorrect.

45. The method of claim 28 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

46. The method of claim 28 wherein correlating at least a portion of the recognition hypothesis includes correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

47. The method of claim 28 wherein correlating at least a portion of the recognition hypothesis includes correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

48. The method of claim 28 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

49. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal including a utterance comprising:

locating an error within the recognition hypothesis;

generating a repair hypothesis for said located error from a secondary input signal of different modality than the primary input signal, said secondary input signal including a paraphrasing of at least a portion of the utterance; and

replacing said located error with the repair hypothesis.

50. The method of claim 49 additionally comprising repeating the method in the event that the new hypothesis is incorrect.

51. The method of claim 49 wherein locating an error includes highlighting the error by using a pointing device.

52. The method of claim 49 wherein locating an error includes highlighting the error by using a keyboard.

53. The method of claim 49 wherein locating an error includes speaking the portion that is in error.

54. The method of claim 49 additionally comprising correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis, and wherein said new hypothesis is used to replace said located error instead of the repair hypothesis.

55. The method of claim 54 wherein correlating at least a portion of the recognition hypothesis includes correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

56. The method of claim 54 wherein correlating at least a portion of the recognition hypothesis includes correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

57. The method of claim 54 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

58. The method of claim 49 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

59. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal including a writing, comprising:

locating an error within the recognition hypothesis;

generating a repair hypothesis for said located error from a secondary input signal of a different modality than the primary input signal, said secondary input signal including paraphrasing of at least a portion of the writing; and

replacing said located error with the repair hypothesis.

60. The method of claim 59 additionally comprising repeating the method in the event that the new hypothesis is incorrect.

61. The method of claim 59 wherein locating an error includes highlighting the error by using a pointing device.

62. The method of claim 59 wherein locating an error includes highlighting the error by using a keyboard.

63. The method of claim 59 wherein locating an error includes speaking the portion that is in error.

64. The method of claim 59 additionally comprising correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis, and wherein said new hypothesis is used to replace said located error instead of the repair hypothesis.

65. The method of claim 64 wherein correlating at least a portion of the recognition hypothesis includes correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

66. The method of claim 64 wherein correlating at least a portion of the recognition hypothesis includes correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

67. The method of claim 64 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

68. The method of claim 59 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

69. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal including a verbal spelling, comprising:

locating an error within the recognition hypothesis;

generating a repair hypothesis for said located error from a secondary input signal of a different modality than the primary input signal, said secondary input signal including a paraphrasing of at least a portion of the verbal spelling; and

replacing said located error with the repair hypothesis.

70. The method of claim 69 additionally comprising repeating the method in the event that the new hypothesis is incorrect.

71. The method of claim 69 wherein locating an error includes highlighting the error by using a pointing device.

72. The method of claim 69 wherein locating an error includes highlighting the error by using a keyboard.

73. The method of claim 69 wherein locating an error includes speaking the portion that is in error.

74. The method of claim 69 additionally comprising correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis, and wherein said new hypothesis is used to replace said located error instead of the repair hypothesis.

75. The method of claim 74 wherein correlating at least a portion of the recognition hypothesis includes correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

76. The method of claim 74 wherein correlating at least a portion of the recognition hypothesis includes correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

77. The method of claim 74 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

78. The method of claim 69 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

79. A method of repairing the output from a recognition engine generating at least one recognition hypothesis from a primary input signal, comprising:

locating an error within the recognition hypothesis, wherein locating an error includes speaking the portion that is in error;

generating a repair hypothesis for said located error from a secondary input signal of a different modality than the primary input signal; and

replacing said located error with the repair hypothesis.

80. The method of claim 79 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a verbal spelling of at least a portion of the utterance.

81. The method of claim 79 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a non-verbal spelling of at least a portion of the utterance.

82. The method of claim 79 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a writing of at least a portion of the utterance.

83. The method of claim 79 wherein the primary input signal includes an utterance and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a paraphrasing of at least a portion of the utterance.

84. The method of claim 79 wherein the primary input signal includes a writing and wherein said step of generating a repair hypothesis from a secondary input signal includes the step of generating a repair hypothesis from a speaking of at least a portion of the writing.

85. The method of claim 79 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a verbal spelling of at least a portion of the writing.

86. The method of claim 79 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a nonverbal spelling of at least a portion of the writing.

87. The method of claim 79 wherein the primary input signal includes a writing and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a paraphrasing of at least a portion of the writing.

88. The method of claim 79 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a speaking of at least a portion of the verbal spelling.

89. The method of claim 79 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a non-verbal spelling of at least a portion of the verbal spelling.

90. The method of claim 79 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a writing of at least a portion of the verbal spelling.

91. The method of claim 79 wherein the primary input signal includes a verbal spelling and wherein generating a repair hypothesis from a secondary input signal includes generating a repair hypothesis from a paraphrasing of at least a portion of the verbal spelling.

92. The method of claim 79 additionally comprising repeating the method in the event that the repair hypothesis is incorrect.

93. The method of claim 79 additionally comprising correlating at least a portion of the recognition hypothesis with said repair hypothesis to produce a new hypothesis, and wherein said new hypothesis is used to replace said located error instead of the repair hypothesis.

94. The method of claim 93 wherein correlating at least a portion of the recognition hypothesis includes the step of correlating said repair hypothesis with the context of at least a portion of said recognition hypothesis.

95. The method of claim 93 wherein correlating at least a portion of the recognition hypothesis includes the step of correlating an n-best list of said repair hypothesis with an n-best list of at least a portion of said recognition hypothesis.

96. The method of claim 93 wherein correlating at least a portion of the recognition hypothesis includes correlating a lattice of said repair hypothesis with a lattice of at least a portion of said recognition hypothesis.

97. The method of claim 79 additionally comprising constraining the generation of said repair hypothesis to exclude known incorrect hypotheses.

98. An apparatus, comprising:

a recognition engine for generating at least one recognition hypothesis in response to a primary input signal including an utterance;

a location module for locating an error within said recognition hypothesis, said recognition engine generating a repair hypothesis for said located error in response to a secondary input signal including a paraphrasing of at least a portion of the utterance;

a replacement module for replacing said located error with said new hypothesis.

99. The apparatus of claim 98 additionally comprising a second recognition engine responsive to said secondary input signal for generating a repair hypothesis in place of the repair hypothesis being generated by said recognition engine.

100. An apparatus, comprising:

a recognition engine for generating at least one recognition hypothesis in response to a primary input signal including a writing;

a location module for locating an error within said recognition hypothesis, said recognition engine generating a repair hypothesis for said located error in response to a secondary input signal including a paraphrasing of at least a portion of the writing;

a correlation module for correlating at least a portion of said recognition hypothesis with said repair hypothesis to produce a new hypothesis; and

a replacement module for replacing said located error with said new hypothesis.

101. The apparatus of claim 100 additionally comprising a second recognition engine responsive to said secondary input signal for generating a repair hypothesis in place of the repair hypothesis being generated by said recognition engine.

102. An apparatus, comprising:

a recognition engine for generating at least one recognition hypothesis in response to a primary input signal including verbal spelling;

a location module for locating an error within said recognition hypothesis, said recognition engine generating a repair hypothesis for said located error in response to a secondary input signal including a paraphrasing of at least a portion of the verbal spelling;

a correlation module for correlating at least a portion of said recognition hypothesis with said repair hypothesis to produce a new hypothesis; and

a replacement module for replacing said located error with said new hypothesis.

103. The apparatus of claim 102 additionally comprising a second recognition engine responsive to said secondary input signal for generating a repair hypothesis in place of the repair hypothesis being generated by said recognition engine.

104. An apparatus, comprising:

a first recognition engine for generating at least one recognition hypothesis from a primary input signal including an utterance;

a location module for locating an error within said recognition hypothesis;

a second recognition engine for generating a repair hypothesis for said located error from a secondary input signal of a different modality than said primary input signal, said secondary input signal including a paraphrasing of at least a portion of the utterance; and

a replacement module for replacing said located error with said repair hypothesis.

105. The apparatus of claim 104 additionally comprising a correlation module for correlating at least a portion of said recognition hypothesis with said repair hypothesis to produce a new hypothesis.

106. An apparatus, comprising:

a first recognition engine for generating at least one recognition hypothesis from a primary input signal including a writing;

a location module for locating an error within said recognition hypothesis;

a second recognition engine for generating a repair hypothesis for said located error from a secondary input signal of a different modality than said primary input signal, said secondary input signal including a paraphrasing of at least a portion of the writing; and

a replacement module for replacing said located error with said repair hypothesis.

107. The apparatus of claim 106 additionally comprising a correlation module for correlating at least a portion of said recognition hypothesis with said repair hypothesis to produce a new hypothesis.

108. An apparatus, comprising:

a first recognition engine for generating at least one recognition hypothesis from a primary input signal including a verbal spelling;

a location module for locating an error within said recognition hypothesis;

a second recognition engine for generating a repair hypothesis for said located error from a secondary input signal of a different modality than said primary input signal, said secondary input signal including a paraphrasing of at least a portion of the verbal spelling; and

a replacement module for replacing said located error with said repair hypothesis.

109. The apparatus of claim 108 additionally comprising a correlation module for correlating at least a portion of said recognition hypothesis with said repair hypothesis to produce a new hypothesis.

110. An apparatus, comprising:

a recognition engine for generating at least one recognition hypothesis in response to a primary input signal;

a location module for locating an error within said recognition hypothesis in response to a speaking of the portion that is in error, said recognition engine generating a repair hypothesis for said located error in response to a secondary input signal;

a correlation module for correlating at least a portion of said recognition hypothesis with said repair hypothesis to produce a new hypothesis; and

a replacement module for replacing said located error with said new hypothesis.

111. The apparatus of claim 110 additionally comprising a second recognition engine responsive to said secondary input signal for generating a repair hypothesis in place of the repair hypothesis being generated by said recognition engine.

112. An apparatus, comprising:

a first recognition engine for generating at least one recognition hypothesis from a primary input signal;

a location module for locating an error within said recognition hypothesis in response to a speaking of the portion that is in error;

a second recognition engine for generating a repair hypothesis for said located error from a secondary input signal of a different modality than said primary input signal; and

a replacement module for replacing said located error with said repair hypothesis.

113. The apparatus of claim 112 additionally comprising a correlation module for correlating at least a portion of said recognition hypothesis with said repair hypothesis to produce a new hypothesis.