Content-Based Audio Playback Emphasis

• US 20100318347A1
• Filed: 08/20/2010
• Published: 12/16/2010
• Est. Priority Date: 07/22/2005
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: Multimodal Technologies Incorporated (3M Company)
• Inventors: Koll, Detlef, Schubert, Kjell, Finke, Michael, Fritsch, Juergen
• Granted Patent: US 8,768,706 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/04 Time compression or expansion

Abstract

Techniques are disclosed for facilitating the process of proofreading draft transcripts of spoken audio streams. In general, proofreading of a draft transcript is facilitated by playing back the corresponding spoken audio stream with an emphasis on those regions in the audio stream that are highly relevant or likely to have been transcribed incorrectly. Regions may be emphasized by, for example, playing them back more slowly than regions that are of low relevance and likely to have been transcribed correctly. Emphasizing those regions of the audio stream that are most important to transcribe correctly and those regions that are most likely to have been transcribed incorrectly increases the likelihood that the proofreader will accurately correct any errors in those regions, thereby improving the overall accuracy of the transcript.

Claims

1. A method performed by a computer processor executing computer program instructions tangibly stored on a first computer-readable medium to perform a method comprising:

(A) deriving, from a region of a document and a corresponding region of a spoken audio stream, a likelihood that the region of the document correctly represents content in the corresponding region of the spoken audio stream, and tangibly storing a representation of the likelihood in a second computer-readable medium;

(B) selecting a measure of relevance of the region of the spoken audio stream, the measure of relevance representing a measure of importance that the region of the spoken audio stream be brought to the attention of a human proofreader, and tangibly storing a representation of the measure of relevance in a third computer-readable medium; and

(C) deriving, from the stored representation of the likelihood and the stored representation of the measure of relevance, an emphasis factor for modifying emphasis placed on the region of the spoken audio stream when played back, and storing a representation of the emphasis factor in a fourth computer-readable medium.

2. The method of claim 1, wherein (C) comprises deriving an emphasis factor for increasing the emphasis placed on the region of the spoken audio stream when played back.

3. The method of claim 1, wherein (C) comprises deriving an emphasis factor for decreasing the emphasis placed on the region of the spoken audio stream when played back.

4. The method of claim 1, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a timescale adjustment factor for adjusting a playback rate of the region of the spoken audio stream.

5. The method of claim 1, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a signal power adjustment factor for adjusting a signal power of the region of the spoken audio stream.

6. The method of claim 1, further comprising:

(D) modifying an emphasis of the region of the spoken audio stream in accordance with the emphasis factor to produce an emphasis-adjusted audio stream.

7. The method of claim 6, further comprising:

(E) modifying an emphasis of the region of the document in accordance with the emphasis factor to produce an emphasis-adjusted document region.

8. The method of claim 6, further comprising:

(E) using a text-to-speech engine to play the emphasis-adjusted audio stream.

9. The method of claim 8, further comprising:

(F) correcting errors in the document based on the emphasis-adjusted audio stream.

10. The method of claim 6, further comprising:

(E) modifying an emphasis of a region adjacent to the region of the spoken audio stream by an amount less than that specified by the emphasis factor.

11. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the likelihood from a prior likelihood of correctness of the region of the document.

12. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the likelihood from a feature of the spoken audio stream.

13. The method of claim 12, wherein the feature comprises an identity of a speaker of the spoken audio stream.

14. The method of claim 12, wherein the feature comprises a signal-to-noise ratio of the spoken audio stream.

15. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the likelihood from a confidence measure representing a degree of confidence that the region of the document correctly represents the content in the corresponding region of the spoken audio stream, wherein the confidence measure is provided by an automatic transcription system that produced the region of the document based on the region of the spoken audio stream.

16. The method of claim 15, wherein (A)(1) comprises deriving the estimate of the likelihood from the confidence measure, a prior likelihood of correctness of the region of the document, and a feature of the spoken audio stream.

17. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the estimate of the likelihood from a prior likelihood of correctness of the region of the document.

18. The method of claim 1, wherein (B) comprises:

(B)(1) selecting a prior relevance of the region of the document; and

(B)(2) selecting the measure of relevance of the region of the spoken audio stream based on the identified prior relevance of the region of the document.

19. The method of claim 18, wherein (B)(1) comprises selecting the prior relevance of the region of the document as a relatively high prior relevance if the region of the document contains content in a predetermined set of highly-relevant content.

20. The method of claim 1, wherein (B) comprises selecting the measure of relevance of the region of the spoken audio stream as a relatively low relevance if the region of the spoken audio stream contains no speech.

21. The method of claim 1, wherein the region of the document comprises a hypothesis generated by an automatic transcription system for the corresponding region of the spoken audio stream, and wherein (B) comprises:

(B)(1) identifying a competing hypothesis generated by the automatic transcription system for the corresponding region of the spoken audio stream;

(B)(2) selecting a prior relevance of the competing hypothesis; and

(B)(3) selecting the measure of relevance based on the prior relevance of the competing hypothesis.

22. The method of claim 21, wherein (B)(3) comprises selecting the measure of relevance based on the prior relevance of the competing hypothesis and a prior relevance of the region of the document.

23. The method of claim 1, wherein (C) comprises:

(C)(1) identifying a rule for identifying the emphasis factor based on the identified likelihood and the identified measure of relevance; and

(C)(2) applying the rule to the identified likelihood and the identified measure of relevance to derive the emphasis factor.

24. The method of claim 23, wherein (C)(2) comprises:

(C)(2)(a) identifying a first weight associated with the identified likelihood;

(C)(2)(b) identifying a second weight associated with the measure of relevance; and

(C)(2)(c) deriving the emphasis factor from a combination of the identified likelihood and the measure of relevance weighted by the first and second weights, respectively.

25. The method of claim 1, further comprising:

(A′

) prior to (A), generating the document based on the spoken audio stream.

26. The method of claim 25, wherein (C) comprises using an automated transcription system to generate the document based on the spoken audio stream.

27. An apparatus comprising a computer-readable medium tangibly storing instructions executable by a computer processor to perform a method comprising:

(A) deriving, from a region of a document and a corresponding region of a spoken audio stream, a likelihood that the region of the document correctly represents content in the corresponding region of the spoken audio stream;

(B) selecting a measure of relevance of the region of the spoken audio stream, the measure of relevance representing a measure of importance that the region of the spoken audio stream be brought to the attention of a human proofreader; and

(C) deriving, from the likelihood and the measure of relevance, an emphasis factor for modifying emphasis placed on the region of the spoken audio stream when played back.

28. The apparatus of claim 27, wherein (C) comprises deriving an emphasis factor for increasing the emphasis placed on the region of the spoken audio stream when played back.

29. The method of claim 27, wherein (C) comprises deriving an emphasis factor for decreasing the emphasis placed on the region of the spoken audio stream when played back.

30. The apparatus of claim 27, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a timescale adjustment factor for adjusting a playback rate of the audio stream.

31. The apparatus of claim 27, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a signal power adjustment factor for adjusting a signal power of the region of the spoken audio stream.

32. The apparatus of claim 27, wherein the method further comprises:

(D) modifying an emphasis of the region of the spoken audio stream in accordance with the emphasis factor to produce an emphasis-adjusted audio stream.

33. The apparatus of claim 32, wherein the method further comprises:

(E) modifying an emphasis of the region of the document in accordance with the emphasis factor to produce an emphasis-adjusted document region.

34. The apparatus of claim 32, further comprising:

a text-to-speech engine to play the emphasis-adjusted audio stream.

35. The apparatus of claim 34, wherein the method further comprises:

(E) correcting errors in the document based on the emphasis-adjusted audio stream.

36. The apparatus of claim 32, wherein the method further comprises:

(E) modifying an emphasis of a region adjacent to the region of the spoken audio stream to a lesser extent than specified by the emphasis factor.

37. The apparatus of claim 32, wherein (A) comprises:

(A)(1) deriving the likelihood from a prior likelihood of correctness of the region of the document.

38. The apparatus of claim 27, wherein (A) comprises:

(A)(1) deriving the likelihood from a feature of the spoken audio stream.

39. The apparatus of claim 27, wherein (B) comprises:

(B)(1) identifying a prior relevance of the region of the document; and

(B)(2) selecting the measure of relevance of the region of the spoken audio stream based on the identified prior relevance of the region of the document.

40. The apparatus of claim 27, wherein (C) comprises:

(C)(1) identifying a rule for identifying the emphasis factor based on the identified likelihood and the identified measure of relevance; and

(C)(2) applying the rule to the identified likelihood and the identified measure of relevance to identify the emphasis factor.

41. The apparatus of claim 40, wherein (C)(2) comprises:

(C)(2)(a) identifying a first weight associated with the identified likelihood;

(C)(2)(b) identifying a second weight associated with the measure of relevance; and

(C)(2)(c) deriving the emphasis factor from a combination of the identified likelihood and the measure of relevance weighted by the first and second weights, respectively.

42. The apparatus of claim 27, wherein the method further comprises:

(D) generating the document based on the spoken audio stream.