Content-based audio playback emphasis

• US 8,768,706 B2
• Filed: 08/20/2010
• Issued: 07/01/2014
• Est. Priority Date: 07/22/2005
• Status: Expired due to Fees

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: Multimodal Technologies Incorporated (3M Company)
• Inventors: Schubert, Kjell, Fritsch, Juergen, Finke, Michael, Koll, Detlef
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US12/859,883
• Publication Number:
  US 20100318347A1
• Time in Patent Office: 1,411 Days
• Field of Search: 704/270.1, 704/270, 704/275, 704/236, 704/235, 704/9, 704/249, 704/251, 704/256, 704/254
• US Class Current: 704/270
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/04 Time compression or expansion

Abstract

Techniques are disclosed for facilitating the process of proofreading draft transcripts of spoken audio streams. In general, proofreading of a draft transcript is facilitated by playing back the corresponding spoken audio stream with an emphasis on those regions in the audio stream that are highly relevant or likely to have been transcribed incorrectly. Regions may be emphasized by, for example, playing them back more slowly than regions that are of low relevance and likely to have been transcribed correctly. Emphasizing those regions of the audio stream that are most important to transcribe correctly and those regions that are most likely to have been transcribed incorrectly increases the likelihood that the proofreader will accurately correct any errors in those regions, thereby improving the overall accuracy of the transcript.

Claims

1. A method performed by a computer processor executing computer program instructions tangibly stored on a first computer-readable medium to perform a method comprising:

(A) deriving, from a region of a document and a corresponding region of a spoken audio stream, a likelihood score representing a likelihood that the region of the document correctly represents content in the corresponding region of the spoken audio stream, and tangibly storing a representation of the likelihood score in a second computer-readable medium;

(B) selecting a relevance score representing a measure of relevance of the region of the spoken audio stream, the measure of relevance representing a measure of importance that the region of the spoken audio stream be brought to the attention of a human proofreader, and tangibly storing a representation of the relevance score in a third computer-readable medium; and

(C) deriving, by dividing the relevance score by the likelihood score, an emphasis factor for modifying emphasis placed on the region of the spoken audio stream when played back, and storing a representation of the emphasis factor in a fourth computer-readable medium.

2. The method of claim 1, wherein (C) comprises deriving an emphasis factor for increasing the emphasis placed on the region of the spoken audio stream when played back.

3. The method of claim 1, wherein (C) comprises deriving an emphasis factor for decreasing the emphasis placed on the region of the spoken audio stream when played back.

4. The method of claim 1, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a timescale adjustment factor for adjusting a playback rate of the region of the spoken audio stream.

5. The method of claim 1, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a signal power adjustment factor for adjusting a signal power of the region of the spoken audio stream.

6. The method of claim 1, further comprising:

(D) modifying an emphasis of the region of the spoken audio stream in accordance with the emphasis factor to produce an emphasis-adjusted audio stream.

7. The method of claim 6, further comprising:

(E) modifying an emphasis of the region of the document in accordance with the emphasis factor to produce an emphasis-adjusted document region.

8. The method of claim 6, further comprising:

(E) playing the emphasis-adjusted audio stream.

9. The method of claim 8, further comprising:

(F) correcting errors in the document based on the emphasis-adjusted audio stream.

10. The method of claim 6, further comprising:

(E) modifying an emphasis of a region adjacent to the region of the spoken audio stream by an amount less than that specified by the emphasis factor.

11. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the likelihood from a prior likelihood of correctness of the region of the document.

12. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the likelihood from a feature of the spoken audio stream.

13. The method of claim 12, wherein the feature comprises an identity of a speaker of the spoken audio stream.

14. The method of claim 12, wherein the feature comprises a signal-to-noise ratio of the spoken audio stream.

15. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the likelihood from a confidence measure representing a degree of confidence that the region of the document correctly represents the content in the corresponding region of the spoken audio stream, wherein the confidence measure is provided by an automatic transcription system that produced the region of the document based on the region of the spoken audio stream.

16. The method of claim 15, wherein (A)(1) comprises deriving the likelihood from the confidence measure, a prior likelihood of correctness of the region of the document, and a feature of the spoken audio stream.

17. The method of claim 1, wherein (A) comprises:

(A)(1) deriving the likelihood from a prior likelihood of correctness of the region of the document.

18. The method of claim 1, wherein (B) comprises:

(B)(1) selecting a prior relevance of the region of the document; and

(B)(2) selecting the measure of relevance of the region of the spoken audio stream based on the identified prior relevance of the region of the document.

19. The method of claim 18, wherein (B)(1) comprises selecting the prior relevance of the region of the document as a relatively high prior relevance if the region of the document contains content in a predetermined set of highly-relevant content.

20. The method of claim 1, wherein (B) comprises selecting the measure of relevance of the region of the spoken audio stream as a relatively low relevance if the region of the spoken audio stream contains no speech.

21. The method of claim 1, wherein the region of the document comprises a hypothesis generated by an automatic transcription system for the corresponding region of the spoken audio stream, and wherein (B) comprises:

(B)(1) identifying a competing hypothesis generated by the automatic transcription system for the corresponding region of the spoken audio stream;

(B)(2) selecting a prior relevance of the competing hypothesis; and

(B)(3) selecting the measure of relevance based on the prior relevance of the competing hypothesis.

22. The method of claim 21, wherein (B)(3) comprises selecting the measure of relevance based on the prior relevance of the competing hypothesis and a prior relevance of the region of the document.

23. The method of claim 1, wherein (C) comprises:

(C)(1) identifying a rule for identifying the emphasis factor based on the identified likelihood and the identified measure of relevance; and

(C)(2) applying the rule to the identified likelihood and the identified measure of relevance to derive the emphasis factor.

24. The method of claim 1, further comprising:

(A′

) prior to (A), generating the document based on the spoken audio stream.

25. The method of claim 24, wherein (A′

) comprises using an automated transcription system to generate the document based on the spoken audio stream.

26. The method of 1, further comprising:

(D) before (C), identifying a default playback rate of the spoken audio stream; and

(E) identifying an emphasized playback rate of the spoken audio stream as a quotient of the default playback rate and the emphasis factor.

27. The method of claim 26, further comprising:

(F) playing the spoken audio stream at the emphasized playback rate.

28. An apparatus comprising a computer-readable medium tangibly storing instructions executable by a computer processor to perform a method comprising:

(A) deriving, from a region of a document and a corresponding region of a spoken audio stream, a likelihood score representing a likelihood that the region of the document correctly represents content in the corresponding region of the spoken audio stream;

(B) selecting a relevance score representing a measure of relevance of the region of the spoken audio stream, the measure of relevance representing a measure of importance that the region of the spoken audio stream be brought to the attention of a human proofreader; and

(C) deriving, by dividing the relevance score by the likelihood score, an emphasis factor for modifying emphasis placed on the region of the spoken audio stream when played back.

29. The apparatus of claim 28, wherein (C) comprises deriving an emphasis factor for increasing the emphasis placed on the region of the spoken audio stream when played back.

30. The apparatus of claim 28, wherein (C) comprises deriving an emphasis factor for decreasing the emphasis placed on the region of the spoken audio stream when played back.

31. The apparatus of claim 28, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a timescale adjustment factor for adjusting a playback rate of the audio stream.

32. The apparatus of claim 28, wherein (C) comprises deriving, from the likelihood and the measure of relevance, a signal power adjustment factor for adjusting a signal power of the region of the spoken audio stream.

33. The apparatus of claim 28, wherein the method further comprises:

(D) modifying an emphasis of the region of the spoken audio stream in accordance with the emphasis factor to produce an emphasis-adjusted audio stream.

34. The apparatus of claim 33, wherein the method further comprises:

(E) modifying an emphasis of the region of the document in accordance with the emphasis factor to produce an emphasis-adjusted document region.

35. The apparatus of claim 33, further comprising:

an audio playback engine play the emphasis-adjusted audio stream.

36. The apparatus of claim 35, wherein the method further comprises:

(E) correcting errors in the document based on the emphasis-adjusted audio stream.

37. The apparatus of claim 33, wherein the method further comprises:

(E) modifying an emphasis of a region adjacent to the region of the spoken audio stream to a lesser extent than specified by the emphasis factor.

38. The apparatus of claim 33, wherein (A) comprises:

(A)(1) deriving the likelihood from a prior likelihood of correctness of the region of the document.

39. The apparatus of claim 28, wherein (A) comprises:

(A)(1) deriving the likelihood from a feature of the spoken audio stream.

40. The apparatus of claim 28, wherein (B) comprises:

(B)(1) identifying a prior relevance of the region of the document; and

(B)(2) selecting the measure of relevance of the region of the spoken audio stream based on the identified prior relevance of the region of the document.

41. The apparatus of claim 28, wherein (C) comprises:

(C)(1) identifying a rule for identifying the emphasis factor based on the identified likelihood and the identified measure of relevance; and

(C)(2) applying the rule to the identified likelihood and the identified measure of relevance to identify the emphasis factor.

42. The apparatus of claim 28, wherein the method further comprises:

(D) generating the document based on the spoken audio stream.

43. The apparatus of 28, wherein the method further comprises:

(D) before (C), identifying a default playback rate of the spoken audio stream; and

(E) identifying an emphasized playback rate of the spoken audio stream as a quotient of the default playback rate and the emphasis factor.

44. The apparatus of claim 43, wherein the method further comprises:

(E) playing the spoken audio stream at the emphasized playback rate.