Mass-Scale, User-Independent, Device-Independent Voice Messaging System

• US 20070127688A1
• Filed: 02/12/2007
• Published: 06/07/2007
• Est. Priority Date: 02/10/2006
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: SpinVox Limited (Microsoft Corporation)
• Inventors: Doulton, Daniel
• Granted Patent: US 8,903,053 B2
• Time in Patent Office: Days
• US Class Current: 379/265.10
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04M 2201/60 Medium conversion
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/5183 Call or contact centers with computer-telephony arrangements
• CPC Class Codes: H04M 3/53333 Message receiving aspects

Abstract

A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen is disclosed. The system comprises (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control; the system being adapted to optimise the effectiveness of the human operators by further comprising 3 core sub-systems, namely (i) a pre-processing front end that determines an appropriate conversion strategy; (ii) one or more conversion resources; and (iii) a quality control sub-system.

Claims

1. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

3 core sub-systems, namely (i) a pre-processing front end that determines an appropriate conversion strategy;

(ii) one or more conversion resources; and

(iii) a quality control sub-system.

2. The system of claim 1, in which the conversion resources include one or more of the following:

one or more ASR engines;

signal processing resources;

the human operators.

3. The system of claim 2 in which the signal processing resources optimise the quality of the audio for conversion by performing one or more of the following functions:

removing noise, cleaning up known defects, normalising volume/signal energy, removing silent/empty sections.

4. The system of claim 1 in which the human operators perform random quality assurance testing on converted messages and provide feedback to the pre-processing front-end and/or the conversion resources.

74. The system of claim 1 in which the message is a voicemail intended for a mobile telephone and the voice message is converted to text and sent to that mobile telephone.

75. The system of claim 1 in which the message is a voice message intended for an instant messaging service and the voice message is converted to text and sent to an instant messaging service for a display on a screen.

76. The system of claim 1 in which the message is a voice message intended for a web blog and the voice message is converted to text and sent to a server for display as part of the web blog.

77. The system of claim 1 in which the message is a voice message intended to be converted to text format and sent as a text message.

78. The system of claim 1 in which the message is a voice message intended to be converted to text format and sent as an email message.

79. The system of claim 1 in which the message is a voice message intended to be converted to text format and sent as a note or memo, by email or text, to an originator of the message.

80. A mobile telephone network that is connected to the system of claim 1.

81. A mobile telephone when displaying a message converted by the system of claim 1.

82. A computer display screen when displaying a message converted by the system of claim 1.

83. A method of providing voice messaging, comprising the step of a user sending a voice message to a messaging system as claimed in claim 1.

5. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

a computer implemented context sub-system adapted to use information about the context of a message or a part of a message to improve the conversion accuracy.

6. The system of claim 5 in which the context information is used to limit the vocabulary used in any ASR engine or refine search or matching processes used by the ASR engine.

7. The system of claim 5 in which the context information is used to select a particular conversion resource or combination of conversion resources, such as a particular ASR engine.

8. The system of claim 5 in which context information includes one or more of caller ID, recipient ID, whether the caller or recipient is a business or other type of classifiable entity or not;

caller-specific language;

call-pair history;

time of call;

day of call;

geo-reference or other location data of the caller or callee;

PIM data (personal information management data, including address book, diary) of the caller or callee;

the message type, including whether the message is a voice mail, a spoken text, an instant message, a blog entry, an e-mail, a memo, or a note;

message length;

information discoverable using an online corpus of knowledge;

presence data;

speech density of the message;

speech quality of the message.

9. The system of claim 5 in which the context sub-system includes or operates with a recogniser confidence sub-system that determines automatically the confidence level associated with a conversion of a specific message, or part of a message, using the context information.

10. The system of claim 9 in which the context sub-system includes or operates with a recogniser confidence sub-system that determines automatically the confidence level associated with a conversion of a specific message, or part of a message, using the output of one or more ASR engines.

11. The system of claim 10 in which the recogniser confidence sub-system can dynamically weight how it uses the output of different ASR engines depending on their likely effectiveness or accuracy.

12. The system of claim 5 in which knowledge of the context of a message is extracted by one sub-system and fed-forward to a downstream sub-system that uses that context information to improve conversion performance.

13. The system of claim 12 in which a downstream sub-system is a quality monitoring and/or assurance and control sub-system.

14. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

a computer implemented call-pair sub-system adapted to use call-pair history information to improve conversion accuracy.

15. The system of claim 14 in which the call-pair history enables the system to be user-independent but to acquire over time, without explicit user training, user-dependent data that enables conversion performance to be improved.

16. The system of claim 14 in which the call-pair history is associated with a pair of numbers, including numbers associated with mobile telephones, fixed telephones, IP addresses, e-mail addresses, or unique addresses provided by a network.

17. The system of claim 14 in which the call-pair history includes information relating to one or more of:

likely language or dialect being used;

country called from or called to;

time zones;

time of call;

day of call;

specific phrases used;

caller-specific language;

intonation;

PIM data personal information management data, including address book, diary).

18. The system of claim 14 further comprising a computer implemented dynamic language model sub-system adapted to build a dynamic language model using one or more of:

caller dependence;

call-pair dependence;

callee dependence.

19. The system of claim 14 in which the caller is anyone depositing a voice message, irrespective of whether they intend to place a voice call;

and the callee is anyone who reads the converted message, irrespective of whether they were meant to receive a voice call.

20. The system of claim 14 further comprising a computer implemented personal profile sub-system adapted to build a personal profile of a caller to improve conversion accuracy.

21. The system of claim 20 in which the personal profile includes words, phrases, grammar, or intonation of the caller.

22. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

a computer implemented boundary selection sub-system adapted to process a message by looking for the boundaries between sections of the message which carry different types of content or carry different types of message.

23. The system of claim 22 in which the computer implemented boundary selection sub-system analyses for one or more of the following component parts:

a greeting part;

a body part;

a goodbye part.

24. The system of claim 22 in which different conversion strategies are applied to each part, the applied strategy being optimal for converting that part.

25. The system of claim 22 in which different parts of the message have different quality requirements and a quality assessment sub-system applies different standards to those different parts.

26. The system of claim 22 in which a speech quality estimator detects boundaries between sections of the message which carry different types of content or carry different types of message.

27. The system of claim 26 in which boundaries are detected or inferred at regions in the message where the speech density alters.

28. The system of claim 26 in which boundaries are detected or inferred at a pause in the message.

29. The system of claim 26 in which boundaries are inferred as arising at a pre-defined proportion of the message.

30. The system of claim 26 in which a greeting boundary is inferred at approximately 15% of the entire message length.

31. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

a computer implemented pre-processing front-end sub-system that determines an appropriate conversion strategy used to convert the voice messages.

32. The system of claim 31 in which the pre-processing front end optimises the quality of the audio for conversion by performing one or more of the following functions:

removing noise, cleaning up known defects, normalising volume/signal energy, removing silent/empty sections and classifies the message type for optimally routing the message for conversion, or not.

33. The system of claim 31 in which the pre-processing front-end determines the language being used by the caller, based on one or more of the following:

knowledge about registration, location and call history of caller and/or receiver.

34. The system of claim 31 in which the pre-processing front-end selects a particular ASR engine to convert a message or part of a message.

35. The system of claim 31 in which different conversion resources, such as ASR engines, are used for different parts of the same message.

36. The system of claim 31 in which different conversion resources, such as ASR engines, are used for different messages.

37. The system of claim 31 in which the human operators are treated as ASR engines.

38. The system of claim 31 in which the pre-processing front-end uses or is connected with a recogniser confidence sub-system to determine automatically the confidence level associated with a conversion of a specific message, or part of a message, and a particular conversion resource, such as an ASR engine, is then deployed depending on that confidence level.

39. The system of claim 31 in which the conversion strategy involves the selection a conversion strategy from a set of conversion strategies that include the following:

(i) messages for which an ASR conversion confidence is sufficiently high are checked automatically by a quality assessment sub-system for conformance to quality standards;

(ii) messages for which the ASR conversion confidence is not sufficiently high are routed to a human operator for checking and, if necessary, correction;

(iii) messages for which the ASR conversion confidence is very low are flagged as unconvertible and the user is informed of the receipt of an unconvertible message.

40. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

a computer implemented queue manager sub-system that intelligently manages loading and calls in resources as required to ensure that converted message delivery times meet a pre-defined standard.

41. The system of claim 40 in which the queue manager sub-system determines what should happen to a voice message at each stage of processing through the system.

42. The system of claim 40 in which if at any stage of the automated conversion, confidence intervals or other measures suggest any part of a message are not good enough, then the queue manager directs this to the correct human operator for assistance.

43. The system of claim 40 in which the queue manager sub-system makes decisions by calculating trade-offs between conversion time and quality.

44. The system of claim 40 in which the queue manager sub-system uses state-machines that, for any given language queue, can decide how best to process the messages through the system.

45. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising

a computer implemented lattice sub-system that generates a lattice of possible word or phrase sequences and enables a human operator to guide a conversion sub-system by being shown one or more candidate converted words or phrases from the lattice and enabling the operator to either select that candidate word or phrase or, by entering one or more characters for a different converted word or phrase, to trigger the conversion sub-system to propose an alternative word or phrase.

46. The system of claim 45 in which the lattice sub-system receives inputs from a sub-system that handles call-pair history information.

47. The system of claim 45 in which the lattice sub-system receives inputs from conversion resources.

48. The system of claim 45 in which the lattice sub-system receives inputs from a context sub-system that has knowledge of the context of a message.

49. The system of claim 45 in which the lattice sub-system learns, from the human operator inputs, likely words corresponding to a sound pattern.

50. The system of claim 45 in which the human operator is required to select only a single key to accept a word or phrase.

51. The system of claim 45 in which the lattice sub-system automatically provides capitalisation and punctuation.

52. The system of claim 45 in which the lattice sub-system can propose candidate numbers, real nouns, web addresses, e-mail addresses, physical addresses, location information, or other coordinates.

53. The system of claim 45 in which the lattice sub-system automatically differentiates between parts of the message that are likely to be important and those that are likely to be unimportant.

54. The system of claim 45 in which unimportant parts of the message are confirmed by the operator as belonging to a class proposed by the lattice sub-system and are then converted solely by a machine ASR engine.

55. The system of claim 45 in which the human operator can speak the correct word to the conversion system, which then automatically transcribes it.

56. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

a computer implemented search sub-system that analyses a converted message against an on-line corpus of knowledge.

57. The system of claim 56 in which the on-line corpus of knowledge is the internet, as accessed by a search engine.

58. The system of claim 56 in which the on-line corpus of knowledge is a search engine database, such as Google.

59. The system of claim 56 in which the analysis of the converted message enables the accuracy of the conversion to be assessed by a human operator an/or a recogniser confidence sub-system.

60. The system of claim 56 in which the analysis of the converted message enables ambiguities in the message to be resolved by a human operator and/or an ASR engine.

61. A mass-scale, user-independent, device-independent, voice messaging system that converts unstructured voice messages into text for display on a screen;

the system comprising (i) computer implemented sub-systems and also (ii) a network connection to human operators providing transcription and quality control;

the system being adapted to optimise the effectiveness of the human operators by further comprising;

a computer implemented detector sub-system that is adapted to detect slam-downs.

62. The system of claim 61 in which the slam down detector is implemented as part of a pre-processing front-end.

63. The system of claim 61 further comprising a computer implemented detector sub-system that is tuned to detect different spoken languages, such as English, Spanish, French etc.

64. The system of claim 63 in which the language detector can detect changes in language part way through a message.

65. The system of claim 63 in which the language detector can use inputs from a sub-system that has call-pair history information that records how changes in language occurred in earlier messages.

66. The system of claim 61 further comprising a computer implemented detector sub-system that is adapted to estimate speech quality.

67. The system of claim 66 in which the speech quality estimator finds drop-outs, estimate noise levels and calculate an overall measure of voice quality and uses an adaptive threshold to reject lowest quality messages.

68. The system of claim 61 further comprising a computer implemented detector sub-system that is adapted to detect inadvertent calls.

69. The system of claim 68 in which the inadvertent call detector is implemented as part of a pre-processing front-end.

70. The system of claim 61 further comprising a computer implemented detector sub-system that is adapted to detect and convert pre-recorded messages.

71. The system of claim 61 further comprising a computer implemented detector sub-system that is adapted to detect and convert spoken numbers.

72. The system of claim 61 further comprising a computer implemented detector sub-system that is adapted to detect and convert spoken addresses.

73. The system of claim 61 further comprising a computer implemented detector sub-system that is adapted to detect and convert real nouns, numbers, web addresses, e-mail addresses, physical addresses, location information, other coordinates.