Distributed realtime speech recognition system

• US 6,633,846 B1
• Filed: 11/12/1999
• Issued: 10/14/2003
• Est. Priority Date: 11/12/1999
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Phoenix Solutions Incorporated (CDC Corporation)
• Inventors: Babu, Bandi Ramesh, Morkhandikar, Kishor, Gururaj, Pallaki, Bennett, Ian M.
• Primary Examiner(s): Banks-Harold, Marsha D.
• Assistant Examiner(s): Lerner, Martin
• Application Number:
  US09/439,145
• Time in Patent Office: 1,432 Days
• Field of Search: 704/9, 704/10, 704/232, 704/251, 704/255, 704/256, 704/257, 704/270, 704/275, 704/270.1, 434/185
• US Class Current: 704/257
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

A real-time system incorporating speech recognition and linguistic processing for recognizing a spoken query by a user and distributed between client and server, is disclosed. The system accepts user'"'"'s queries in the form of speech at the client where minimal processing extracts a sufficient number of acoustic speech vectors representing the utterance. These vectors are sent via a communications channel to the server where additional acoustic vectors are derived. Using Hidden Markov Models (HMMs), and appropriate grammars and dictionaries conditioned by the selections made by the user, the speech representing the user'"'"'s query is fully decoded into text (or some other suitable form) at the server. This text corresponding to the user'"'"'s query is then simultaneously sent to a natural language engine and a database processor where optimized SQL statements are constructed for a full-text search from a database for a recordset of several stored questions that best matches the user'"'"'s query. Further processing in the natural language engine narrows the search to a single stored question. The answer corresponding to this single stored question is next retrieved from the file path and sent to the client in compressed form. At the client, the answer to the user'"'"'s query is articulated to the user using a text-to-speech engine in his or her native natural language. The system requires no training and can operate in several natural languages.

Claims

1. A machine executable program for assisting a computing system to effectuate distributed voice query recognition comprising:

a first audio signal receiving routine for receiving user speech utterance signals representing speech utterances to be recognized, said speech utterances including sentences comprised of one or more words; and

a first signal processing routine adapted to generate representative speech data values from said speech utterance signals, said representative speech data values being characterized by a first data content that is substantially inadequate by itself for permitting recognition of words articulated in said speech utterance; and

a formatting routine for rendering said representative speech data values into a transmission format suitable for transmission over a communications channel to a second processing routine executing on a separate computing system

wherein said representative speech data values are transmitted continuously during said speech utterances within streaming packets and without waiting for silence to be detected and/or said speech utterances to be completed; and

wherein said first data content in said representative speech data values is used by said second processing routine to compute additional data content that when combined with said first data content is sufficient for a speech recognition routine to complete recognition of words articulated in said speech utterance at said separate computing system

and further wherein an amount of said first data content transmitted is configured and can be varied for said speech utterances based on signal processing capabilities of the computing system and/or transmission characteristics of said communications channel.

2. The program of claim 1, wherein said program works within a browser program executing on said computing system as part of a client-server based system.

3. The program of claim 1, wherein said first signal processing routine is further adapted to handle a stream of continuous speech utterances, such that a plurality of representative speech data values are generated for each of said speech utterances in real-time.

4. The program of claim 1, wherein said first data content is sufficiently small that said formatting routine can handle in real-time a continuous stream of representative speech data values that may be generated for a corresponding stream of speech utterances.

5. The program of claim 1, wherein each of said representative speech data values corresponds to a separate cepstral coefficient value for a corresponding frequency component of said user speech utterance signals, and said first data content corresponds to a set of said frequency components spanning an audible speech frequency range.

6. The program of claim 5, wherein said additional data content corresponds to a set of delta and acceleration coefficients computed from a corresponding set of said ceptstral coefficient values.

7. The program of claim 6, wherein said additional data content is generated by said second processing routine with less latency than would that resulting if said additional data content were generated by said first signal processing routine.

8. The program of claim 1, wherein signal processing functions required to generate said first data content and said additional data content can be allocated as needed between said first signal processing routine operating on a client device and said second signal processing routine operating on a server device based on a determination of computing resources available to said first and second signal processing routines respectively on a client device-by-device basis.

9. The program of claim 1, wherein said first signal processing routine is a set of instructions executed by any one of a host microprocessor, an embedded processor, and/or a digital signal processor (DSP).

10. The program of claim 1, wherein said first signal processing routine and said first audio signal are implemented as part of a desktop computing system, a portable computing system, a personal digital assistant (PDA), a cell-phone, and/or an electronic interactive toy.

31. The system of claim 1, wherein said partially recognized speech data includes an observation vector O_t, and second signal processing operation includes a Viterbi decoding operation for mapping a sequence of said observation vectors O_t with a speech data symbol.

32. The system of claim 10, wherein said second signal processing operation further includes a text conversion operation for converting said speech data symbol into one or more text words.

11. A distributed voice recognition system comprising:

a sound processing circuit adapted to receive a speech utterance and to generate associated speech utterance signals therefrom; and

a first signal processing circuit adapted to generate a first set of speech data values from said speech utterance signals, said first set of speech data values being insufficient by themselves for permitting recognition of words articulated in said speech utterance; and

a transmission circuit for formatting and transmitting said first set of speech data values over a communications channel to a second signal processing circuit;

wherein said first set of speech data values are sent in a streaming fashion over said channel before silence is detected and/or said speech utterance is completed; and

said second signal processing circuit being configured to generate a second set of speech data values based on receiving and processing said speech data values during said speech utterance and before silence is detected, such that second set of speech data values contain sufficient information to be usable by a word recognition engine for recognizing words in said speech utterance;

and further wherein at least some words are recognized in real-time and output as text before said speech utterance is completed.

12. The system of claim 11, wherein said second set of speech data values include said first set of speech data values and a derived set of speech data values, which derived set of speech data values are computed based on said first speech data values.

13. The system of claim 12, wherein said first set of data values are MFCC vector coefficients, and said derived set of speech data values are MFCC delta coefficients and a MFCC acceleration coefficients derived from said MFCC vector coefficients.

14. The system of claim 11, wherein said second set of speech data values can be generated by said second signal processing circuit in a time that is less than the combination of a first time which would be required by said first signal processing circuit to generate said second set of speech data values from said first set of speech data values combined with a second time which would be required by said transmission circuit to format and transmit said second set of speech data values.

15. The system of claim 11, wherein said second set of speech data values can be generated by said second signal processing circuit in less time than that which would be required by said first signal processing circuit to generate said second set of speech data values from said first set of speech data values.

16. The system of claim 11, wherein signal processing responsibilities of said first and second signal processing circuits are allocated such that said first signal processing circuit performs less than approximately ½

the required signal processing operations needed to convert said speech utterance signals into a form usable by a word recognition engine.

17. The system of claim 11, wherein signal processing functions required to generate said first and second set of speech data values can be allocated between said first signal processing circuit and second signal processing circuit as needed based on computing resources available to said first and second signal processing circuits respectively.

18. The system of claim 11, wherein signal processing functions required to generate said first and second set of speech data values can be allocated between said first signal processing circuit and second signal processing circuit as needed based on computing resources available to said first and second signal processing circuits respectively.

19. The system of claim 11, wherein signal processing functions performed by said first signal processing circuit and second signal processing circuit are configured based on:

(i) computing resources available to said first and second signal processing circuits;

(ii) performance characteristics of said transmission circuit; and

(iii) transmission latencies of said communications channel.

20. The system of claim 11, wherein said first signal processing circuit is also configured to assist said second signal processing circuit with signal processing computations required to generate said second set of speech data values.

21. The system of claim 11, wherein said first set of speech data values represent the least amount of data that can used by said second signal processing circuit to generate said second set of data values usable for a word recognition process.

22. A system for recognizing speech information, comprising:

storage means for storing one or more words to be recognized by the system based on the speech information; and

means for capturing speech signals corresponding to the speech information; and

first processing means for generating partially recognized speech data from said speech signals, said first processing means performing a first signal processing operation on said speech signals, said first signal processing operation being insufficient to permit said partially recognized speech data to be correlated with said one or more words; and

second processing means for generating fully recognized speech data from said partially recognized speech data, using a second signal processing operation, such that said fully recognized speech data can be correlated with said one or more words, said second processing means being distinct and physically separated from said first processing means; and

third processing means for generating recognized sentence data from said one or more words using natural language processing operations including word phrase analysis performed on said one or more words;

a non-permanent data transmission connection coupling said first and second processing means;

transmitting means for transmitting said partially processed speech data signals from said first processing means through said non-permanent data transmission connection to said second processing means, said transmitting means using a continuously generated byte stream that is transmitted while a speech utterance is occurring and before silence is detected;

wherein the system recognizes a complete sentence included in the speech information based on said recognized sentence data and determines a best response to said complete sentence in real-time.

23. The system of claim 22, wherein said first processing means is located at a client site, and said second processing means is located at a remote server site.

24. The system of claim 23, wherein said storage means is also located at said server site.

25. The system of claim 22, wherein said non-permanent connection is either a circuit switched or packet switched connection.

26. The system of claim 22, wherein said non-permanent connection includes an intranet network linking said first and second signal processing means.

27. The system of claim 22, wherein said non-permanent connection is a wireless communications channel.

28. The system of claim 22, wherein said first signal processing operation includes an operation for extracting spectral parameter vectors from said speech signals.

29. The system of claim 28, wherein said first signal processing operation further includes an operation for decomposing the spectral parameter vectors with a Mel-frequency transfer process to obtain MFCC coefficients for said spectral parameter vectors.

30. The system of claim 28, wherein said partially recognized speech data comprises an observation vector O_t which includes said MFCC coefficients and delta and acceleration coefficients obtained from said spectral parameter vectors.

33. The system of claim 22, wherein said second processing means further utilizes environment variables for determining which of said one or more text words correspond to such speech information.

34. A method of performing speech recognition comprising the steps of:

(a) receiving user speech utterance signals representing speech utterances to be recognized, said speech utterances including sentences comprised of one or more words; and

(b) generating representative speech data values with a first computing device which by themselves are insufficient for completing recognition of said one or more words contained in said speech utterance signals; and

(c) formatting said representative speech data values into a transmission format suitable for transmission over a communications channel from said first computing device to a second computing device; and

wherein said representative speech data values are transmitted continuously during said speech utterances within streaming packets and without waiting for silence to be detected and/or said speech utterances to be completed; and

(d) performing a recognition of said one or more words at said second computing device using said representative speech data values and additional speech data values derived from said representative speech data values to generate recognized text;

(e) performing a natural language processing operation on said recognized text to determine a meaning associated with said sentences in real-time.

35. The method of claim 34, wherein recognition of said one or more words is achieved with less latency than that resulting if said recognition were performed entirely by said first computing device.

36. The method of claim 34, wherein signal processing functions required to perform said recognition can be allocated and configured between said first computing device and said second computing device as needed based on an evaluation of computing resources available to said first and second computing devices respectively.

37. The method of claim 34, wherein said first computing device is part of a client computing system, and said second computing device is part of a server computing system, and said communications channel is a network.

38. A speech recognition program operating on a server coupled through a network to a client device, the program comprising:

a receiving routine for receiving speech data from the client device, said speech data being associated with a speech utterance from a user of the client device;

wherein said speech data has a data content that is adjusted based on signal processing capabilities of the client device and resources available to the server for processing speech data;

further wherein said speech data is transmitted continuously by the client device during said speech utterance within streaming packets and without waiting for silence to be detected and/or said speech utterance to be completed;

a speech recognition processing routine, for recognizing words contained in said speech utterance;

wherein when said speech data from the client device is insufficient for recognizing words, additional speech related data is computed by the server to generate additional speech data to be combined with said speech data as an input to said speech recognition processing routine;

a natural language processing routine, for recognizing word sentences contained in said speech utterance by performing one or more natural language processing operations on words contained in said speech utterance.

39. The program of claim 38, wherein said data content is also adjusted by considering transmission characteristics of the network, and a transmission speed capability of the client device.

40. The program of claim 38, wherein a calibration routine automatically determines an optimal setting for said data content.

41. The program of claim 40, wherein said calibration routine executes on the client device.

42. The program of claim 38, wherein said speech data includes mel frequency cepstral coefficients (MFCCs), and said additional speech related data includes MFCC delta coefficients and a MFCC acceleration coefficients derived from said MFCC vector coefficients.

43. The program of claim 38, wherein the network includes one of an INTERNET connection, a LAN connection, a wireless connection, or a combination thereof.

44. The program of claim 38, wherein said words contained in said speech utterance are recognized in real-time by said speech recognition routine.

45. The program of claim 44, wherein said word sentences contained in said speech utterance are recognized in real-time by said natural language processing routine.

46. The program of claim 38, wherein the client device includes one of a desktop computer, a portable computer, a personal digital assistant, and a cell phone.

47. The program of claim 38, wherein said natural language processing operations include word phrase analysis, including a noun phrase analysis.

48. A method of performing speech recognition at a server coupled to a client device through a network, the method comprising the steps of:

(a) determining signal processing capabilities of the client device available for processing speech data associated with a user speech utterance;

(b) determining resources available to the server for processing speech related data that is to be transmitted by the client device to the server;

(c) configuring a data content to be used for transmitting said speech related data through the network based on the results of steps (a) and (b);

(d) receiving said speech related data with said data content at the server continuously from the client device during said speech utterance within streaming packets and without waiting for silence to be detected and/or said speech utterance to be completed;

(e) processing said speech related data at the server so that additional speech related data is computed at the server from said speech data and is used to augment said speech related data when said speech related data contains acoustic feature data from said speech utterance that is insufficient for the server to perform accurate recognition of words contained in said speech utterance;

(f) generating a speech data observation vector suitable for processing by a speech recognition routine, said speech data observation vector being based on said speech related data as received from the client device, and/or said speech related data as augmented by the server;

(g) recognizing words contained in said speech utterance with a speech recognition engine using said speech data observation vector;

(h) recognizing word sentences contained in said speech utterance by performing one or more natural language processing operations on words contained in said speech utterance.

49. The method of claim 48, wherein said data content is also configured by considering transmission characteristics of the network, and a transmission speed capability of the client device.

50. The method of claim 48, further including a step of executing a calibration routine to automatically determine an optimal setting for said data content.

51. The method of claim 50, wherein said calibration routine executes on the client device.

52. The method of claim 48, wherein said acoustic feature data includes mel frequency cepstral coefficients (MFCCs), and said additional speech related data includes MFCC delta coefficients and a MFCC acceleration coefficients derived from said MFCC vector coefficients.

53. The method of claim 48, wherein the network includes one of an INTERNET connection, a LAN connection, a wireless connection, or a combination thereof.

54. The method of claim 48, wherein said words contained in said speech utterance are recognized in real-time by said speech recognition routine.

55. The method of claim 54, wherein said word sentences contained in said speech utterance are recognized in real-time by said natural language processing routine, and a response is sent to the client device in real-time.

56. The method of claim 48, wherein the client device includes one of a desktop computer, a portable computer, a personal digital assistant, and a cell phone.

57. The method of claim 48, wherein said natural language processing operations include word phrase analysis, including a noun phrase analysis.