Adjustable resource based speech recognition system

• US 20050119896A1
• Filed: 01/07/2005
• Published: 06/02/2005
• Est. Priority Date: 11/12/1999
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Phoenix Solutions Incorporated (CDC Corporation)
• Inventors: Babu, Bandi Ramesh, Gururaj, Pallaki, Morkhandikar, Kishor, Bennett, Ian M.
• Granted Patent: US 7,139,714 B2
• Time in Patent Office: Days
• US Class Current: 704/270.100
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A real-time speech recognition system includes distributed processing across a client and server for recognizing a spoken query by a user. Both the client and server can dedicate a variable number of processing resources for performing speech recognition functions. In some implementations the partitioning of responsibility for speech recognition operations can be done on a client by client or query by query basis.

Claims

1. A method of recognizing a speech utterance from a user at a network server system comprising the steps of:

a) evaluating an amount of computing resources available at the network server system for performing speech recognition operations on speech related data transmitted by a client device being used by the user;

wherein said speech related data corresponds to partially recognized speech data;

b) specifying a first set of speech recognition operations to be performed by the network server system on said speech related data from the client device in response to step (a);

wherein said first set of speech recognition operations are specified during an initialization period and are configurable on a connection-by-connection basis with each client device;

further wherein full recognition of the speech utterance is performed across a distributed client—

server system.

2. The method of claim 1, wherein said partially recognized speech data includes a byte stream of mel frequency cepstrum coefficients (MFCC).

3. The method of claim 1, wherein the client device is configured to perform a second set of speech recognition operations based on evaluating an amount of computing resources available at the client device.

4. The method of claim 1, wherein said first set of speech recognition operations are configured based on an application executing on the client device.

5. The method of claim 1, wherein said speech related data is communicated under control of an INTERNET browser over an INTERNET connection to the network server system.

6. The method of claim 1 wherein said speech utterance is mapped to a predefined query/answer pair stored in a query database maintained by an operator of the network server system.

17. The method of claim 1 further including a step:

providing an electronic agent to present interactive real-time responses to the speech utterance in a same language as said speech utterance, said electronic agent being adapted to mimic behavior and responses of a human agent.

18. The method of claim 17 wherein said electronic agent has a visual appearance and natural language speech output correlated to a context experienced by a user providing the speech utterance, an environment experienced by said user, and/or a group associated with said user.

19. The method of claim 17 wherein a different electronic agent can be presented to different users providing speech utterances received by the network server system.

20. The method of claim 17 wherein characteristics of said electronic agent, including visual appearance parameters and/or vocal parameters can be controlled by a user of the client device.

7. A method of performing recognition of a speech utterance at a network server system comprising the steps of:

a) evaluating an amount of computing resources available at the network server system for performing speech recognition operations on speech related data;

b) specifying a first set of speech recognition operations to be performed by the network server system on said speech related data in response to step (a);

c) receiving first speech data through a communications interface of the network server system, said first speech data being associated with a partial recognition of the speech utterance completed by a client device;

d) completing recognition of the speech utterance using software routines executing at the network server system which implement said first set of speech recognition operations.

8. The method of claim 7 including the steps of:

(e);

evaluating an amount of computing resources available at the client device for performing speech recognition operations on speech data; and

(f) specifying a second number of speech recognition operations to be performed by the client device on said speech data in response to step (e).

9. The method of claim 8, wherein said computing resources include computers and memory.

10. The method of claim 7 wherein said first set of speech recognition operations can be changed for a subsequent speech utterance.

11. The method of claim 7 wherein said first set of speech recognition operations is changed in response to changes in said amount of computing resources.

12. The method of claim 7 further including a step:

performing natural language processing of said speech utterance to recognize a meaning of a sentence of words contained therein.

13. The method of claim 12 wherein said network server system includes a plurality of separate natural language engines.

14. The method of claim 7 wherein said network server system includes a plurality of separate servers.

15. The method of claim 7 further including a step:

dynamically switching a grammar at the network server system in response to the speech utterance, which grammar includes words correlated to a status of a user within an application program.

16. The method of claim 7 further including the steps:

identifying a natural language used in the speech utterance and providing an audible response in said natural language.

21. The method of claim 7 wherein said first speech data is received as a byte stream.

22. The method of claim 21 wherein said first speech data is received from a browser program running on the client device and connected over the INTERNET to the network server system.

23. The method of claim 7 wherein said first speech data is received continuously during a speech utterance.

24. The method of claim 23 wherein said first speech data is received continuously until silence is detected.

25. A method of performing query recognition at a network server system comprising the steps of:

a) evaluating an amount of computing resources available at a client device for performing query recognition operations on speech utterance data received from a user;

b) evaluating an amount of computing resources available at the network server system for performing query recognition operations on speech related data received over an INTERNET connection from the client device;

c) specifying a first set of query recognition operations to be performed by the client device, and a second set of query recognition operations to be performed by the network server system in response to steps (a) and (b);

wherein partitioning of query recognition operations between the client device and network server system is performed on a client device-by-client device basis;

c) receiving first speech data through a communications interface of the network server system, said first speech data being associated with a partial recognition of the speech utterance completed by the client device;

d) completing recognition of the speech utterance using software routines executing at the network server system which implement said first set of speech recognition operations to generate a set of words associated with the speech utterance;

wherein grammars used for recognizing said speech utterance are dynamically switched by the network server system based on identifying a computer application which the user is interacting with during the speech utterance;

e) processing said set of words using one or more natural language engines operating at the network server system to complete recognition of a user query.

26. The method of claim 25, wherein a predefined query/answer pair database selected for said computer application is also dynamically switched to recognize said user query.

27. A system for recognizing a speech utterance from a user at a network server system comprising:

a) a first routine adapted for evaluating an amount of computing resources available at the network server system for performing speech recognition operations on speech related data transmitted by a client device being used by the user;

wherein said speech related data corresponds to partially recognized speech data;

b) a second routine adapted for performing a first set of speech recognition operations at the network server system on said speech related data from the client device in response to step (a);

wherein said first set of speech recognition operations are specified during an initialization period and are configurable on a connection-by-connection basis with each client device;

further wherein full recognition of the speech utterance is performed across a distributed client-server system.

28. The system of claim 27, wherein said first routine is included as part of an INTERNET web browser program executing on the client device.

29. The system of claim 27, wherein the client device is a portable electronic appliance adapted for INTERNET communications.

30. The system of claim 29, wherein said speech related data is transmitted using a byte stream that is continuous during non-silent periods.

31. A system for performing recognition of a speech utterance at a network server system comprising:

a) a first routine adapted to evaluate an amount of computing resources available at the network server system for performing speech recognition operations on speech related data;

wherein a first set of speech recognition operations are performed by the network server system on said speech related data in response to step (a);

c) a second routine adapted to receive first speech data through a communications interface of the network server system, said first speech data being associated with a partial recognition of the speech utterance completed by a client device;

d) a third routine adapted to complete recognition of the speech utterance based on said first set of speech recognition operations.