Speech recognition system and method

• US 7,827,035 B2
• Filed: 09/29/2008
• Issued: 11/02/2010
• Est. Priority Date: 11/13/1998
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: McGregor, Pedro, Grant, Richard
• Primary Examiner(s): Chawan; Vijay B
• Application Number:
  US12/240,983
• Publication Number:
  US 20090024391A1
• Time in Patent Office: 764 Days
• Field of Search: 704270-275, 704/251, 704/231, 704/257, 704/260, 704/277
• US Class Current: 704/275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 2015/228 of application context

Abstract

According to the present invention, a method for integrating processes with a multi-faceted human centered interface is provided. The interface is facilitated to implement a hands free, voice driven environment to control processes and applications. A natural language model is used to parse voice initiated commands and data, and to route those voice initiated inputs to the required applications or processes. The use of an intelligent context based parser allows the system to intelligently determine what processes are required to complete a task which is initiated using natural language. A single window environment provides an interface which is comfortable to the user by preventing the occurrence of distracting windows from appearing. The single window has a plurality of facets which allow distinct viewing areas. Each facet has an independent process routing its outputs thereto. As other processes are activated, each facet can reshape itself to bring a new process into one of the viewing areas. All activated processes are executed simultaneously to provide true multitasking.

Claims

1. A speech-input enabled computing method for processing spoken commands, comprising:

supporting user interaction with a plurality of concurrently active software constructs, comprising at least one application which executes under an operating system, through a graphic user interface, the focus being granted to a software construct based on at least a user speech input;

storing a representation of a set of command grammars corresponding to commands for at least a portion of the plurality of software constructs, at least a portion of which stored representation represent commands currently available for processing;

processing the user speech input with a speech analyzer based on at least a portion of said stored representation, to determine if a user speech input corresponds to a represented command grammar available for processing, and if so processing a corresponding command by the operating system or the respective application to which it relates; and

modifying the graphic user interface in dependence on said processing.

2. The method according to claim 1, wherein if the user speech input corresponds to a represented command grammar and which does not satisfy the conditions for execution thereof, further sequentially processing the user speech input for further elements of a represented command grammar.

3. The method according to claim 1, wherein the graphic user interface displays windows associated with a plurality of software constructs concurrently, at least one of which is adapted to receive at least one command from a user speech input.

4. The method according to claim 1, wherein the graphic user interface comprises a plurality of windows, wherein a plurality of active software constructs are associated each with a respective window, and wherein the windows are manipulated based at least on a user speech input.

5. The method according to claim 1, wherein at least one unitary command grammar requires a functional execution sequence by at least two of a set of software constructs.

6. The method according to claim 5, wherein the processing step controls a sequence of corresponding command execution of the at least two of the set of software constructs, in dependence on at least the output of the speech analyzer.

7. The method according to claim 1, wherein at least one command requires a plurality of software constructs to execute concurrently.

8. The method according to claim 7, wherein the plurality of concurrently executing software constructs exchange data therebetween in dependence on at least a user speech input.

9. The method according to claim 1, wherein the processing step is context-sensitive, and context of the user speech input is determined based on at least one of:

a preceding and subsequent user speech input or absence thereof;

a time in relation to at least one of the preceding and subsequent user speech input;

a time elapsed between successive sequences of user speech input;

a beginning of a user speech input; and

a user speech input preceding and following the represented command; and

wherein the user speech input is processed as a command or data in accordance with the determination.

10. The method according to claim 1, wherein at least one command grammar and the corresponding set of stored representations of commands currently available for processing is modified in dependence on a manipulation of the graphical user interface.

11. The method according to claim 1, wherein a command grammar for a set of available commands is registered in the stored representation.

12. The method according to claim 1, wherein at least one of a command grammar or data on at least a set of commands potentially available for processing used to build a command grammar is persistently stored, and the representation of corresponding commands currently available for processing is derived therefrom.

13. The method according to claim 12, wherein the stored representation of at least one command grammar is defined dynamically.

14. The method according to claim 12, wherein at least a portion of the stored representation of at least one command grammar is flagged as available based on represented commands then currently available.

15. The method according to claim 1, wherein the processing step is responsive to a command activation statement.

16. The method according to claim 1, wherein the processing step responds to natural language input.

17. The method according to claim 1, further comprising converting text to speech for output thereof.

18. The method according to claim 1, wherein the plurality of software constructs comprise a plurality of concurrently active applications comprising at least one process.

19. The method according to claim 1, wherein the graphic user interface is part of a multi-tasking operating system, and wherein the speech analyzer is a software construct executing under the multi-tasking operating system.

20. The method according to claim 1, wherein the stored representation is a representation of a set of command grammars representing at least a plurality of commands available for processing, and processing of the user speech input by the speech analyzer is done in dependence on said representation, to determine if a speech input corresponds to a command available for processing, and if so processing the command.

21. The method according to claim 1, further comprising at least one of:

determining whether a user speech input is associated with a process executing within a window, and if the process has not been activated, creating a new window associated with the activated process;

determining of whether a user speech input is intended as a command in dependence on a recognition of a command activation statement in the user speech input;

applying a the predetermined grammar to validate the user speech input command comprises at least applying the user speech input to a grammar derived structure as it is received, and waiting until the grammar derived structure is fulfilled prior to passing the command for processing;

applying a predetermined grammar to validate the user speech input command comprises at least applying the user speech input to the context based data structure as it is received, and prompting the user to provide required information for validation of the command;

storing a data stream representing the user speech input, which is then searched for an indication that the user speech input is intended as a command;

storing an indicator of whether a user speech input is intended as a command or intended as data;

reading data from a registration database comprising a plurality grammars associated with respective available commands;

distinguishing a target command processor for a user speech input intended as a command, and selectively passes a validated command to respective one of a plurality of command processors in dependence thereon.

reading data from a registration database comprising a plurality grammars associated with respective available commands, and determines a context based on at least a past user interaction.

22. A computer readable medium for controlling a programmable computer to perform the steps of claim 1.

23. A speech-input enabled computing system adapted to receive spoken commands, comprising:

a set of executable software constructs adapted to execute within a processor having a memory, comprising an operating system having a graphic user interface, supporting user interaction with a plurality of concurrently active software constructs comprising at least one application which executes under the operating system, said computing system processing commands in dependence on at least a user speech input;

a memory for storing a representation of a set of command grammars corresponding to commands for at least a portion of the plurality of software constructs, at least a portion of which stored representation represent commands currently available for processing;

a speech analyzer adapted for processing the user speech input based on at least a portion of said stored representation, to determine if a user speech input corresponds to a represented command grammar available for processing, and if so processing a corresponding command by the operating system or the respective application to which it relates; and

the graphic user interface being modified in dependence on an output of the speech analyzer.

24. The speech enabled computing system according to claim 23, wherein:

(i) if the speech input corresponds to a representation of a command and satisfies conditions for processing thereof, commencing processing of the command by the operating system or a respective application to which the command relates; and

(ii) if the speech input corresponds to a representation of a command, wherein said input does not unambiguously satisfy conditions for processing the command, generating a set of information adapted to generate a prompt for the user indicating further input needed to satisfy unambiguous conditions for processing.