System and method of processing speech within a graphic user interface

• US 7,447,637 B1
• Filed: 03/02/2007
• Issued: 11/04/2008
• Est. Priority Date: 12/23/1998
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Eastern Investments LLC
• Inventors: Grant, Richard, McGregor, Pedro
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US11/681,449
• Time in Patent Office: 613 Days
• Field of Search: 704270-275, 704/251, 704/231, 704/277, 704/260, 704/257
• US Class Current: 704/275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 2015/228 of application context

Abstract

According to the present invention, a method for integrating processes with a multi-faceted human centered interface is provided. The interface is facilitated to implement a hands free, voice driven environment to control processes and applications. A natural language model is used to parse voice initiated commands and data, and to route those voice initiated inputs to the required applications or processes. The use of an intelligent context based parser allows the system to intelligently determine what processes are required to complete a task which is initiated using natural language. A single window environment provides an interface which is comfortable to the user by preventing the occurrence of distracting windows from appearing. The single window has a plurality of facets which allow distinct viewing areas. Each facet has an independent process routing its outputs thereto. As other processes are activated, each facet can reshape itself to bring a new process into one of the viewing areas. All activated processes are executed simultaneously to provide true multitasking.

Claims

1. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus, and

wherein said speech processing module further selectively determines whether a user speech input is associated with a process executing within a facet, and if not displaying an executing process within a facet and otherwise activating the process and displaying said newly executed process within a facet.

2. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, wherein a determination of whether a user speech input is intended as a command is dependent on a recognition of a command activation statement in the user speech input, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

3. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, wherein the application of the predetermined grammar to validate the user speech input command comprises at least applying the user speech input to a grammar derived structure as it is received, and waiting until the grammar derived structure is fulfilled prior to passing the command for processing, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

4. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof, wherein the application of the predetermined grammar to validate the user speech input command comprises at least applying the user speech input to the context based data structure as it is received, and prompting the user to provide required information for validation of the command, and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

5. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech, wherein said speech processing module stores a data stream representing the user speech input, which is then searched for an indication that the user speech input is intended as a command, and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

6. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, wherein said speech processing module stores an indicator of whether a user speech input is intended as a command or intended as data, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

7. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module, wherein said speech processing module comprises a speech to text processor, adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

8. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech, wherein said speech processing module reads data from a registration database comprising a plurality grammars associated with respective available commands, and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present,

and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

9. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, wherein said speech processing module further distinguishes a target command processor for a user speech input intended as a command, and selectively passes a validated command to a respective one of a plurality of command processors in dependence thereon, for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

10. A graphic user interface system, comprising:

a multi-tasking operating system;

a graphic user interface, supported by said multi-tasking operating system, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

a speech processing module adapted for receiving user communications as speech and analyzing a context-based data structure corresponding to the user communications to determine whether a valid command is present, wherein said speech processing module reads data from a registration database comprising a plurality grammars associated with respective available commands, and determines a context based on at least a past user interaction, and in dependence thereon to at least selectively;

determine that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determine that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

11. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

analyzing user speech input in accordance with a context based data structure to determine whether a valid command is present, and in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determining that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus,

said determining further comprising selectively determining whether a user speech input is associated with a process executing within a facet, and if not, displaying an executing process within a facet, and otherwise, activating the process and displaying said process within a facet.

12. A computer readable medium, storing thereon computer readable instructions to cause a processor to execute said instructions for controlling said processor to perform the steps of claim 11.

13. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

analyzing user speech input in accordance with a context based data structure to determine whether a valid command is present, comprising determining whether a user speech input is intended as a command, dependent on a recognition of a command activation statement in the user speech input; and

in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determining that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

14. A computer readable medium, storing thereon computer readable instructions to cause a processor to execute said instructions for controlling said processor to perform the steps of claim 13.

15. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

analyzing user speech input in accordance with a context based data structure to determine whether a valid command is present, and in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof, wherein the application of the predetermined grammar to validate the user speech input command comprises at least applying the user speech input to a grammar derived structure as it is received, and waiting until the grammar derived structure is fulfilled prior to passing the command for processing; and

determining that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

16. A computer readable medium, storing thereon computer readable instructions to cause a processor to execute said instructions for controlling said processor to perform the steps of claim 15.

17. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

analyzing user speech input in accordance with a context based data structure to determine whether a valid command is present, wherein the application of the predetermined grammar to validate the user speech input command comprises at least applying the user speech input to the context based data structure as it is received, and prompting the user to provide required information for validation of the command, and in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof; and

determining that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

18. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

storing a data stream representing the user speech input, and then searching the stored data for an indication that the user speech input is intended as a command, in accordance with a context based data structure to determine whether a valid command is present, and in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof;

determining that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

19. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications, and converting received speech into a text representation;

analyzing user speech input in accordance with a context based data structure to determine whether a valid command is present, and in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof;

determining that a user speech input is intended as a data input, and passing the text representation of user speech input to a process associated with the respective facet having focus.

20. A computer readable medium, storing thereon computer readable instructions to cause a processor to execute said instructions for controlling said processor to perform the steps of claim 19.

21. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

reading data from a registration database comprising a plurality grammars associated with respective available commands;

analyzing user speech input in accordance with a context based data structure to determine whether a valid command is present,

and in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, and pass a validated command for processing thereof;

determining that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.

22. A graphic user interface method, comprising:

providing a multi-tasking operating system having a graphic user interface, supporting a plurality of concurrently active facets, and granting a focus to a respective facet based on a request relating to that facet, said focus indicating a target facet of received user communications;

analyzing user speech input in accordance with a context based data structure to determine whether a valid command is present, and in dependence thereon at least selectively;

determining that a user speech input is intended as a command, apply a predetermined grammar to validate the user speech input command, distinguishing one of a plurality of available command processors for a user speech input intended as a command, and selectively passing a validated command to a respective one of the plurality of available command processors for processing thereof in dependence thereon;

determining that a user speech input is intended as a data input, and pass the user speech input to a process associated with the respective facet having focus.