Controlling user interfaces with contextual voice commands

• US 8,788,271 B2
• Filed: 12/22/2004
• Issued: 07/22/2014
• Est. Priority Date: 12/22/2004
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: James, Frances, Gurram, Rama
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Ortiz Sanchez, Michael
• Application Number:
  US11/020,756
• Publication Number:
  US 20060136221A1
• Time in Patent Office: 3,499 Days
• Field of Search: 704/275, 704/270
• US Class Current: 704/275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

One or more voice-enabled user interfaces include a user interface, and a voice extension module associated with the user interface. The voice extension module is configured to voice-enable the user interface and includes a speech recognition engine, a preprocessor, and an input handler. The preprocessor registers with the speech recognition engine one or more voice commands for signaling for execution of one or more semantic operations that may be performed using a first user interface. The input handler receives a first voice command and communicates with the preprocessor to execute a semantic operation that is indicated by the first voice command. The first voice command is one of the voice commands registered with the speech recognition engine by the preprocessor.

Claims

1. A method for enabling a user interface to be controlled with voice commands, the method comprising:

receiving information specifying a first user interface at a client computer system, the first user interface enabling interaction with an existing application executed on an application server and being a standalone interface that is separate from the existing application;

constructing one or more semantic operations by processing the information specifying the first user interface on the client computer system, each of the one or more semantic operations including sub-operations that are executable using user interface elements of the first user interface, the sub-operations comprising identifying a user interface element, selecting the user interface element, and inputting data using the user interface element;

registering one or more voice commands to enable voice control of the first user interface and for initiating execution of the one or more semantic operations, each voice command corresponding to one of the semantic operations, the application executed on the application server remaining unmodified in view of the constructing one or more semantic operations and the registering one or more voice commands;

displaying the first user interface at the client computer system upon registering the one or more voice commands; and

performing one of the semantic operations in response to a first voice command, the first voice command not explicitly referencing the user interface element, wherein performing one of the semantic operations includes automatic execution of each of the sub-operations in response to the first voice command.

2. The method of claim 1 further comprising:

registering one or more additional voice commands to enable switching to a second user interface of a second application, each additional voice command corresponding to a switch to the second user interface; and

performing, in response to a second voice command that is one of the additional voice commands and that corresponds to a switch to the second application, the switch to the second user interface corresponding to the second voice command.

3. The method of claim 1 further comprising:

registering one or more additional voice commands to enable voice control of a second user interface, each additional voice command corresponding to a semantic operation that may be performed with the second user interface; and

performing one of the semantic operations in response to a second voice command, the second voice command being one of the additional voice commands, the performed semantic operation corresponding to the second voice command.

4. The method of claim 1 further comprising:

identifying one or more graphical elements of the first user interface that may be controlled;

registering one or more additional voice commands to enable control of the identified graphical elements of the first user interface, each additional voice command corresponding to at least one of the identified graphical elements; and

controlling one of the identified graphical elements in response to a second voice command, the second voice command being one of the additional voice commands, the controlled graphical element corresponding to the second voice command.

5. The method of claim 1 further comprising prompting for additional information that further specifies the semantic operation that is performed in response to the first voice command.

6. The method of claim 1 further comprising providing feedback indicating that the semantic operation indicated by the first voice command has been performed.

7. The method of claim 1, wherein the information comprises one or more of a hypertext markup language (HTML) document that identifies user interface elements, a control mechanism implemented by a web browser, and metadata that describes the first user interface and functions provided by the first user interface.

8. The method of claim 1, wherein the first user interface is at least one from a group including a hypertext markup language (HTML) document presented in a web browser, a standalone application, and a user interface for a web services application.

9. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving information specifying a first user interface at a client computer system, the first user interface enabling interaction with an existing application executed on an application server and being a standalone interface that is separate from the existing application;

constructing one or more semantic operations by processing the information specifying the first user interface on the client computer system, each of the one or more semantic operations including sub-operations that are executable using user interface elements of the first user interface, the sub-operations comprising identifying a user interface element, selecting the user interface element, and inputting data using the user interface element;

registering one or more voice commands to enable voice control of the first user interface and for initiating execution of the one or more semantic operations, each voice command corresponding to one of the semantic operations, the application executed on the application server remaining unmodified in view of the constructing one or more semantic operations and the registering one or more voice commands;

displaying the first user interface at the client computer system upon registering the one or more voice commands; and

performing one of the semantic operations in response to a first voice command, the first voice command not explicitly referencing the user interface element, wherein performing one of the semantic operations includes automatic execution of each of the sub-operations in response to the first voice command.

10. The computer storage medium of claim 9, wherein the operations further comprise:

registering one or more additional voice commands to enable switching to a second user interface of a second application, each additional voice command corresponding to a switch to the second user interface; and

performing, in response to a second voice command that is one of the additional voice commands and that corresponds to a switch to the second application, the switch to the second user interface corresponding to the second voice command.

11. The computer storage medium of claim 9, wherein the operations further comprise:

registering one or more additional voice commands to enable voice control of a second user interface, each additional voice command corresponding to a semantic operation that may be performed with the second user interface; and

performing one of the semantic operations in response to a second voice command, the second voice command being one of the additional voice commands, the performed semantic operation corresponding to the second voice command.

12. The computer storage medium of claim 9, wherein the operations further comprise:

identifying one or more graphical elements of the first user interface that may be controlled;

registering one or more additional voice commands to enable control of the identified graphical elements of the first user interface, each additional voice command corresponding to at least one of the identified graphical elements; and

controlling one of the identified graphical elements in response to a second voice command, the second voice command being one of the additional voice commands, the controlled graphical element corresponding to the second voice command.

13. The computer storage medium of claim 9, wherein the operations further comprise prompting for additional information that further specifies the semantic operation that is performed in response to the first voice command.

14. The computer storage medium of claim 9, wherein the operations further comprise providing feedback indicating that the semantic operation indicated by the first voice command has been performed.

15. The computer storage medium of claim 9, wherein the information comprises one or more of a hypertext markup language (HTML) document that identifies user interface elements, a control mechanism implemented by a web browser, and metadata that describes the first user interface and functions provided by the first user interface.

16. The computer storage medium of claim 9, wherein the first user interface is at least one from a group including a hypertext markup language (HTML) document presented in a web browser, a standalone application, and a user interface for a web services application.

17. A system, comprising:

a display; and

a computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving information specifying a first user interface at a client computer system, the first user interface enabling interaction with an existing application executed on an application server and being a standalone interface that is separate from the existing application;

constructing one or more semantic operations by processing the information specifying the first user interface on the client computer system, each of the one or more semantic operations including sub-operations that are executable using user interface elements of the first user interface, the sub-operations comprising identifying a user interface element, selecting the user interface element, and inputting data using the user interface element;

registering one or more voice commands to enable voice control of the first user interface and for initiating execution of the one or more semantic operations, each voice command corresponding to one of the semantic operations, the application executed on the application server remaining unmodified in view of the constructing one or more semantic operations and the registering one or more voice commands;

displaying the first user interface at the client computer system upon registering the one or more voice commands; and

performing one of the semantic operations in response to a first voice command, the first voice command not explicitly referencing the user interface element, wherein performing one of the semantic operations includes automatic execution of each of the sub-operations in response to the first voice command.

18. The system of claim 17, wherein the operations further comprise:

registering one or more additional voice commands to enable switching to a second user interface of a second application, each additional voice command corresponding to a switch to the second user interface; and

performing, in response to a second voice command that is one of the additional voice commands and that corresponds to a switch to the second application, the switch to the second user interface corresponding to the second voice command.

19. The system of claim 17, wherein the operations further comprise:

registering one or more additional voice commands to enable voice control of a second user interface, each additional voice command corresponding to a semantic operation that may be performed with the second user interface; and

performing one of the semantic operations in response to a second voice command, the second voice command being one of the additional voice commands, the performed semantic operation corresponding to the second voice command.

20. The system of claim 17, wherein the operations further comprise:

identifying one or more graphical elements of the first user interface that may be controlled;

registering one or more additional voice commands to enable control of the identified graphical elements of the first user interface, each additional voice command corresponding to at least one of the identified graphical elements; and

controlling one of the identified graphical elements in response to a second voice command, the second voice command being one of the additional voice commands, the controlled graphical element corresponding to the second voice command.