Adapting a user interface for voice control

• US 7,246,063 B2
• Filed: 06/28/2002
• Issued: 07/17/2007
• Est. Priority Date: 02/15/2002
• Status: Expired due to Term

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: James, Frankie, Swan, Richard, Gurram, Rama, Roelands, Jeff
• Primary Examiner(s): ARMSTRONG, ANGELA A
• Application Number:
  US10/184,072
• Publication Number:
  US 20030156130A1
• Time in Patent Office: 1,845 Days
• Field of Search: 704/270.1, 704/258, 704/235, 704/275, 704/257, 704/270, 704/251, 715/513, 345/746
• US Class Current: 704/270.1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Method and systems to voice-enable a user interface using a voice extension module are provided. A voice extension module includes a preprocessor, a speech recognition engine, and an input handler. The voice extension module receives user interface information, such as, a hypertext markup language (HTML) document, and voice-enables the document so that a user may interact with any user interface elements using voice commands.

Claims

1. A method to voice-enable a user interface using a voice extension module comprising:

receiving user interface information;

parsing the user interface information to determine user interface elements, the user interface elements being configured to accept user input or control;

processing the user interface elements to enable voice control;

receiving a voice control to identify a user interface element from among the user interface elements, the user interface element to be a focus for future voice input;

displaying visual feedback identifying the user interface element as a focus for future voice input in response to the voice control, wherein the visual feedback identifying the user interface element is associated positionally with and is supplemental to the user interface element;

receiving voice input directed to the user interface element; and

displaying visual feedback corresponding to the voice input, wherein the visual feedback corresponding to the voice input is associated positionally with and is supplemental to the user interface element.

2. The method of claim 1 wherein the user interface information is a hypertext markup language (HTML) document.

3. The method of claim 2 wherein the HTML document includes code to enable dynamic content.

4. The method of claim 1 wherein user interface elements include one or more from the group consisting of:

a text field;

a selection list;

a scroll bar;

a slider;

a radio button;

a checkbox;

an icon; and

a button.

5. The method of claim 1 wherein processing the user interface elements includes:

for each user interface element, registering the user interface element with a speech recognition engine.

6. The method of claim 5 wherein registering the user interface element with a speech recognition engine includes registering the name of the user interface element.

7. The method of claim 5 wherein registering the user interface element with a speech recognition engine includes registering the type of the user interface element.

8. The method of claim 5 wherein processing the user interface elements further comprises enabling a user to specify a grammar to be associated with at least one of the user interface elements.

9. The method of claim 1 further comprising receiving input corresponding to the user interface elements.

10. The method of claim 9 wherein the user input is voice input recognized by a speech recognition engine.

11. The method of claim 9 wherein the input includes data obtained from one or more of the following devices:

a keyboard;

a mouse; and

a microphone.

12. The method of claim 1 wherein displaying visual feedback further comprises visually highlighting at least one of the user interface elements based on a correspondence to the voice control.

13. The method of claim 12 wherein visually highlighting at least one of the user interface elements comprises visually highlighting at least one of the user interface elements with a colored border.

14. The method of claim 12 wherein visually highlighting at least one of the user interface elements comprises visually highlighting at least one of the user interface elements with an overlaid label.

15. The method of claim 1 wherein displaying visual feedback comprises altering the user interface information to display the visual feedback.

16. A computer readable medium encoded with a voice extension module for voice-enabling a user interface wherein the voice extension module comprises:

a speech recognition engine;

a preprocessor that receives user interface information, registers corresponding voice commands with the speech recognition engine, and enables visual feedback; and

an input handler configured to;

receive a voice control to identify a user interface element to be a focus for future voice input and to communicate with the preprocessor in order to display visual feedback identifying the user interface element as a focus for future voice input in response to the voice control, wherein the visual feedback identifying the user interface element is associated positionally with and is supplemental to the user interface element, and,

receive voice input directed to the user interface element and to communicate with the preprocessor in order to display visual feedback corresponding to the voice input, wherein the visual feedback corresponding to the voice input is associated positionally with and is supplemental to the user interface element.

17. The voice extension module of claim 16 wherein the preprocessor comprises:

a parser to identify the user interface element; and

a translator to voice-enable the user interface element.

18. The voice extension module of claim 17 wherein the user interface elements include one or more from the group consisting of:

a text field;

a selection list;

a scroll bar;

a slider;

a radio button;

a checkbox;

an icon; and

a button.

19. The voice extension module of claim 16 wherein the preprocessor enables visual feedback by enabling the user interface element to be highlighted visually based on a correspondence to the voice control.

20. The voice extension module of claim 16 wherein the preprocessor alters the user interface information to enable the visual feedback.

21. The voice extension module of claim 16 wherein the input handler enables a user to specify a grammar to be associated with the user interface element.

22. A system for voice-enabling user interfaces comprising:

a browser that generates a user interface corresponding to user interface information; and

a voice extension module associated with the browser and operable to voice-enable the user interface generated by the browser, the voice extension module including;

a speech recognition engine;

a preprocessor that receives the user interface information, registers corresponding voice commands with the speech recognition engine, and enables visual feedback; and

an input handler configured to;

receive a voice control to identify a user interface element to be a focus for future voice input and to communicate with the preprocessor in order to display visual feedback identifying the user interface element as a focus for future voice input in response to the voice control, wherein the visual feedback identifying the user interface element is associated positionally with and is supplemental to the user interface element, and

receive voice input directed to the user interface element and to communicate with the preprocessor in order to display visual feedback corresponding to the voice input, wherein the visual feedback corresponding to the voice input is associated positionally with and is supplemental to the user interface element.

23. The system of claim 22 wherein the browser is Internet Explorer™

.

24. The system of claim 22 further comprising an output device for presenting the user interface including one or more from the group consisting of:

a monitor;

a speaker; and

a liquid crystal display (LCD) panel.

25. The system of claim 22 wherein the user interface information is a hypertext markup language (HTML) document.

26. The system of claim 25 wherein the preprocessor comprises:

a parser to identify the user interface element; and

a translator to voice-enable the user interface element.

27. The system of claim 26 wherein the user interface elements include one or more from the group consisting of:

a text field;

a selection list;

a scroll bar;

a slider;

a radio button;

a checkbox;

an icon; and

a button.

28. The system of claim 22 wherein the voice extension module is a component of the browser.

29. The system of claim 22 wherein the preprocessor enables visual feedback by enabling the user interface element to be highlighted visually based on a correspondence to the voice control.

30. The system of claim 22 wherein the preprocessor alters the user interface information to enable the visual feedback before the browser generates the user interface corresponding to the user interface information.

31. The system of claim 22 wherein the input handler enables a user to specify a grammar to be associated with the user interface element.