Systems and methods of processing annotations and multimodal user inputs

• US 20070124507A1
• Filed: 11/28/2005
• Published: 05/31/2007
• Est. Priority Date: 11/28/2005
• Status: Abandoned Application

• Current Assignee: SAP AG (SAP SE)
• Original Assignee: SAP AG (SAP SE)
• Inventors: James, Frankie, Gurram, Rama
• Application Number:
  US11/287,850
• Publication Number:
  US 20070124507A1
• Time in Patent Office: Days
• US Class Current: 710/1
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Embodiments of the present invention provide multimodal input capability. In one embodiment the present invention includes an input method comprising displaying one or more display objects to a user, associating at least one voice mode with one of said display objects, associating at least one stylus mode with the display object, and associating at least one voice navigation command with the display object. The system may prompt a user for a plurality of inputs, receive a voice command or a touch screen command specifying one of the plurality of inputs, activate a voice and touch screen mode associated with the specified input, and process the voice input in accordance with the associated voice mode or the associated touch screen mode.

Claims

1. A computer-implemented method for processing user inputs comprising:

prompting a user for a plurality of inputs;

receiving a command specifying one of the plurality of inputs, wherein the system is activated to receive both a voice command and a manual selection command;

activating a voice and manual selection mode associated with the specified input; and

if a voice input is detected, processing the voice input in accordance with the associated voice mode, or if a manual selection input is detected, processing the touch screen input in accordance with the associated manual selection mode.

2. The method of claim 1 wherein the plurality of inputs are display objects each having an associated voice command, voice mode, and touch screen mode.

3. The method of claim 2 further comprising storing metadata for defining associations between display objects and voice commands, voice modes, and touch screen modes.

4. The method of claim 2 wherein the display objects include a page, a section of a page, a particular field of a page, an image, a button, a radio button, a check box, a menu, a list, an icon, a link, a table, a slider, a scroll bar, an user interface control, or a step of a program that is illustrated graphically on a screen.

5. The method of claim 1 wherein the voice mode is a short text entry mode for translating a voice input into text and inserting the text into a field.

6. The method of claim 1 wherein the voice mode is a free form dictation mode for translating voice dictations into text.

7. The method of claim 1 wherein the voice mode is voice annotation mode for associating a voice input with a particular display object.

8. The method of claim 1 wherein the voice mode is a voice authorization mode for performing an authorization using a received input.

9. A computer-implemented method for processing user inputs comprising:

displaying one or more display objects to a user;

associating at least one voice mode with one of said display objects;

associating at least one touch screen mode with the display object; and

associating at least one voice command with the display object.

10. The method of claim 9 further comprising receiving a voice command or a touch screen command specifying one of the display objects, and in accordance therewith, activating a voice and touch screen mode associated with the specified input.

11. The method of claim 10 further comprising detecting a voice input or touch screen input, wherein if a voice input is detected, processing the voice input in accordance with an associated voice mode, or if a touch screen input is detected, processing the touch screen input in accordance with an associated touch screen mode.

12. The method of claim 9 wherein the voice mode translates a voice input into text.

13. The method of claim 9 wherein the voice mode associates an annotation with the display object.

14. The method of claim 9 wherein the voice mode performs an authorization.

15. The method of claim 9 wherein the display object is an element of a screen displayed to a user by a computer system.

16. The method of claim 9 wherein the display object is an application page or element of a page displayed to a user by an application.

17. The method of claim 9 wherein the display objects include a page, a section of a page, a particular field of a page, an image, a button, a radio button, a drop down menu, an icon, a link, or a step of a program that is illustrated graphically on a screen.

18. The method of claim 9 wherein the display objects include a web page.

19. A computer system including software for processing user inputs, the software comprising:

an annotation component for associating voice or touch screen inputs with particular objects in a display;

an input controller for selecting between voice and touch screen inputs;

a speech recognition component for receiving grammars and voice inputs and providing recognition results; and

metadata for specifying said grammars and said associations of voice or touch screen inputs with particular objects in a display.

20. The computer system of claim 19 further comprising an association model for defining the association between voice and touch screen inputs with particular objects in a display.

21. The computer system of claim 19 further comprising an authorization component for performing an authorization using a received input.

22. The computer system of claim 19 wherein the objects in the display include a page, a section of a page, a particular field of a page, an image, a button, a radio button, a drop down menu, an icon, a link, or a step of a program that is illustrated graphically on a screen.

23. The computer system of claim 19 wherein the system is a client system that downloads pages over a network, and wherein the pages include said metadata.

24. The computer system of claim 23 wherein said metadata further defines associations between objects in the display and voice commands, voice modes, and touch screen modes.

25. A computer-readable medium containing instructions for controlling a computer system to perform a method of processing user inputs comprising:

displaying a plurality of display objects;

receiving a command specifying one of the plurality of display objects, wherein the command is a voice command or a touch screen command;

activating a voice and touch screen mode associated with the specified display object; and

if a voice input is detected, processing the voice input in accordance with the associated voice mode, or if a touch screen input is detected, processing the touch screen input in accordance with the associated touch screen mode.

26. The computer-readable medium of claim 25 wherein the method further comprises storing metadata for defining associations between display objects and voice commands, voice modes, and touch screen modes.

27. The computer-readable medium of claim 25 wherein the voice mode translates a voice input into text.

28. The computer-readable medium of claim 25 wherein the voice mode associates an annotation with the display object.

29. The computer-readable medium of claim 25 wherein the voice mode performs an authorization.

30. A computer-readable medium containing instructions for controlling a computer system to perform a method of processing user inputs comprising:

displaying one or more display objects to a user;

associating at least one voice mode with one of said display objects;

associating at least one touch screen mode with the display object; and

associating at least one voice command with the display object.

31. The computer-readable medium of claim 30 wherein the method further comprises:

receiving a voice command or a touch screen command specifying one of the display objects;

activating a voice and touch screen mode associated with the specified object; and

detecting a voice input or touch screen input,

wherein if a voice input is detected, processing the voice input in accordance with an associated voice mode, or if a touch screen input is detected, processing the touch screen input in accordance with an associated touch screen mode.

32. The computer-readable medium of claim 30 wherein the voice mode translates a voice input into text.

33. The computer-readable medium of claim 30 wherein the voice mode associates an annotation with the display object.

34. The computer-readable medium of claim 30 wherein the voice mode performs an authorization.

35. The computer-readable medium of claim 30 wherein the display objects include a web page.