MULTIMODAL INTERFACE

• US 20130030815A1
• Filed: 07/06/2012
• Published: 01/31/2013
• Est. Priority Date: 07/28/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Madhvanath, Sriganesh, Freeman, Dustin
• Granted Patent: US 9,292,112 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

Provided is a multimodal graphical user interface. The multimodal graphical user interface includes a menu with at least one menu item, wherein the at least one menu item is displayed as command name along with a unique hand shape, wherein the at least one menu item is configured to receive a combination of cursor and selection gesture input.

Claims

1. A multimodal graphical user interface, comprising:

a menu with at least one menu item, wherein the at least one menu item is displayed as command name along with a unique hand shape, wherein the at least one menu item is configured to receive a combination of cursor and selection gesture input.

2. The multimodal graphical user interface of claim 1, wherein the at least one menu item is configured to receive speech input.

3. The multimodal graphical user interface of claim 2, wherein the speech input includes the command name.

4. The multimodal graphical user interface of claim 1, wherein the at least one menu item is configured to receive hand shape gesture input.

5. The multimodal graphical user interface of claim 4, wherein the hand shape gesture input corresponds to the unique hand shape in the menu item.

6. The multimodal graphical user interface of claim 1, wherein the selection gesture may include at least one of the following:

an air-tap gesture, a snap of the fingers;

a grabbing action and a pulling action.

7. The multimodal graphical user interface of claim 1, wherein the at least one menu item further includes at least one sub-menu item, wherein the at least one sub-menu item is displayed as a command name along with a unique hand shape, wherein the at least one sub-menu item is configured to receive a combination of cursor and air-tap gesture input.

8. The multimodal graphical user interface of claim 7, wherein the at least one sub-menu item is configured to receive speech input.

9. The multimodal graphical user interface of claim 8, wherein the speech input includes the command name.

10. The multimodal graphical user interface of claim 7, wherein the at least one sub-menu item is configured to receive hand shape gesture input.

11. The multimodal graphical user interface of claim 10, wherein the hand shape gesture input corresponds to the unique hand shape in the sub-menu item.

12. The multimodal graphical user interface of claim 1, wherein the at least one menu item comprising the command name along with the unique hand shape is displayed in at least one of the following layouts:

radial, vertical and horizontal.

13. A method for allowing a user to interface with a computing device, comprising:

displaying, with a display device, a multimodal graphical user interface, wherein the multimodal graphical user interface comprises a menu with at least one menu item;

accepting a combination of cursor and air-tap gesture input from a user for selecting the at least one menu item, wherein the cursor input comprises a user moving a cursor to the at least one menu item and the air-tap gesture comprises a user tapping the air subsequent to aforesaid curser movement; and

performing a function associated with the at least one menu item.

14. A method for allowing a user to interface with a computing device, comprising:

displaying, with a display device, a multimodal graphical user interface, wherein the multimodal graphical user interface comprises a menu with at least one menu item and the at least one menu item is configured to receive a combination of cursor input another input;

accepting the another input from a user for selecting the at least one menu item, wherein the another input comprises a speech input or a hand shape gesture input speech, and the another input corresponds to a command name in the at least one menu item; and

performing a function associated with the at least one menu item.

15. The method of claim 14, wherein the hand shape gesture input corresponds to a hand shape in the at least one menu item.