Multimodal interface

• US 9,292,112 B2
• Filed: 07/06/2012
• Issued: 03/22/2016
• Est. Priority Date: 07/28/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Madhvanath, Sriganesh, Freeman, Dustin
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US13/543,419
• Publication Number:
  US 20130030815A1
• Time in Patent Office: 1,355 Days
• Field of Search: 704/270, 704/275, 715/727, 715/810, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

Provided is a multimodal graphical user interface. The multimodal graphical user interface includes a menu with at least one menu item, wherein the at least one menu item is displayed as command name along with a unique hand shape, wherein the at least one menu item is configured to receive a combination of cursor and selection gesture input.

Claims

1. A system comprising:

a processor; and

a memory storing program instructions, which when executed by the processor cause the system to generate a multimodal graphical user interface, the multimodal graphical user interface comprising a display of a menu of a plurality of different selectable menu items, wherein the display of each of the plurality of different selectable menu items includes a command name and a unique hand shape corresponding to the command name,

wherein one of the plurality of different selectable menu items is selected when the menu item receives a combination of a cursor input and a selection gesture input, wherein the selection gesture input includes an air-tap gesture.

2. The system of claim 1, wherein each of the plurality of different selectable menu items is selected in response to a speech input.

3. The system of claim 2, wherein the speech input includes the command name.

4. The system of claim 1, wherein each of the plurality of different selectable menu items is selected in response to a hand shape gesture input.

5. The system of claim 4, wherein the hand shape gesture input corresponds to the unique hand shape in the menu.

6. The system of claim 1, wherein the selection gesture input further includes a snap of fingers.

7. The system of claim 1, wherein the one of the plurality of different selectable menu items further includes at least one sub-menu item, wherein the at least one sub-menu item is displayed as another command name and another unique hand shape, wherein the at least one sub-menu item is selected in response to a combination of the cursor input and the air-tap gesture input.

8. The system of claim 7, wherein the at least one sub-menu item is selected in response to a speech input.

9. The system of claim 8, wherein the speech input includes the command name.

10. The system of claim 7, wherein the at least one sub-menu item is selected in response to a hand shape gesture input.

11. The system of claim 10, wherein the hand shape gesture input corresponds to the another unique hand shape in the sub-menu item.

12. The system of claim 1, wherein the plurality of different selectable menu items is displayed in at least one of the following layouts:

radial, vertical and horizontal.

13. A method for allowing a user to interface with a computing device, comprising:

displaying, with a display device, a multimodal graphical user interface, wherein the multimodal graphical user interface comprises a display of a menu of a plurality of different selectable menu items, wherein the display of each of the plurality of different selectable menu items includes a command name and a unique hand shape corresponding to the command name;

accepting, by the computing device, a combination of a cursor input and an air-tap gesture input for selecting one of the plurality of different selectable menu items, wherein the cursor input comprises a user moving a cursor to the selected menu item and the air-tap gesture input comprises the user tapping the air subsequent to aforesaid cursor movement; and

performing, by the computing device, a function associated with the selected menu item.

14. The method of claim 13, further comprising:

accepting, by the computing device, a combination of the cursor input and a snap of fingers for selecting the one of the plurality of different selectable menu items.

15. A method for allowing a user to interface with a computing device, comprising:

displaying, with a display device, a multimodal graphical user interface, wherein the multimodal graphical user interface comprise a display of a menu of a plurality of different selectable menu items, wherein the display of each of the plurality of different selectable menu items includes a command name and a unique hand shape corresponding to the command name;

accepting, by the computing device, a combination of a cursor input and another input for selecting one of the plurality of different selectable menu items, wherein the another input comprises an air-tap gesture; and

performing, by the computing device, a function associated with the selected menu item.

16. The method of claim 15, wherein the another input further includes a hand shape gesture input corresponding to the unique hand shape in the plurality of different selectable menu items.

17. The method of claim 15, wherein the another input further includes a snap of fingers.