MULTIMODAL INTERFACE

• US 20160179309A1
• Filed: 02/29/2016
• Published: 06/23/2016
• Est. Priority Date: 07/28/2011
• Status: Abandoned Application

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: SRIGANESH, MADHVANATH, FREEMAN, DUSTIN
• Application Number:
  US15/056,728
• Publication Number:
  US 20160179309A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

A system includes a processor and a multimodal graphical user interface. The multimodal graphical user interface causes the processor to display a menu of a plurality of selectable menu items. Each menu item has an associated command name and an associated unique hand shape that corresponds to the command name. The multimodal graphical user interface causes the processor to select a given menu item of the plurality of selectable menu items based at least in part on a combination of a cursor input and a selection gesture input.

Claims

1. A system comprising:

a processor; and

a multimodal graphical user interface to cause the processor to;

display a menu of a plurality of selectable menu items, wherein each menu item has an associated command name and an associated unique hand shape corresponding to the command name; and

select a given menu item of the plurality of selectable menu items based at least in part on a combination of a cursor input and a selection gesture input.

2. The system of claim 1, wherein the multimodal graphical interface causes the processor to select the given menu item based at least in part on a speech input.

3. The system of claim 2, wherein the speech input corresponds to the command name associated with the given menu item.

4. The system of claim 1, wherein the multimodal graphical user interface causes the processor to select the given menu item in response to a hand shape gesture input.

5. The system of claim 4, wherein the hand shape gesture input corresponds to the unique hand shape associated with the given menu.

6. The system of claim 1, wherein the given menu item of the plurality of selectable menu items includes at least one sub-menu item, wherein the at least one sub-menu item is associated with another command name and another unique hand shape, and wherein the multimodal graphical user interface causes the processor to select a sub-menu item of the at least one sub-menu item in response to a combination of the cursor input and the selection gesture input.

7. The system of claim 6, wherein the multimodal graphical user interface causes the processor to select a given sub-menu of the at least one sub-menu item in response to a speech input.

8. The system of claim 1, wherein the multimodal graphical user interface causes the processor to display the plurality of selectable menu items in at least one of the following layouts:

radial, vertical and horizontal.

12. The system of claim 1, wherein displaying comprises displaying the plurality of selectable menu items in at least one of the following layouts:

radial, vertical and horizontal.

9. A method comprising:

displaying, with a display device, a display associated with a multimodal graphical user interface, wherein the display comprises a menu of a plurality of selectable menu items, and wherein the displaying comprises displaying an associated command name and an associated unique hand shape corresponding to the command name for each menu item of the plurality of selectable menu items;

detecting, by a computing device, a combination of a cursor input and a selection gesture input and associating the detected combination with a given menu item of the plurality of selectable menu items; and

performing, by the computing device, a function associated with the given menu item.

10. The method of claim 9, further comprising detecting by the computing device, the given menu item in response to a hand shape gesture input.

11. The method of claim 10, wherein the hand shape gesture input corresponds to the unique hand shape associated with the given menu item.

13. An article comprising a non-transitory computer readable storage medium storing instructions that when executed by a processor-based system cause the processor-based system to:

display a menu of a plurality of selectable menu items;

detect a cursor input;

detect a selection gesture input; and

in response to a combination of the cursor input and the hand selection gesture representing selection of a given menu item of the plurality of selectable menu items, perform a function associated with the given menu item.

14. The article of claim 13, the storage medium storing instructions that when executed by the processor-based system cause the processor-based system to select a sub-menu item associated with the given menu item.

15. The article of claim 14, the storage medium storing instructions that when executed by the processor-based system cause the processor-based system to select the sub-menu based at least in part on a speech input or a hand shape gesture input.