MULTI-MODAL BEHAVIOR AWARENESS FOR HUMAN NATURAL COMMAND CONTROL

• US 20140007115A1
• Filed: 06/29/2012
• Published: 01/02/2014
• Est. Priority Date: 06/29/2012
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Lu, Ning, Bhowmik, Achintya K.
• Application Number:
  US13/539,107
• Publication Number:
  US 20140007115A1
• Time in Patent Office: Days
• US Class Current: 718/102
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

A computer system is controlled using natural commands in multiple modes. In one example, a method includes receiving a first command in a first modality, receiving a second command in a second modality, determining whether the second command confirms the first command, and executing the first command if the second command confirms the first command.

Claims

1. A method comprising:

receiving a first command in a first modality;

receiving a second command in a second modality;

determining whether the second command confirms the first command; and

executing the first command if the second command confirms the first command.

2. The method of claim 1, wherein the second command is an observed behavior of the user.

3. The method of claim 1, wherein the second command is in response to a visual prompt from the system.

4. The method of claim 1, wherein the second command is in response to an audio prompt from the system.

5. The method of claim 1, wherein the second command is received before the first command.

6. The method of claim 1, wherein the first modality is a spoken command and the second modality is a hand gesture.

7. The method of claim 1, wherein the first modality is a hand gesture and the second modality is a response to a prompt.

8. The method of claim 7, wherein the response to the prompt is a spoken command.

9. The method of claim 1, further comprising:

accessing a list of approved command confirmations after receiving the first command;

comparing the received second command to the accessed list of approved command confirmations; and

executing the first command if the second command is determined to be an approved command confirmation based on the comparison.

10. The method of claim 9, further comprising prompting the user for explicit confirmation of the first command if the second command is not determined to be an approved command confirmation.

11. The method of claim 10, further comprising executing the first command if an explicit confirmation is received from a user in response to the prompt.

12. An article including a non-transitory computer-readable medium having instructions that, when operated on by the computer, cause the computer to perform operations comprising:

receiving a first command in a first modality;

receiving a second command in a second modality;

determining whether the second command confirms the first command; and

executing the first command if the second command confirms the first command.

13. The medium of claim 12, wherein the second command is in response to at least one of a visual and audio prompt from the system.

14. The medium of claim 12, the operations further comprising:

accessing a list of approved command confirmations after receiving the first command;

comparing the received second command to the accessed list of approved command confirmations; and

executing the first command if the second command is determined to be an approved command confirmation based on the comparison.

15. The medium of claim 14, the operations further comprising:

prompting the user for explicit confirmation of the first command if the second command is not determined to be an approved command confirmation; and

executing the first command if an explicit confirmation is received from a user in response to the prompt.

16. An apparatus comprising:

a first monitor to receive a first command in a first modality;

a second monitor to receive a second command in a second modality; and

a processor to determine whether the second command confirms the first command and to execute the first command if the second command confirms the first command.

17. The apparatus of claim 16, wherein the first monitor is coupled to a microphone and wherein the first modality is a spoken command from the user.

18. The apparatus of claim 16, wherein the second monitor is coupled to a camera and wherein the second modality is a visual modality comprising at least one of a gesture, eye tracking, and a hand signal.

19. The apparatus of claim 16, further comprising a display to present a visual prompt to the user in response to the first command, the prompt being to prompt the user to provide the second command.

20. The apparatus of claim 19, wherein the prompt is a question presented to the user on the display.