System and method for incorporating gesture and voice recognition into a single system

• US 8,744,645 B1
• Filed: 02/26/2013
• Issued: 06/03/2014
• Est. Priority Date: 02/26/2013
• Status: Active Grant

• Current Assignee: Edge Technologies, Inc, Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Vaghefinazari, Pedram, Yamamoto, Stuart, El Dokor, Tarek, King, Josh Tyler
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Dunn, Alex C
• Application Number:
  US13/777,255
• Time in Patent Office: 462 Days
• Field of Search: 701/1
• US Class Current: 701/1
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/23 using reflectors
• CPC Class Codes: B60K 2370/146 Input by gesture
• CPC Class Codes: B60K 2370/148 Input by voice
• CPC Class Codes: B60K 2370/23 using reflectors
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60R 16/0231 Circuits relating to the driving or the functioning of the vehicle monitoring tyres B60C23/00; indicating overspeed B60K31/00; for dash boards B60K37/00, B60Q3/10; for indicating emergencies B60Q1/52; brake control systems B60T; registering or indicating the working of vehicles G07C5/00; measuring distance G01C, e.g. combinations of speed and distance G01C23/00; engine indicators G01L; measuring speed or acceleration G01P
• CPC Class Codes: G01C 21/3608 using speech input, e.g. using speech recognition
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A system and method for combining two separate types of human machine interfaces, e.g., a voice signal and a gesture signal, performing voice recognition to a voice signal and gesture recognition to the gesture signal. Based on a confidence determination using the voice recognition result and the gesture recognition result the system can, for example, immediately perform the command/request, request confirmation of the command/request or determine that the command/request was not identified.

Claims

1. A computer based method for executing a command in a vehicle, comprising the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first voice command;

performing a gesture recognition procedure to generate a gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to a first gesture command; and

executing a first command in a vehicle corresponding to said first voice command and first gesture command when said voice recognition score exceeds a first threshold and said gesture recognition score exceeds a second threshold.

2. The method of claim 1, further comprising the step of:

generating a confirmation request, prior to executing said first command, when either (a) said voice recognition score exceeds a first threshold and said gesture recognition score does not exceed a second threshold or (b) said voice recognition score does not exceed a first threshold and said gesture recognition score exceeds a second threshold.

3. The method of claim 2, further comprising the step of:

not executing a command corresponding to said first voice command or said first gesture command when said voice recognition score does not exceed a first threshold and said gesture recognition score does not exceed a second threshold.

4. The method of claim 1, wherein said confirmation request is at least one of an audible request and a visual request.

5. The method of claim 1, further comprising the step of:

not executing a command corresponding to said first voice command or said first gesture command when said voice recognition score does not exceed a first threshold and said gesture recognition score does not exceed a second threshold.

6. A computer based method for executing a command in a vehicle, comprising the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first voice command;

performing a gesture recognition procedure to generate a gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to a first gesture command;

generating a combined recognition score based on a combination of said VR score and said GR score; and

executing a first command in the vehicle corresponding to at least one of said first voice command and said first gesture command when said combined recognition score exceed a first threshold.

7. The method of claim 6, further comprising the step of:

generating a confirmation request, prior to executing said first command, when said combined recognition score does not exceed a second threshold.

8. The method of claim 7, wherein said first and second thresholds are substantially the same.

9. The method of claim 6, wherein said confirmation request is at least one of an audible request and a visual request.

10. The method of claim 6, wherein said voice signal and gesture signal are received in a vehicle.

11. The method of claim 6, wherein said step of performing said gesture recognition procedure is performed in a mobile communication device.

12. The method of claim 11, wherein said mobile communication device is one of a cellular phone, a personal device assistant, a smart phone, a pocket personal computer, a laptop computer, a tablet computer, or a smart watch.

13. A non-transitory computer readable medium comprising computer executable code which when executed by a computer performs the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first voice command;

performing a gesture recognition procedure to generate a gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to a first gesture command;

executing a first command in a vehicle corresponding to said first voice command and first gesture command when said voice recognition score exceeds a first threshold and said gesture recognition score exceeds a second threshold.

14. The computer readable medium of claim 13, further comprising the step of:

generating a confirmation request, prior to executing said first command, when either (a) said voice recognition score exceeds a first threshold and said gesture recognition score does not exceed a second threshold or (b) said voice recognition score does not exceed a first threshold and said gesture recognition score exceeds a second threshold.

15. The computer readable medium of claim 14, further comprising the step of:

not executing a command corresponding to said first voice command or said first gesture command when said voice recognition score does not exceed a first threshold and said gesture recognition score does not exceed a second threshold.

16. The computer readable medium of claim 13, wherein said confirmation request is at least one of an audible request and a visual request.

17. The computer readable medium of claim 13, further comprising the step of:

not executing a command corresponding to said first voice command or said first gesture command when said voice recognition score does not exceed a first threshold and said gesture recognition score does not exceed a second threshold.

18. A non-transitory computer readable medium comprising computer executable code which when executed by a computer performs the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first voice command;

performing a gesture recognition procedure to generate a gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to a first gesture command;

generating a combined recognition score based on a combination of said VR score and said GR score; and

executing a first command in a vehicle corresponding to at least one of said first voice command and said first gesture command when said combined recognition score exceed a first threshold.

19. The computer readable medium of claim 18, further comprising the step of:

generating a confirmation request, prior to executing said first command, when said combined recognition score does not exceed a second threshold.

20. The computer readable medium of claim 18, wherein said confirmation request is at least one of an audible request and a visual request.

21. A system for executing a command in a vehicle comprising:

a microphone positioned in the vehicle, for receiving a voice signal;

a optical sensing device, positioned in the vehicle, for receiving a gesture signal;

a processor for performing a voice recognition procedure to generate a voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first voice command, the processor for performing a gesture recognition procedure to generate a gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to a first gesture command; and

an execution unit positioned in the vehicle, for executing a first command corresponding to said first voice command and first gesture command when said voice recognition score exceeds a first threshold and said gesture recognition score exceeds a second threshold.

22. The system of claim 21, further comprising:

confirmation unit, for generating a confirmation request, prior to executing said first command, when either (a) said voice recognition score exceeds a first threshold and said gesture recognition score does not exceed a second threshold or (b) said voice recognition score does not exceed a first threshold and said gesture recognition score exceeds a second threshold.