System and method for incorporating gesture and voice recognition into a single system

• US 8,914,163 B2
• Filed: 05/01/2014
• Issued: 12/16/2014
• Est. Priority Date: 02/26/2013
• Status: Active Grant

• Current Assignee: Edge Technologies, Inc, Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Edge Technologies, Inc, Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Vaghefinazari, Pedram, Yamamoto, Stuart Masakazu, El Dokor, Tarek, King, Josh Tyler
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Dunn, Alex C
• Application Number:
  US14/267,690
• Publication Number:
  US 20140244072A1
• Time in Patent Office: 229 Days
• Field of Search: 701/2
• US Class Current: 701/2
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/23 using reflectors
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60R 16/0231 Circuits relating to the driving or the functioning of the vehicle monitoring tyres B60C23/00; indicating overspeed B60K31/00; for dash boards B60K37/00, B60Q3/10; for indicating emergencies B60Q1/52; brake control systems B60T; registering or indicating the working of vehicles G07C5/00; measuring distance G01C, e.g. combinations of speed and distance G01C23/00; engine indicators G01L; measuring speed or acceleration G01P
• CPC Class Codes: G01C 21/3608 using speech input, e.g. using speech recognition
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A system and method for combining two separate types of human machine interfaces, e.g., a voice signal and a gesture signal, performing voice recognition to a voice signal and gesture recognition to the gesture signal. Based on a confidence determination using the voice recognition result and the gesture recognition result the system can, for example, immediately perform the command/request, request confirmation of the command/request or determine that the command/request was not identified.

Claims

1. A computer based method for executing a command in a vehicle, comprising the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;

performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;

determining a selected command, said selected command being one of the first command or the second command, based upon the first and second VR scores and said first and second GR scores; and

executing, in a vehicle, said selected command.

2. The method of claim 1, wherein said step of determining a selected command comprises:

generating a first combined recognition score for said first command based upon said first VR score and said first GR score;

generating a second combined recognition score for said second command based upon said second VR score and said second GR score; and

identifying said selected command based on said first and second combined recognition scores.

3. The method of claim 2, wherein said step of identifying said selected command comprises

identifying said first command as said selected command if said first combined recognition score exceeds said second combined recognition score;

and

identifying said second command as said selected command if said second combined recognition score exceeds said first combined recognition score.

4. A computer based method for executing a command in a vehicle, comprising the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;

performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;

determining a selected command, said selected command being one of the first command or the second command, based upon said first VR score and said second GR score; and

executing, in a vehicle, said selected command.

5. The method of claim 4, wherein said step of determining a selected command comprises:

generating a query for asking which of said first and second command to perform when said first VR score exceeds a first threshold and said second GR score exceeds a second threshold.

6. The method of claim 4, wherein said query is at least one of an audible request and a visual query.

7. The method of claim 4, wherein said step of determining a selected command comprises:

outputting to a user interface, a confirmation request that said first command should be executed, prior to executing said first command, when said first VR score exceeds a first threshold and said second GR score does not exceeds a second threshold.

8. The method of claim 7, wherein said confirmation request is at least one of an audible request and a visual request.

9. The method of claim 4, wherein said step of determining a selected command comprises:

outputting to a user interface, a confirmation request that said second command should be executed, prior to executing said second command, when said first VR score does not exceed a first threshold and said second GR score exceeds a second threshold.

10. The method of claim 9, wherein said confirmation request is at least one of an audible request and a visual request.

11. A non-transitory computer readable medium comprising computer executable code which when executed by a computer performs the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;

performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;

determining a selected command, said selected command being one of the first command or the second command, based upon the first and second VR scores and said first and second GR scores; and

executing, in a vehicle, said selected command.

12. The non-transitory computer readable medium of claim 11, wherein said step of determining a selected command comprises:

generating a first combined recognition score for said first command based upon said first VR score and said first GR score;

generating a second combined recognition score for said second command based upon said second VR score and said second GR score; and

identifying said selected command based on said first and second combined recognition scores.

13. The non-transitory computer readable medium of claim 12, wherein said step of identifying said selected command comprises

identifying said first command as said selected command if said first combined recognition score exceeds said second combined recognition score;

and

identifying said second command as said selected command if said second combined recognition score exceeds said first combined recognition score.

14. A non-transitory computer readable medium comprising computer executable code which when executed by a computer performs the steps of:

receiving a voice signal;

receiving a gesture signal;

performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;

performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;

determining a selected command, said selected command being one of the first command or the second command, based upon said first VR score and said second GR score; and

executing, in a vehicle, said selected command.

15. The non-transitory computer readable medium of claim 14, wherein said step of determining a selected command comprises:

generating a query for asking which of said first and second command to perform when said first VR score exceeds a first threshold and said second GR score exceeds a second threshold.

16. The non-transitory computer readable medium of claim 14, wherein said query is at least one of an audible request and a visual query.

17. The non-transitory computer readable medium of claim 14, wherein said step of determining a selected command comprises:

outputting to a user interface, a confirmation request that said first command should be executed, prior to executing said first command, when said first VR score exceeds a first threshold and said second GR score does not exceeds a second threshold.

18. The non-transitory computer readable medium of claim 17, wherein said confirmation request is at least one of an audible request and a visual request.

19. The non-transitory computer readable medium of claim 14, wherein said step of determining a selected command comprises:

outputting to a user interface, a confirmation request that said second command should be executed, prior to executing said second command, when said first VR score does not exceed a first threshold and said second GR score exceeds a second threshold.

20. The non-transitory computer readable medium of claim 19, wherein said confirmation request is at least one of an audible request and a visual request.