SYSTEM AND METHOD FOR MULTIMODAL HUMAN-VEHICLE INTERACTION AND BELIEF TRACKING

• US 20140361973A1
• Filed: 07/03/2013
• Published: 12/11/2014
• Est. Priority Date: 06/06/2013
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company), Capio, Inc. (Twilio, Inc.)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company), Ian Lane
• Inventors: Raux, Antoine, Gupta, Rakesh, Lane, Ian
• Granted Patent: US 9,286,029 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

A method and system for multimodal human-vehicle interaction including receiving input from an occupant in a vehicle via more than one mode and performing multimodal recognition of the input. The method also includes augmenting at least one recognition hypothesis based on at least one visual point of interest and determining a belief state of the occupant'"'"'s intent based on the recognition hypothesis. The method further includes selecting an action to take based on the determined belief state.

Claims

1. A method for multimodal human-vehicle interaction, comprising:

receiving input from an occupant in a vehicle via more than one mode;

performing multimodal recognition of the input;

augmenting at least one recognition hypothesis based on at least one visual point of interest;

determining a belief state of the occupant'"'"'s intent based on the recognition hypothesis; and

selecting an action to take based on the determined belief state.

2. The method of claim 1, wherein the input includes a speech input and a gesture input.

3. The method of claim 2, wherein performing multimodal recognition includes speech recognition of the speech input and gesture recognition of the gesture input.

4. The method of claim 3, wherein the speech recognition includes determining a reference to a point of interest in the speech input and whether the point of interest is present in a current dialog.

5. The method of claim 4, including extracting the visual point of interest from the gesture input based on the reference to the point of interest in the speech input.

6. The method of claim 1, including extracting the visual point of interest based on the input and a location of the vehicle, the input including a speech input and a gesture input.

7. The method of claim 6, wherein the gesture input includes at least an eye gaze of the occupant.

8. The method of claim 1, wherein the belief state is determined based on probabilistic ontology trees.

9. A method for multimodal human-vehicle interaction, comprising:

receiving an input from an occupant of a vehicle including a first input and a second input, wherein the first and second inputs represent different modalities;

performing multimodal recognition of the first and second inputs;

modifying a recognition hypothesis of the first input with the second input;

determining a belief state of the occupant'"'"'s intent; and

selecting an action to take based on the determined belief state.

10. The method of claim 9, including determining a reference to a point of interest in the first input.

11. The method of claim 10, including extracting a visual point of interest from the second input based on the reference, wherein the visual point of interest includes a location of the vehicle.

12. The method of claim 11, wherein the recognition hypothesis is modified based on the visual point of interest.

13. The method of claim 11, including determining whether the reference to the visual point of interest is within a current dialog.

14. The method of claim 9, wherein the belief state is determined based on probabilistic ontology trees.

15. A system for multimodal human-vehicle interaction, comprising:

a plurality of sensors for sensing interaction data from a vehicle occupant, the interaction data representing more than one mode;

a multimodal recognition module for performing multimodal recognition of the interaction data and augmenting a recognition hypothesis based on a visual point of interest;

a belief tracking module for determining a belief state of the occupant'"'"'s intent based on the recognition; and

a dialog management and action module for selecting an action to take based on the determined belief state.

16. The system of claim 15, wherein the interaction data includes a speech input and a gesture input.

17. The system of claim 16, including a point of interest identification module for determining a reference to a point of interest in the speech input.

18. The system of claim 17, including a point of interest history database wherein the point of interest identification module utilizes the database to determine whether the reference to the point of interest is within a current dialog.

19. The system of claim 17, including extracting a visual point of interest from the gesture input based on the reference, wherein the visual point of interest includes a location of the vehicle.

20. The system of claim 15, wherein the belief tracking module includes probabilistic ontology trees.