Apparatus and method for predicting user's intention based on multimodal information

• US 8,606,735 B2
• Filed: 04/29/2010
• Issued: 12/10/2013
• Est. Priority Date: 04/30/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Jeong-Su, Bang, Won-Chul, Kim, Nam-Hoon, Cho, Jeong-Mi
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Chubb, Mikayla
• Application Number:
  US12/770,168
• Publication Number:
  US 20100280983A1
• Time in Patent Office: 1,321 Days
• Field of Search: None
• US Class Current: 706/46
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Disclosed are an apparatus and method of deducing a user'"'"'s intention using multimodal information. The user'"'"'s intention deduction apparatus includes a first predictor to predict a part of a user'"'"'s intention using at least one piece of motion information, and a second predictor to predict the user'"'"'s intention using the predicted part of the user'"'"'s intention and multimodal information received from at least one multimodal sensor.

Claims

1. A user'"'"'s intention deduction apparatus, comprising:

a detector configured to generate motion information based upon detected motion of a user;

a first predictor configured to predict a part of the user'"'"'s intention using the motion information; and

a second predictor configured to predict the user'"'"'s intention using the part of the user'"'"'s intention predicted by the first predictor and multimodal information received from at least one multimodal sensor, wherein

the first predictor predicts the part of the user'"'"'s intention to be the selection of an object displayed by a screen based upon an angle and a distance between the user'"'"'s two hands.

2. The user'"'"'s intention deduction apparatus of claim 1, wherein the first predictor is further configured to generate a control signal for executing an operation to be performed when predicting the user'"'"'s intention using the predicted part of the user'"'"'s intention.

3. The user'"'"'s intention deduction apparatus of claim 2, wherein the control signal is configured to control operation of the multimodal sensor which is controlled by the user'"'"'s intention deduction apparatus.

4. The user'"'"'s intention deduction apparatus of claim 1, wherein the second predictor is further configured to interpret the multimodal information received from the multimodal sensor in association with the predicted part of the user'"'"'s intention to predict the user'"'"'s intention.

5. The user'"'"'s intention deduction apparatus of claim 4, wherein, in response to the predicted part of the user'"'"'s intention being an intention of selecting an object displayed on a display screen and voice being input from the multimodal sensor, the second predictor is further configured to interpret the voice in association with the intention of selecting the object to predict the user'"'"'s intention.

6. The user'"'"'s intention deduction apparatus of claim 1, wherein the second predictor is further configured to predict the user'"'"'s intention using the multimodal information received from the at least one multimodal sensor, within a range of the predicted part of the user'"'"'s intention.

7. The user'"'"'s intention deduction apparatus of claim 6, wherein, in response to the predicted part of the user'"'"'s intention being an operation of moving a microphone close to the user'"'"'s mouth, the second predictor is further configured to:

sense a sound signal; and

extract and analyze characteristics of the sound signal to predict the user'"'"'s intention.

8. The user'"'"'s intention deduction apparatus of claim 7, wherein the second predictor is further configured to:

detect a voice section from the sound signal; and

predict, in response to detecting the voice section from the sound signal, the user'"'"'s intention as a voice instruction input.

9. The user'"'"'s intention deduction apparatus of claim 8, wherein the second predictor is further configured to predict, in response to detecting breath sound from the sound signal, the user'"'"'s intention as a blow.

10. The user'"'"'s intention deduction apparatus of claim 1, wherein, in response to the predicted part of the user'"'"'s intention being to select an object displayed on a display screen, the second predictor is further configured to predict the user'"'"'s intention as at least one of deletion, classification, and arrangement of the selected object using the multimodal information.

11. The user'"'"'s intention deduction apparatus of claim 1, further comprising a user'"'"'s intention applying unit configured to control software or hardware that is under control of the user'"'"'s intention deduction apparatus using the prediction of the user'"'"'s intention.

18. The apparatus of claim 11, wherein the first predictor further predicts the part of the user'"'"'s intention to be the selection of the object displayed by the screen based upon an angle and a distance between the user'"'"'s face and two hands.

12. A user'"'"'s intention deduction method, comprising:

receiving motion information;

predicting a part of a user'"'"'s intention using the received motion information;

receiving multimodal information from at least one multimodal sensor; and

predicting the user'"'"'s intention using the predicted part of the user'"'"'s intention and the multimodal information, wherein

the part of the user'"'"'s intention is predicted to be the selection of an object displayed by a screen based upon an angle and a distance between the user'"'"'s two hands.

13. The user'"'"'s intention deduction method of claim 12, further comprising generating a control signal for executing an operation to be performed when predicting the user'"'"'s intention using the predicted part of the user'"'"'s intention.

14. The user'"'"'s intention deduction method of claim 13, wherein the control signal controls the operation of the multimodal sensor.

15. The user'"'"'s intention deduction method of claim 12, wherein the predicting of the user'"'"'s intention comprises interpreting the multimodal information received from the multimodal sensor in association with the predicted part of the user'"'"'s intention.

16. The user'"'"'s intention deduction method of claim 12, wherein the user'"'"'s intention is predicted using the multimodal information received from the at least one multimodal sensor, within a range of the predicted part of the user'"'"'s intention.

17. The user'"'"'s intention deduction method of claim 12, further comprising controlling software or hardware using the prediction of the user'"'"'s intention.

19. The method of claim 12, wherein the part of the user'"'"'s intention is further predicted to be the selection of the object displayed by the screen based upon an angle and a distance between the user'"'"'s face and two hands.