APPARATUS AND METHOD FOR CONTROLLING MULTI-MODAL HUMAN-MACHINE INTERFACE (HMI)

• US 20140145931A1
• Filed: 08/28/2013
• Published: 05/29/2014
• Est. Priority Date: 11/28/2012
• Status: Abandoned Application

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: HAN, Tae Man, KIM, Jin Woo
• Application Number:
  US14/012,461
• Publication Number:
  US 20140145931A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/149 Instrument input by detecting viewing direction not otherwise provided for
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence

Abstract

Provided are an apparatus and method for controlling a multi-modal human-machine interface (HMI) generating a multi-modal control signal based on the voice information and the gesture information, selecting an object from among at least one object recognized in a direction of an LOS of the user based on the multi-modal control signal, and displaying object-related information associated with the selected object based on the multi-modal control signal.

Claims

1. An apparatus for controlling a multi-modal human-machine interface (HMI), the apparatus comprising:

a voice recognizer to recognize voice information associated with a voice of a user;

a gesture recognizer to recognize gesture information associated with a gesture of the user;

a multi-modal engine unit to generate a multi-modal control signal based on the voice information and the gesture information;

an object selector to select an object from among at least one object recognized in a direction of a line of sight (LOS) of the user based on the multi-modal control signal; and

a display unit to display object-related information about the selected object based on the multi-modal control signal.

2. The apparatus of claim 1, further comprising:

an LOS recognizer to recognize an LOS of the user.

3. The apparatus of claim 2, wherein the LOS recognizer calculates a focal distance at which the user gazes at the selected object, based on a movement speed of the selected object.

4. The apparatus of claim 2, wherein the LOS recognizer calculates a focal distance at which the user gazes at the selected object, based on a distance between the selected object and a vehicle being driven by the user.

5. The apparatus of claim 1, further comprising:

an object recognizer to recognize an object located ahead of a vehicle being driven by the user; and

a lane recognizer to recognize a lane in which a vehicle corresponding to the object is traveling.

6. The apparatus of claim 1, further comprising:

a user experience (UX) analyzer to analyze UX information by collecting the multi-modal control signal.

7. The apparatus of claim 6, wherein the multi-modal engine unit generates the multi-modal control signal to select and move the object based on the UX information.

8. The apparatus of claim 1, wherein the object selector selects the object corresponding to the gesture information at a point in time at which the voice information is recognized.

9. The apparatus of claim 1, wherein the display unit displays the object-related information using an augmented reality method.

10. The apparatus of claim 1, wherein the object-related information comprises at least one of a distance between a vehicle being driven by the user and the selected object, a movement speed of the selected object, and a lane in which a vehicle corresponding to the object is traveling.

11. A method of controlling a multi-modal human-machine interface (HMI), the method comprising:

recognizing voice information associated with a voice of a user;

recognizing gesture information associated with a gesture of the user;

generating a multi-modal control signal based on the voice information and the gesture information;

selecting an object from among at least one object recognized in a direction of a line of sight (LOS) of the user based on the multi-modal control signal; and

displaying object-related information associated with the selected object based on the multi-modal control signal.

12. The method of claim 11, further comprising:

recognizing an LOS of the user.

13. The method of claim 12, wherein the recognizing comprises calculating a focal distance at which the user gazes at the selected object, based on a movement speed of the selected object.

14. The method of claim 12, wherein the recognizing comprises a calculating a focal distance at which the user gazes at the selected object, based on a distance between the selected object a vehicle being driven by the user.

15. The method of claim 11, further comprising:

a recognizing an object located ahead of a vehicle being driven by the user; and

a recognizing a lane in which a vehicle corresponding to the object is traveling.

16. The method of claim 11, further comprising:

analyzing user experience (UX) information by collecting the multi-modal control signal.

17. The method of claim 16, further comprising:

generating the multi-modal control signal to select and move the object based on the UX information.

18. The method of claim 11, wherein the selecting comprises selecting the object corresponding to the gesture information at a point in time at which the voice information is recognized.

19. The method of claim 11, wherein the displaying comprises displaying the object-related information using an augmented reality method.

20. The method of claim 11, wherein the object-related information comprises at least one of a distance between a vehicle being driven by the user and the selected object, a movement speed of the selected object, and a lane in which a vehicle corresponding to the object is traveling.