Multi-modal interface apparatus and method

• US 6,118,888 A
• Filed: 02/25/1998
• Issued: 09/12/2000
• Est. Priority Date: 02/28/1997
• Status: Expired due to Term

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Ikeda, Tomoo, Chino, Tetsuro, Tanaka, Katsumi, Yano, Takehide, Kono, Yasuyuki
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Werner, Brian P.
• Application Number:
  US09/030,213
• Time in Patent Office: 930 Days
• Field of Search: 382/100, 382/103, 382/117, 382/118, 382/115, 382/325, 382/311, 345/173, 345/175, 345/358, 345/336, 345/337, 345/338, 345/978, 345/473, 345/326, 345/327, 348/78, 348/169-172, 340/825.34, 704/275, 463/34
• US Class Current: 382/118
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015

Abstract

In the multi-modal interface apparatus of the present invention, a gaze object detection section always detects a user'"'"'s gaze object. The user inputs at least one medium of sound information, character information, image information and operation information through a media input section. In order to effectively input and output information between the user and the apparatus, a personified image presentation section presents a personified image to the user based on the user'"'"'s gaze object. A control section controls a reception of the inputted media from the media input section based on the user'"'"'s gaze object.

Claims

1. Multi-modal interface apparatus, comprising:

gaze object detection means for detecting a user'"'"'s gaze object;

media input means for inputting at least one medium of sound information, character information, image information and operation information from the user;

personified image presentation means for displaying an agent image to the user, the agent image expressing predetermined gesture and looks as non-language message, the predetermined gesture and looks being different based on a kind of the user'"'"'s gaze object;

information output means for outputting at least one medium of sound information, character information and image information to the user;

interpretation rule memory means for previously storing a plurality of rules consisting of a present status information, a gaze object information, a kind of input information or output information, an interpretation result information, wherein the present status information represents a status of the apparatus, the gaze object information represents the kind of the user'"'"'s gaze object, the kind of input information or output information represents signal status of said media input means or said information output means, the interpretation result information represents a next event of the apparatus;

control rule memory means for previously storing a plurality of rules consisting of the present status information, an event condition information, an action information, a next status information, wherein the event condition information corresponds to the interpretation result information of said interpretation rule memory means, the action information represents processing list to execute, the next status information represents a next desired status of the apparatus; and

control means for searching the interpretation result information from the interpretation rule memory means based on the present status of the apparatus as the present status information, the user'"'"'s gaze object as the gaze object information, the signal status of said media input means or said information output means as the kind of input information or output information when the user'"'"'s gaze object is detected by said gaze object detection means, for searching the action information and the next status information from the control rule memory means based on the present status of the apparatus as the present status information and the interpretation result information as the event condition information if the interpretation result information is retrieved from said interpretation rule memory means, for controlling said media input means to execute at least one of start, end, interruption, restart of input processing based on the action information, for controlling said personified image presentation means to display the agent image whose gesture and looks represent a reception or a completion of the user'"'"'s input in synchronization with the input processing of said media input means, and for controlling a least one of start, end, interpretation, restart of the at least one medium outputted by said information output means based on the action information.

2. The multi-modal interface apparatus according to claim 1,

further comprising an attention calling means for calling the user'"'"'s attention through said personified image presentation means or said information output means,

wherein said control means controls said attention calling means to call the user'"'"'s attention based on the kind of the user'"'"'s gaze object when said information output means outputs the at least one medium to the user.

3. The multi-modal interface apparatus according to claim 2,

further comprising a reaction detection means for detecting the user'"'"'s reaction in response to the calling of said attention calling means based on the user'"'"'s gaze object or the inputted media from said media input means,

wherein said control means controls at least one activation of said information output means and said attention calling means based on the user'"'"'s reaction.

4. The multi-modal interface apparatus according to claim 1,

wherein said control means executes processing of calling the user'"'"'s attention based on the action information through at least one of said personified image presentation means and said information output means, and updates the present status of the apparatus based on the next status information, if the action information and the next status information are retrieved from said control rule memory means.

5. The multi-modal interface apparatus according to claim 1,

wherein said control means searches the action information and the next status information from the control rule memory means based on the present status of the apparatus as the present status information and a control signal as the event condition information, if the control signal is outputted from said control means to at least one means.

6. The multi-modal interface apparatus according to claim 5,

wherein said control means executes processing of calling the user'"'"'s attention based on the action information through at least one of said personified image presentation means and said information output means, and updates the present status of the apparatus based on the next status information, if the action information and the next status information are retrieved from said control rule memory means.

7. A multi-modal interface method, comprising the steps of:

detecting a user'"'"'s gaze object;

inputting at least one medium of sound information, character information, image information and operation information from the user;

displaying an agent image to the user, the agent image expressing predetermined gesture and looks as non-language message, the predetermined gesture and the looks being different based on a kind of the user'"'"'s gaze object;

outputting at least one medium of sound information, character information and image information to the user;

storing a plurality of rules consisting of a present status information, a gaze object information, a kind of input information or output information, an interpretation result information in an interpretation rule memory, wherein the present status information represents a status of the apparatus, the gaze object information represents the kind of the user'"'"'s gaze object, the kind of input information or output information represents signal status of a media input means or an information output means, the interpretation result information represents a next event of the apparatus;

storing a plurality of rules consisting of the present status information, an event condition information, an action information, a next status information in a control rule memory, wherein the event condition information corresponds to the interpretation result information of the interpretation rule memory, the action information represents processing list to execute, the next status information represents a next desired status of the apparatus;

searching the interpretation result information from said interpretation rule memory based on the present status of the apparatus as the present status information, the user'"'"'s gaze object as the gaze object information, the signal status of said media input means or said information output means as the kind of input information or output information, when the user'"'"'s gaze object is detected by said detecting step;

searching the action information and the next status information from said control rule memory based on the present status of the apparatus as the present status information and the interpretation result information as the event condition information, if the interpretation result information is retrieved from said interpretation rule memory;

controlling the input processing to execute at least one of start, end, interpretation, restart based on the action information;

controlling the gesture and looks of the agent image to represent a reception or a completion of the user'"'"'s input in synchronization with the input processing; and

controlling at least one of start, end, interruption, restart of the at least one medium outputted by said information output means based on the action information.

8. A computer readable memory containing computer readable instructions, comprising:

instruction means for causing a computer to detect a user'"'"'s gaze object;

instruction means for causing a computer to input at least one medium of sound information, character information, image information and operation information from the user;

instruction means for causing a computer to display an agent image to the user, the agent image expressing predetermined gesture and looks as non-language message, the predetermined gesture and the looks being different based on a kind of the user'"'"'s gaze object;

instruction means for causing a computer to output at least one medium of sound information, character information and image information to the user;

instruction means for causing a computer to store a plurality of rules consisting of a present status information, a gaze object information, a kind of input information or output information, an interpretation result information in an interpretation rule memory, wherein the present status information represents a status of the apparatus, the gaze object information represents the kind of the user'"'"'s gaze object, the kind of input information or output information represents signal status of a media input means or an information output means, the interpretation result information represents a next event of the apparatus;

instruction means for causing a computer to store a plurality of rules consisting of the present status information, an event condition information, an action information, a next status information in a control rule memory, wherein the event condition information corresponds to the interpretation result information of the interpretation rule memory, the action information represents processing list to execute, the next status information represents a next desired status of the apparatus;

instruction means for causing a computer to search the interpretation result information from said interpretation rule memory based on the present status of the apparatus as the present status information, the user'"'"'s gaze object as the gaze object information, the signal status of said media input means or said information output means as the kind of input information or output information, when the user'"'"'s gaze object is detected by said detection means;

instruction means for causing a computer to search the action information and the next status information from said control rule memory based on the present status of the apparatus as the present status information and the interpretation result information as the event condition information, if the interpretation result information is retrieved from said interpretation rule memory;

instruction means for causing a computer to control the input processing to execute at least one of start, end, interruption, restart based on the action information;

instruction means for causing a computer to control the gesture and looks of the agent image to represent a reception or a completion of the user'"'"'s input in synchronization with the input processing; and

instruction means for causing a computer to control at least one of start, end, interruption, restart of the at least one medium outputted by said information output means based on the action information.

9. Multi-modal interface apparatus, comprising:

image input means for inputting a user'"'"'s image including face and hand;

recognition means for extracting the face and hand from the user'"'"'s image, and for recognizing a face position as the user'"'"'s view position, a finger end position as the user'"'"'s gesture part, and a reference object pointed by the finger end as the user'"'"'s gesture input in a world coordinate space;

a location information memory means for storing a representative position and a direction by unit of a label information in the world coordinate space, wherein the label information is one of a presentation position of a personified image, the face, the finger end and the reference object, a plurality of the presentation positions being predetermined, the face, the finger end and the object being recognized by said recognition means;

control means for deciding a visual relation between the presentation position and the user'"'"'s recognized position by referring to said location information memory means, wherein the user'"'"'s recognized position includes the face position and the finger end position, and for generating the personified image expressing at least one of looks and action based on the visual relation; and

personified image presentation means for presenting the personified image at the presentation position to the user as a feedback information of the gesture input;

wherein said control means decides whether the user'"'"'s gesture part is watched from a current presentation position of the personified image and whether the personified image is watched from the user'"'"'s view position, and generates the personified image which gazes the user'"'"'s gesture part if the user'"'"'s gesture part is watched from the current presentation position of the personified image and the personified image is watched from the user'"'"'s view position.

10. The multi-modal interface apparatus according to claim 9,

wherein said control means searches other presentation position where the user'"'"'s gesture part is watched from the other presentation position of the personified image and the personified image is watched from the user'"'"'s view position by referring to said location information memory means, if the user'"'"'s gesture part is not watched from the current presentation position of the personified image or the personified image of the current presentation position is not watched from the user'"'"'s view position.

11. The multi-modal interface apparatus according to claim 10,

wherein said control means moves the personified image from the current presentation position to the other presentation position, if the other presentation position is found in said location information memory means.

12. The multi-modal interface apparatus according to claim 11,

wherein said control means generates the personified image whose looks represents a reception of the user'"'"'s gesture input, if the other presentation position is not found in said location information memory means.

13. The multi-modal interface apparatus according to claim 12,

wherein said control means generates the personified image whose looks represents an analysis failure of the user'"'"'s gesture input, if the user'"'"'s gesture part is positioned outside a view area of said image input means.

14. The multi-modal interface apparatus according to claim 13,

wherein said control means decides whether the reference object of the gesture input is watched from the presentation position of the personified image and whether the personified image is watched from the user'"'"'s view position, if the gesture input is a pointing gesture.

15. The multi-modal interface apparatus according to claim 14,

wherein said control means generates the personified image who gazes the reference object, if the reference object is watched from the presentation position of the personified image and the personified image is watched from the user'"'"'s view position.

16. A multi-modal interface method, comprising the steps of:

inputting a user'"'"'s image including face and hand;

extracting the face and hand from the user'"'"'s image;

recognizing a face position as the user'"'"'s view position, a finger end position as the user'"'"'s gesture part, and a reference object pointed by the finger end as the user'"'"'s gesture input in a world coordinate space;

storing a representative position and a direction as location information by unit of a label information in the world coordinate space, wherein the label information is one of a presentation position of a personified image, the face, the finger end and the reference object, a plurality of the presentation positions being predetermined, the face, the finger end and the reference object being recognized by said recognition step;

deciding a visual relation between the presentation position and the user'"'"'s recognized position by referring to the location information, wherein the user'"'"'s recognized position includes the face position and the finger end position;

generating the personified image expressing at least one of looks and action based on the visual relation; and

presenting the personified image at the presentation position to the user as a feedback information of the gesture input;

wherein said deciding step includes the step of;

deciding whether the user'"'"'s gesture part is watched from a current presentation position of the personified image and whether the personified image is watched from the user'"'"'s view position; and

wherein said generating step includes the step of;

generating the personified image which gazes the user'"'"'s gesture part if the user'"'"'s gesture part is watched from the current presentation position of the personified image and the personified image is watched from the user'"'"'s view position.

17. The multi-modal interface method according to claim 16,

wherein said step of deciding further comprises searching an other presentation position where the user'"'"'s gesture part is watched from the other presentation position of the personified image and the personified image is watched from the user'"'"'s view position by referring to said location information, if the user'"'"'s gesture part is not watched from the current presentation position of the personified image or the personified image of the current presentation position is not watched from the user'"'"'s view position.

18. The multi-modal interface method according to claim 17,

wherein said step of deciding further comprises moving the personified image from the current presentation position to the other presentation position, if the other presentation position is found in said location information.

19. The multi-modal interface method according to claim 18,

wherein said step of deciding further comprises generating the personified image having looks representing a reception of the user'"'"'s gesture input, if the other presentation position is not found in said location information.

20. The multi-modal interface method according to claim 19,

wherein said deciding further comprises generating the personified image having looks representing an analysis failure of the user'"'"'s gesture input, if the user'"'"'s gesture part is positioned outside a view area of said user'"'"'s image.

21. The multi-modal interface method according to claim 20,

wherein said step of deciding further comprises determining whether the reference object of the gesture input is watched from the presentation position of the personified image and whether the personified image is watched from the user'"'"'s view position, if the gesture input is a pointing gesture.

22. The multi-modal interface method according to claim 21,

wherein said step of deciding further comprises generating the personified image which gazes the reference object, if the reference object is watched from the presentation position of the personified image and the personified image is watched from the user'"'"'s view position.

23. A computer readable memory containing computer readable instructions, comprising:

instruction means for causing a computer to input a user'"'"'s image including face and hand;

instruction means for causing a computer to extract the face and hand from the user'"'"'s image;

instruction means for causing a computer to recognize a face position as the user'"'"'s view position, a finger end position as the user'"'"'s gesture part, and a reference object pointed by the finger end as the user'"'"'s gesture input in a world coordinate space;

instruction means for causing a computer to store a representative position and a direction as location information by unit of a label information in the world coordinate space, wherein the label information is one of a presentation position of a personified image, the face, the finger end and the reference object, a plurality of the presentation positions being predetermined, the face, the finger end and the reference object being recognized by said recognition means;

instruction means for causing a computer to decide a visual relation between the presentation position and the user'"'"'s recognized position by referring to the location information, wherein said user'"'"'s recognized position includes the face position and the finger end position;

instruction means for causing a computer to generate the personified image expressing at least one of looks and action based on the visual relation; and

instruction means for causing a computer to present the personified image at the presentation position to the user as a feedback information of the gesture input;

wherein said instruction means for causing a computer to decide further comprises instruction means for causing a computer to decide whether the user'"'"'s gesture part is watched from a current presentation position of the personified image and whether the personified image is watched from the user'"'"'s view position, and

wherein said instruction means for causing a computer to generate further comprises instruction means for causing a computer to generate the personified image which gazes the user'"'"'s gesture part if the user'"'"'s gesture part is watched from the current presentation position of the personified image and the personified image is watched from the user'"'"'s view position.

24. The computer readable memory according to claim 23,

wherein said instruction means for causing a computer to decide further comprises instruction means for causing a computer to search other presentation position where the user'"'"'s gesture part is watched from the other presentation position of the personified image and the personified image is watched from the user'"'"'s view position by referring to said location information, if the user'"'"'s gesture part is not watched from the current presentation position of the personified image or the personified image of the current presentation position is not watched from the user'"'"'s view position.

25. The computer readable memory according to claim 24,

wherein said instruction means for causing a computer to decide further comprises instruction means for causing a computer to move the personified image from the current presentation position to the other presentation position, if the other presentation position is found in said location information.

26. The computer readable memory according to claim 25,

wherein said instruction means for causing a computer to decide further comprises instruction means for causing a computer to generate the personified image having looks representing a reception of the user'"'"'s gesture input, if the other presentation position is not found in said location information.

27. The computer readable memory according to claim 26,

wherein said instruction means for causing a computer to decide further comprises instruction means for causing a computer to generate the personified image having looks representing an analysis failure of the user'"'"'s gesture input, if the user'"'"'s gesture part is positioned outside a view area of said user'"'"'s image.

28. The computer readable memory according to claim 27,

wherein said instruction means for causing a computer to decide further comprises instruction means for causing a computer to determine whether the reference object of the gesture input is watched from the presentation position of the personified image and whether the personified image is watched from the user'"'"'s view position, if the gesture input is a pointing gesture.

29. The computer readable memory according to claim 28,

wherein said instruction means for causing a computer to decide further comprises instruction means for causing a computer to generate the personified image which gazes the reference object, if the reference object is watched from the presentation position of the personified image and the personified image is watched from the user'"'"'s view position.