Man-machine system

• US 5,247,433 A
• Filed: 03/20/1990
• Issued: 09/21/1993
• Est. Priority Date: 03/20/1989
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Ujita, Hiroshi, Fukuda, Mitsuko, Kitaura, Wataru
• Primary Examiner(s): Smith, Jerry
• Assistant Examiner(s): Brown, Thomas E.
• Application Number:
  US07/495,919
• Time in Patent Office: 1,281 Days
• Field of Search: 364/188-190, 364/492, 364/493, 364/146, 364/552, 364/551.01, 364/709.11, 395/155, 395/162, 376/215-217
• US Class Current: 700/83
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: Y10S 715/978 Audio interaction as part of an operator interface

Abstract

A man-machine system connected to a plant has a storage device for storing data of an ideal operator model and recognizing section for inputting voice information generated by the operator and information about the operator'"'"'s line of sight. When information about a failure in the plant is input into the man-machine system, plant data or operation information provided at the corresponding locations on an operational panel recognized by the recognizing section according to voice information or information about the operator'"'"'s line of sight is selected and shown to the operator, so that the operator can take remedial measures for the failure quickly.

Claims

1. A man-machine system comprising:

an operation model;

storage means for storing operation data;

a plurality of regions provided on an operation panel, each one of said plurality of regions being related with said operation data adoptable under a certain operating condition;

means for creating a first operator model including a plurality of focuses of attention to regions to which an operator should pay attention with regard to said certain operating conditions of equipment to be controlled;

first information recognizing means for inputting information generated by a voice of said operator;

means for estimating a stage of operation which said operator is engaged in out of said focuses of attention included in said first operator model by using voice information generated by said operator and inputted to said first information recognizing means;

second information recognizing means for inputting information generated by an operator'"'"'s line of sight;

means for selecting a part of said operation data from said storage means based on said information about said operator'"'"'s line of sight;

means for creating a second operator model corresponding to an operator model including said estimated focus of attention; and

means for editing supplied information by using information about said estimated focus of attention included in said second operator model.

2. A man-machine system according to claim 1, wherein said first operator model includes time information for forming said first operator model.

3. A man-machine system according to claim 1, wherein said second operator model includes information of cognitive process activity estimated based on information from said operator.

4. A man-machine system according to claim 1, further comprising:

means for estimating a stage of an operator'"'"'s cognitive process activity by using said voice information and said information about said operator'"'"'s line of sight, and means for deciding a form in which information is supplied according to said estimated stage of said cognitive process activity, wherein said information editing means edits said supplied information according to said decided form.

5. A man-machine system according to claim 4, wherein said information editing means includes selectedinformation supply means for supplying information selected according to information about said estimated focus of attention.

6. A man-machine system according to claim 1, further comprising:

means for checking whether said operator looked at least one information edited by said information editing means according to said information about said operator'"'"'s line of sight after displaying said edited information, and outputting a signal to arouse said operator'"'"'s attention if information exists that said operator did not look at.

13. A man-machine system according to claim 1, wherein said voice information includes a pronoun and said selecting means selects a part of said operation data based on a portion of one of said plurality of regions at said operator'"'"'s line of sight.

7. A man-machine system comprising:

means for creating a first operator model including a plurality of focuses of attention which an operator should pay with regard to a certain operating condition of equipment having control means;

means for estimating a focus of attention that said operator requires out of said focuses of attention included in said first operator model by using voice information generated by said operator and information about an operator'"'"'s line sight;

means for creating a second operator model corresponding to an operator model including said estimated focus of attention;

means for selecting an unit which supplies information to said operator, said unit having been decided in advance according to said voice information and said information about said operator'"'"'s line of sight; and

means for editing supplied information by using information about said estimated focus of attention included in said second operator model and information obtained from said equipment having said control means, and outputting said supplied information, to said decided unit which supplies information to said operator.

8. A man-machine system comprising:

means for creating a first operator model including a plurality of focuses of attention that an operator should pay with regard to a certain operating condition of equipment having control means;

means for estimating a focus of attention that said operator requires out of said focuses of attention included in said first operator model by using voice information generated and entered by said operator and information about an operator'"'"'s line sight;

means for estimating information to which said operator is paying attention according to said voice information and said information about said operator'"'"'s line of sight;

means for creating a second operator model corresponding to an operator model including information about said estimated focus of attention and information to which said operator is paying attention; and

means for editing supplied information by using said kinds of information includes in said second operator model and information obtained from the equipment having said control means.

9. A man-machine system according to claim 8, further comprising:

means for comparing information obtained by said information editing means with said information included in said second operator model in order to check whether there is any piece of the latter information which is not included in the former information, wherein if there is any piece of said latter information that is not included in said former information, said information is additionally edited by said information editing means.

10. A man-machine system comprising:

means for creating a first operator model including a plurality of focuses of attention which an operator should pay with regard to a certain operating condition of equipment having control means;

means for estimating a focus of attention that said operator requires out of said focuses of attention included in said first operator model by using voice information generated and entered by said operator and information about an operator'"'"'s line sight;

means for creating a second operator model corresponding to an operator model including said estimated focus of attention;

means for editing supplied information by using information about said estimated focus of attention included in said second operator model and information obtained from said equipment having said control means,

display means for displaying said information edited by said information editing means;

voice output means;

a television camera; and

means for deciding according to video signals displayed on said television camera whether said operator is located in a position where said operator can look at said display means, and outputting said edited information to said voice output means if said operator is located in a position where said operator cannot look at said display means.

11. A man-machine system comprising:

means for creating a first operator model including a plurality of focuses of attention that an operator should pay with regard to a certain operating condition of equipment having control means;

means for estimating a focus of attention that said operator requires out of said focuses of attention included in said first operator model by using voice information generated and entered into a display information creating device by said operator and information about an operator'"'"'s line sight;

means for creating a second operator model including information about said estimated focus of attention; and

means for editing information to display by using information about said estimated focus of attention and information obtained form said equipment having said control means.

12. An operating apparatus comprising:

an operation panel having information output means;

means for creating a first operator model including a plurality of focuses of attention that an operator should pay with regard to a certain operating condition of equipment having said control means;

means for estimating a focus of attention that said operator requires out of said focuses of attention included in said first operator model by using voice information generated by said operator and information about an operator'"'"'s line of sight;

means for creating a second operator model including information about said estimated focus of attention; and

means for editing supplied information to said information output means by using said information about said estimated focus of attention included in said second operator model and information obtained from said equipment to be controlled.