Generating visual feedback signals for eye-tracking controlled speech processing

• US 6,889,192 B2
• Filed: 07/29/2002
• Issued: 05/03/2005
• Est. Priority Date: 01/27/2000
• Status: Expired due to Term

• Current Assignee: Siemens AG
• Original Assignee: Siemens AG
• Inventors: Friedrich, Wolfgang, Wohlgemuth, Wolfgang, Ye, Xin
• Primary Examiner(s): Chawan, Vijay B.
• Application Number:
  US10/206,669
• Publication Number:
  US 20030050785A1
• Time in Patent Office: 1,009 Days
• Field of Search: 704/271, 704/275, 704/270, 704231-235, 704/277, 704/200, 704/9, 704/10, 704/251, 704/257, 706/45, 345/156, 345/420, 345/8, 701/14, 703/1, 708/141
• US Class Current: 704/275
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/221 Announcement of recognition results

Abstract

A system and a method for operating and monitoring, in particular, an automation system and/or a production machine or machine tool. The visual field (9) of a user (1) directed onto at least one display (2) is recorded. Speech information (8) from the user (1) is at least intermittently determined and a visual feedback signal is generated in response to a processing status regarding the determined speech information (8). An improved speech interaction is thus obtained, in particular, in the field of augmented-reality applications and in complex technical plants.

Claims

1. A monitoring and operating system, comprising:

detection means for detecting a visual field of a user being directed onto a display;

speech recognition means for recognizing speech information of the user;

control means for controlling the system; and

means for generating a visual feedback signal relating to a processing status of the speech information of the user,

wherein the visual feedback signal comprises a color signal that identifies the processing status of the speech information of the user,

wherein the color signal comprises a plurality of colors, each of the plurality of colors respectively identifying one of a plurality of processing statuses, and

wherein the plurality of processing statuses include a status indicating that the speech information is recognized and a corresponding action has been performed, a status indicating that the speech information is not recognized, and a status indicating that the speech information is recognized, but a corresponding action has not yet been executed.

2. The system as claimed in claim 1, further comprising at least one signal path for signals that operate and monitor at least one of an automation system, a production machine, and a machine tool.

3. The system as claimed in claim 1, wherein the visual feedback signal comprises a software object that is superimposed onto the display in the detected visual field of the user.

4. The system as claimed in claim 1, wherein said detection means comprise a camera for recording the visual field of the user.

5. The system as claimed in claim 1, further comprising acoustic playback means for outputting acoustic information data generated in response to the speech information of the user recognized by said speech recognition means.

6. A method for monitoring and operating a system comprising:

detecting that a user'"'"'s view is directed onto a display;

at least intermittently recognizing speech information spoken by the user;

generating a visual feedback signal that indicates a processing status regarding the speech information spoken by the user,

wherein the visual feedback signal comprises a color signal identifying the processing status of the speech information, and

identifying one of a plurality of processing statuses by one of a plurality of colors,

wherein the plurality of processing statuses include a status indicating that the speech information is recognized and a corresponding action has been performed, a status indicating that the speech information is not recognized, and a status indicating that the speech information is recognized, but a corresponding action has not yet been executed.

7. The method as claimed in claim 6, further comprising:

operating and monitoring at least one of an automation system, a production machine, and a machine tool.

8. The method as claimed in claim 6, wherein the visual feedback signal comprises a software object that is superimposed into the detected user'"'"'s view.

9. The method as claimed in claim 6, wherein the user'"'"'s vision is detected with a camera.

10. The method as claimed in claim 6, further comprising transmitting acoustic information data in response to the speech information spoken by the user.