System and method for eye-tracking controlled speech processing with generation of a visual feedback signal

• US 20030050785A1
• Filed: 07/29/2002
• Published: 03/13/2003
• Est. Priority Date: 01/27/2000
• Status: Active Grant

• Current Assignee: Siemens AG
• Original Assignee: Siemens AG
• Inventors: Friedrich, Wolfgang, Wohlgemuth, Wolfgang, Ye, Xin
• Granted Patent: US 6,889,192 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/221 Announcement of recognition results

Abstract

A system and a method for operating and monitoring, in particular, an automation system and/or a production machine or machine tool. The visual field (9) of a user (1) directed onto at least one display (2) is recorded. Speech information (8) from the user (1) is at least intermittently determined and a visual feedback signal is generated in response to a processing status regarding the determined speech information (8). An improved speech interaction is thus obtained, in particular, in the field of augmented-reality applications and in complex technical plants.

Claims

1. System, comprising:

detection means for detecting a visual field of a user being directed onto a display;

speech recognition means for recognizing speech information of the user;

control means for controlling the system; and

means for generating a visual feedback signal relating to a processing status of the speech information of the user.

2. System as claimed in claim 1, further comprising at least one signal path for signals that operate and monitor at least one of an automation system, a production machine, and a machine tool.

3. System as claimed in claim 1, wherein the visual feedback signal comprises a software object that is superimposed onto the display in the detected visual field of the user.

4. System as claimed in claim 1, wherein the visual feedback signal comprises a color signal that identifies the processing status of the speech information of the user.

5. System as claimed in claim 1, wherein said detection means comprise a camera for recording the visual field of the user.

6. System as claimed in claim 1 further comprising acoustic playback means for outputting acoustic information data generated in response to the speech information of the user recognized by said speech recognition means.

7. A method comprising:

detecting that a user'"'"'s view is directed onto a display;

at least intermittently recognizing speech information spoken by the user; and

generating a visual feedback signal that indicates a processing status regarding the speech information spoken by the user.

8. Method as claimed in claim 7, further comprising:

operating and monitoring at least one of an automation system, a production machine, and a machine tool.

9. Method as claimed in claim 7, wherein the visual feedback signal comprises a software object that is superimposed into the detected user'"'"'s view.

10. Method as claimed in claim 7, wherein the visual feedback signal comprises a color signal identifying the processing status of the speech information.

11. Method as claimed in claim 7, wherein the user'"'"'s vision is detected with a camera.

12. Method as claimed in claim 7, further comprising transmitting acoustic information data in response to the speech information spoken by the user.

13. A system comprising:

a vision-tracking device that outputs a tracking signal identifying which of a plurality of target areas a user'"'"'s gaze is directed towards;

a speech processor that converts the user'"'"'s speech into electronic command signals;

a controller processing the electronic commands into control signals that control a downstream process; and

a feedback component that provides visual information to the user regarding at least either the electronic command signals or the control signals at the identified target area in accordance with the tracking signal.

14. The system according to claim 13, wherein the visual information comprises a confirmation that the user'"'"'s speech was successfully converted into an electronic command signal.

15. The system according to claim 13, wherein the visual information comprises a visual confirmation of the content of the electronic command signals.

16. The system according to claim 13, wherein the information comprises confirmation that said controller has executed the downstream process.