System and method for eye tracking controlled speech processing

• US 20030040914A1
• Filed: 07/29/2002
• Published: 02/27/2003
• Est. Priority Date: 01/27/2000
• Status: Active Grant

• Current Assignee: Siemens Healthcare GMBH (Siemens AG)
• Original Assignee: Siemens AG
• Inventors: Friedrich, Wolfgang, Wohlgemuth, Wolfgang, Ye, Xin
• Granted Patent: US 6,853,972 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A system and method for operating and monitoring, in particular, an automation system and/or a production machine and/or machine tool. The visual field (9) of a user (1) directed onto at least one display (2) is recorded and an item of speech information (8) of the user (1) is evaluated at least intermittently, such that specifically predefined information data, which is linked to the recorded visual field (9) and to the recorded speech information, is depicted on the display (2). The predefined information data is displayed in accordance with the speech information (8) that is given by the user (1) and recognized by a speech recognition component (4) and in accordance with the recorded visual field (9). This provides a hands-free operation of the system, machine and/or tool, and enables the user to navigate in an environment of augmented reality applications even when complex technical systems are involved.

Claims

1. System, comprising:

detection means for detecting a visual field of a user being directed onto a display;

speech recognition means for recognizing speech information of the user; and

control means for controlling the system;

wherein, as a function of the speech information given by the user and recognized by said speech recognition means, and as a function of the visual field recorded by said detection means, said control means displays specifically defined information data linked to the detected visual field and the recognized speech information on the display; and

wherein said control means activates said speech recognition means only if said detection means detects that the user'"'"'s visual field is directed to the display.

2. System as claimed in claim 1, further comprising at least one signal path for signals that operate and monitor at least one of an automation system, a production machine, and a machine tool.

3. System as claimed in claim 1, wherein said detection means detects the visual field of the user being directed to a predefined area of the display;

and wherein said control means activates said speech recognition means only if said detection means detects that the user'"'"'s visual field is directed to the predefined area of the display.

4. System as claimed in claim 1, wherein said detection means comprise a camera for recording the visual field of the user.

5. System as claimed in claim 1 further comprising acoustic playback means for outputting acoustic information data generated in response to the speech information of the user recognized by said speech recognition means.

6. System as claimed in claim 1, further comprising additional display means, which signal a location of the information data that is linked with recognized speech information.

7. System as claimed in claim 1, further comprising means for generating a visual feedback signal relating to a processing status of the speech information of the user.

8. System as claimed in claim 7, wherein the visual feedback signal comprises a software object that is superimposed onto the display in the detected visual field of the user.

9. System as claimed in claim 7, wherein the visual feedback signal comprises a color signal that identifies the processing status of the speech information of the user.

10. A method comprising:

detecting that a user'"'"'s view is directed onto a display;

at least intermittently recognizing speech information spoken by the user;

as a function of the speech information spoken by the user and recognized by the speech recognition means, and as a function of the detected view, displaying predefined information data that is linked to the user'"'"'s view and to the recognized speech information on the display; and

activating the recognition of the user'"'"'s speech information only once the user'"'"'s vision is directed onto the display.

11. Method as claimed in claim 10, further comprising:

operating and monitoring at least one of an automation system, a production machine, and a machine tool.

12. Method as claimed in claim 10, wherein the user'"'"'s vision is detected as being directed to a predefined area of the display.

13. Method as claimed in claim 10, wherein the user'"'"'s vision is detected with a camera.

14. Method as claimed in claim 10, further comprising transmitting acoustic information data in response to the speech information spoken by the user.

15. Method as claimed in claim 10, further comprising signaling a location of the display of the predefined information data that is linked to the user'"'"'s view and to the recognized speech information.

16. Method as claimed in claim 10, further comprising generating a visual feedback signal that indicates a processing status regarding the recognized speech information.

17. Method as claimed in claim 16, wherein the visual feedback signal comprises a software object that is superimposed into the detected user'"'"'s view.

18. Method as claimed in claim 16, wherein the visual feedback signal comprises a color signal identifying the processing status of the speech information.

19. A system comprising:

a vision-tracking device that signals whether a user'"'"'s gaze is directed towards a target area;

a speech processor that converts the user'"'"'s speech into electronic command signals; and

an interconnection between said vision tracking device and said speech processor that enables said speech processor to output the electronic command signals only when said vision tracking device signals that the user'"'"'s gaze is directed towards the target area.

20. The system according to claim 19, further comprising:

a controller processing the electronic commands into control signals that control a downstream process; and

a feedback component that provides information to the user regarding the control signals.