Device for operating an automated machine for handling, assembling or machining workpieces

• US 9,158,298 B2
• Filed: 05/04/2012
• Issued: 10/13/2015
• Est. Priority Date: 05/06/2011
• Status: Active Grant

• Current Assignee: Deckel MAHO Pfronten GmbH (Dmg Mori Co., Ltd.)
• Original Assignee: Deckel MAHO Pfronten GmbH (Dmg Mori Co., Ltd.)
• Inventors: Gronbach, Hans
• Primary Examiner(s): KASENGE, CHARLES R
• Application Number:
  US13/464,864
• Publication Number:
  US 20120290121A1
• Time in Patent Office: 1,257 Days
• Field of Search: 700/65, 700/79, 700/159, 700/177, 700/180, 700/195
• US Class Current: 1/1
• CPC Class Codes: G05B 19/409 characterised by using manual data input [MDI] or by using control panel, e.g. controlling functions with the panel; characterised by control panel details or by setting parameters G05B19/408, G05B19/4093 take precedence
• CPC Class Codes: G05B 2219/35427 User controls machine with eye motion, activates icons on display
• CPC Class Codes: G05B 2219/35444 Gesture interface, controlled machine observes operator, executes commands
• CPC Class Codes: G05B 2219/39441 Voice command, camera detects object, grasp, move
• CPC Class Codes: G05B 2219/40398 Opto-electronic follow-up of movement of head, eyelids, finger to control robot
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/21 the extracted parameters being power information

Abstract

A device for operating an automated machine for handling, assembling or machining workpieces, comprising: a display apparatus having a screen for displaying a graphic user interface for controlling and/or monitoring machine functions of the machine, an operating apparatus for inputting command-triggering operator actions for controlling functions of the machine and controlling functions of the graphic user interface, and a controller for implementing input command-triggering operator actions into control commands for controlling functions of the machine and/or functions of the graphic user interface. The operating apparatus comprises an apparatus for inputting manual operator actions and an apparatus for inputting contact-free operator actions. The apparatus for inputting contact-free operator actions includes: a sensor system for detecting contact-free actions of the operator and a detecting unit for picking up a command-triggering operating action of the operator by means of the contact-free operator actions detected by the sensor system.

Claims

1. A device for operating an automated machine for handling, assembling or machining workpieces, the device comprising:

a display apparatus having a screen for displaying a graphic user interface for controlling and/or monitoring machine functions of the machine;

an operating apparatus for inputting command-triggering operating actions for controlling machine functions of the machine and/or of functions of the graphic user interface by an operator of the machine, comprising a first input apparatus for inputting manual operating actions and a second input apparatus for inputting contact-free operating actions, wherein the second apparatus for inputting contact-free operating actions comprises;

a sensor system for detecting contact-free actions of the operator,

a detecting apparatus for detecting a command-triggering operating action of the operator by means of the contact-free actions of the operator detected by the sensor system, and

a controller for implementing command-triggering operating actions input manually or in a contact-free manner into associated control commands for controlling machine functions of the machine and/or of functions of the graphic user interface by the operator;

wherein the sensor system has a camera system including at least two cameras disposed spaced apart from one another for optically detecting mimic and gestural actions of the operator and an image processing unit adapted to perform a three-dimensional image recognition based on sensor signals of the camera system; and

wherein the sensor system is adapted to optically capture the eyes of the operator and to detect a change of the size of the eye or pupil in the eyes of the operator, the controller being adapted to trigger an emergency stop on the machine if an abrupt eye or pupil enlargement of an operator is detected.

2. The device of claim 1 wherein the camera system is adapted to optically capture the hands of an operator, the image processing unit being adapted to detect a manual gestural action of the operator.

3. The device of claim 2 wherein the image processing unit is adapted to recognize gestural commands of a plurality of predefined gestural commands, each predefined gestural command being associated with a particular command triggering operating action.

4. The device of claim 1 wherein the second input apparatus for the contact-free input of operating commands is adapted to be calibrated to a respective operator in a person-related manner, the second input apparatus having a data interface for importing person-related calibration data so that the calibration to the operator is performed based on the person-related calibration data;

and the calibration data comprise a speech pattern associated with the operator for calibrating a voice recognition unit.

5. A device for operating an automated machine for handling, assembling or machining workpieces, the device comprising:

a display apparatus having a screen for displaying a graphic user interface for controlling and/or monitoring machine functions of the machine;

an operating apparatus for inputting command-triggering operating actions for controlling machine functions of the machine and/or of functions of the graphic user interface by an operator of the machine, comprising a first input apparatus for inputting manual operating actions and a second input apparatus for inputting contact-free operating actions, wherein the second apparatus for inputting contact-free operating actions comprises;

a sensor system for detecting contact-free actions of the operator,

a detecting apparatus for detecting a command-triggering operating action of the operator by means of the contact-free actions of the operator detected by the sensor system, and

a controller for implementing command-triggering operating actions input manually or in a contact-free manner into associated control commands for controlling machine functions of the machine and/or of functions of the graphic user interface by the operator;

wherein the sensor system has a camera system including at least two cameras disposed spaced apart from one another for optically detecting mimic and gestural actions of the operator and an image processing unit adapted to perform a three-dimensional image recognition based on sensor signals of the camera system;

wherein the sensor system is adapted to optically capture the eyes of the operator, the image processing unit being adapted to recognize a viewing direction of the operator based on optically captured pupils of the eyes of the operator; and

wherein the image processing unit is adapted to detect a change of the size of the eye or pupil in the eyes of the operator, the controller being adapted to trigger an emergency stop on the machine if an abrupt eye or pupil enlargement of an operator is detected and a viewing direction of the operator is recognized as facing the machining space of the machine.

6. The device of claim 5 wherein the image processing unit is adapted to detect a viewing position of the operator on the graphic user interface based on the recognized viewing direction.

7. The device of claim 6 wherein the controller is adapted to select screen portions of the graphic user interface based on the detected viewing direction of the operator on the screen.

8. The device of claim 6 wherein the controller is adapted to activate interactive elements of the graphic user interface based on the detected viewing direction of the operator on the screen.

9. The device of claim 6 wherein the controller is adapted to switch between the display of a plurality of screen masks of the graphic user interface based on the detected viewing position of the operator on the screen.

10. The device of claim 5 wherein the image processing unit is adapted to detect a viewing position of the operator on components of the machine in the machining space based on the recognized viewing direction.

11. The device of claim 10 wherein the controller is adapted to display information or control masks associated with a component on the graphic user interface based on the detected viewing position of the operator on said component of the machine.

12. A device for operating an automated machine for handling, assembling or machining workpieces, the device comprising:

a display apparatus having a screen for displaying a graphic user interface for controlling and/or monitoring machine functions of the machine;

an operating apparatus for inputting command-triggering operating actions for controlling machine functions of the machine and/or of functions of the graphic user interface by an operator of the machine, comprising a first input apparatus for inputting manual operating actions and a second input apparatus for inputting contact-free operating actions, wherein the second apparatus for inputting contact-free operating actions comprises;

a sensor system for detecting contact-free actions of the operator,

a detecting apparatus for detecting a command-triggering operating action of the operator by means of the contact-free actions of the operator detected by the sensor system, and

a controller for implementing command-triggering operating actions input manually or in a contact-free manner into associated control commands for controlling machine functions of the machine and/or of functions of the graphic user interface by the operator;

wherein the sensor system has a camera system including at least one camera for optically detecting mimic and gestural actions of the operator, the detecting apparatus comprising an image processing unit adapted to perform an image recognition based on sensor signals of the camera system;

wherein the camera system is adapted to optically capture the eyes of the operator the image processing unit is adapted to recognize an eyelid movement of the operator based on the optically captured eyes of the operator; and

wherein the detecting apparatus is adapted to detect an eyelid movement pattern on the basis of detected eyelid movements, an eyelid movement pattern comprising a number of eyelid movements and time intervals between eyelid movements, the controller being adapted to allocate a detected predetermined eyelid movement pattern with a predetermined number of eyelid movements in a predetermined time interval to a confirmation control command for confirming a contact-free operating action.

13. A device for operating an automated machine for handling, assembling or machining workpieces, the device comprising:

a display apparatus having a screen for displaying a graphic user interface for controlling and/or monitoring machine functions of the machine;

an operating apparatus for inputting command-triggering operating actions for controlling machine functions of the machine and/or of functions of the graphic user interface by an operator of the machine, comprising a first input apparatus for inputting manual operating actions and a second input apparatus for inputting contact-free operating actions, wherein the second apparatus for inputting contact-free operating actions comprises;

a sensor system for detecting contact-free actions of the operator,

a detecting apparatus for detecting a command-triggering operating action of the operator by means of the contact-free actions of the operator detected by the sensor system, and

a controller for implementing command-triggering operating actions input manually or in a contact-free manner into associated control commands for controlling machine functions of the machine and/or of functions of the graphic user interface by the operator;

wherein the sensor system has a microphone system for picking up acoustic actions of the operator; and

wherein the detecting apparatus is adapted to detect a shout or scream of the operator and the loudness thereof captured by the microphone system, the controller being adapted to trigger an emergency stop on the machine if a shout or scream of the operator with a loudness greater than a predetermined loudness limit value is detected by the detecting apparatus.

14. The device of claim 13 wherein the detecting apparatus has a voice recognition unit for recognizing speech input of the operator detected by the microphone system.

15. The device of claim 14 wherein the voice recognition unit is adapted to recognize voice commands of the operator from a plurality of predefined voice commands, each predefined voice command being associated with a particular command-triggering operating action.

16. The device of claim 14 wherein the voice recognition unit is adapted to detect a speech pattern of a voice command picked up by using the microphone system, the controller being adapted to log on an operator with an associated stored speech pattern to the machine and perform only picked-up voice commands that have a detected speech pattern corresponding to the speech pattern associated with the logged-on operator.

17. The device of claim 13 wherein the second input apparatus for the contact-free input of operating commands is adapted to be calibrated to a respective operator in a person-related manner, the second input apparatus having a data interface for importing person-related calibration data so that the calibration to the operator is performed based on the person-related calibration data;

and the calibration data comprise body measurement data associated with the operator for calibrating an image processing unit.