Recognition-based industrial automation control with confidence-based decision support

• US 9,498,885 B2
• Filed: 02/27/2013
• Issued: 11/22/2016
• Est. Priority Date: 02/27/2013
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Scott, Steven J., Nguyen, Thong T., Vasko, David A., Gasperi, Michael, Brandt, David D.
• Primary Examiner(s): Jen, Ian
• Application Number:
  US13/779,480
• Publication Number:
  US 20140244036A1
• Time in Patent Office: 1,364 Days
• Field of Search: 700/1, 700/253
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 19/409 characterised by using manual data input [MDI] or by using control panel, e.g. controlling functions with the panel; characterised by control panel details or by setting parameters G05B19/408, G05B19/4093 take precedence

Abstract

The present disclosure generally relates to a method for performing industrial automation control in an industrial automation system. As such, the method may include detecting, via a sensor system, positions and/or motions of a human. The method may then include determining a possible automation command corresponding to the detected positions and/or motions. After determining the possible automation command, the method may implement a control and/or notification action based upon the detected positions and/or motions.

Claims

1. A method for performing industrial automation control in an industrial automation system, comprising:

detecting, via a sensor system, one or more gestures of a human, a first set of positions of the human, and a second set of positions of an industrial automation device;

determining a probability that the human will contact the industrial automation device based on the first and second sets of positions;

determining a possible automation command corresponding to the detected one or more gestures;

implementing a first control action based upon the possible automation command, wherein the control action is configured to cause an automation controller to control an operation of the industrial automation device;

implementing a second control action based upon when the probability exceeds a value, wherein the second control action comprises adjusting the operation of the industrial automation device to avoid contacting the human.

2. The method of claim 1, wherein the one or more gestures correspond to a plurality of possible gestures, each gesture being associated with a determined automation command.

3. The method of claim 1, comprising:

determining a confidence value associated with the detected one or more gestures and the possible automation command; and

implementing the first control action based upon the confidence value.

4. The method of claim 3, comprising implementing a third control action configured to adjust the operation of the industrial automation device in the industrial automation system when the confidence value is below a threshold.

5. The method of claim 3, wherein the confidence value is indicative of relative accuracy in the determination of at least one of the detected gestures.

6. The method of claim 3, wherein the confidence value is indicative of relative accuracy of matching of the one or more gestures to the first automation command.

7. The method of claim 3, wherein the confidence value is indicative of relative accuracy of matching of the one or more gestures to an expected pattern of movement.

8. The method of claim 3, wherein implementing the first control action comprises sending the determined automation command to the industrial automation device when the confidence value is greater than a threshold.

9. The method of claim 1, comprising sending a notification message indicative of the determined automation command.

10. A method for performing industrial automation control, comprising:

detecting, via a sensor system, gestures of an element of an industrial system;

detecting, via the sensor system, positions and/or motions of a human operating in the industrial system;

detecting, via the sensor system, positions and/or motions of a machine operating in the industrial system;

determining a possible automation action corresponding to the detected gestures;

determining a probability that the human will contact the machine based on the positions and/or motions of the human and the positions and/or the motions of the machine;

implementing a control action based upon the possible automation action, wherein the control action is configured to cause an automation controller to control an operation of an industrial automation device; and

sending a command to alter the operation to cause the machine to avoid contacting the human when the probability is greater than a threshold.

11. The method of claim 10, wherein the element comprises a machine component.

12. The method of claim 10, wherein the element comprises a workpiece.

13. The method of claim 10, wherein the control action comprises a command for operation of the machine.

14. The method of claim 10, comprising:

determining a confidence value associated with the detected gestures and the determined automation action; and

implementing the control action based upon the confidence value.

15. The method of claim 14, comprising sending a notification message indicating that the confidence value is low when the confidence value is not greater than a limit.

16. The method of claim 10, wherein the command to alter the operation comprises stopping the operation of the machine.

17. A system for performing industrial automation control, comprising:

a sensor system configured to;

detect gestures of an element of an industrial system;

detect positions and/or motions of a human operating in the industrial system; and

detect positions and/or motions of a machine operating in the industrial system;

a processing component configured to;

compare the detected gestures with a plurality of automation commands;

identify an automation command from the plurality of automation commands based at least in part on the comparison;

determine a probability that the human will contact the machine based on the positions and/or motions of the human and the positions and/or the motions of the machine;

send the identified automation command to the machine in the industrial system after the automation command is identified;

send a command to alter the identified automation command to cause the machine to avoid contacting the human when the probability is greater than a threshold.

18. The system of claim 17, wherein the sensor system comprises digital cameras, video cameras, infrared sensors, optical sensors, radio frequency energy detectors, sound sensors, vibration sensors, magnetic sensors, heat sensors, pressure sensors, sonar sensors, range imaging technology, or any combination thereof.

19. The system of claim 17, wherein the processing component is configured to:

determine whether the element is within a space of the industrial system; and

send a second command to at least one industrial automation device in the industrial system to stop at least one operation of the at least one industrial automation device when the element is within the space.

20. The system of claim 19, wherein the space is designated using a light curtain.

21. The system of claim 17, wherein the processing component is configured to generate a confidence value that corresponds to the detected gestures and based at least in part the comparisons, wherein the identified automation command is sent to the industrial automation device based at least in part on the confidence value.

22. The system of claim 21, wherein the confidence value is generated based at least in part on visibility of the element, historical data related to the element, expected positions of the element, or any combination thereof.

23. The system of claim 21, wherein the processing component is configured to send a second automation command to the industrial automation device to operate in a safe state when the confidence value is not greater than a threshold.

24. The system of claim 23, wherein the safe state comprises continuing the operation of the industrial automation device.