INTEGRATION OF OPTICAL AREA MONITORING WITH INDUSTRIAL MACHINE CONTROL

• US 20150332463A1
• Filed: 11/25/2014
• Published: 11/19/2015
• Est. Priority Date: 05/19/2014
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Galera, Richard, Bowlby, Anne, Jones, Derek W., Pradhan, Nilesh, Leard, Francis L.
• Granted Patent: US 9,256,944 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: F16P 3/142 using image capturing devices
• CPC Class Codes: G05B 19/048 Monitoring; Safety
• CPC Class Codes: G05B 2219/14006 Safety, monitoring in general
• CPC Class Codes: G05B 9/03 with multiple-channel loop, i.e. redundant control systems
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An industrial safety system is provided that integrates optical safety monitoring with machine control. The safety system includes an imaging sensor device supporting pixel array processing functions that allow time-of-flight (TOF) analysis to be performed on selected portions of the pixel array, while two-dimensional imaging analysis is performed on the remaining portions of the array, reducing processing load and response time relative to performing TOF analysis for all pixels of the array. The portion of the pixel array designated for TOF analysis can be pre-defined through configuration of the imaging sensor device, or can be dynamically selected based on object detection and classification by the two-dimensional imaging analysis. The imaging sensor device can also implement a number of safety and redundancy functions to achieve a high degree of safety integrity.

Claims

1. An optical safety system, comprising:

a memory that stores computer-executable components;

a processor, operatively coupled to the memory, that executes the computer-executable components, the computer-executable components comprising;

a pixel array component configured to, for one or more pixel arrays of one or more images of a hazardous industrial area captured by one or more imaging sensor devices, group pixels of the one or more pixel arrays to yield a subset of the pixels on which two-dimensional (2D) analysis is to be performed;

an image analysis component configured to perform 2D analysis on the subset of the pixels;

a distance determination component configured to perform 3D analysis on point cloud data received from the one or more imaging sensor devices; and

a hazard analysis and decision component configured to classify a first object in the image as motorized industrial equipment and a second object in the images as a human based on results of the 2D analysis and the 3D analysis, to correlate the results of the 2D analysis and the 3D analysis to yield correlated information relating to a relationship between the human and the motorized industrial equipment, and to generate a control output that controls the motorized equipment based on the correlated information.

2. The system of claim 1, wherein the point cloud data comprises distance information determined for selected pixels of the one or more pixel arrays by the one or more imaging sensor devices, and the distance determination component is configured to aggregate the point cloud data from the one or more imaging sensor devices to yield three-dimensional image data for the hazardous industrial area.

3. The system of claim 2, wherein the hazard analysis and decision component is configured to classify the first object and the second object based on analysis of the three-dimensional image data and additional information comprising at least one of passive infrared data, localization data, object tagging data, or temperature sensing data.

4. The system of claim 3, wherein the hazard analysis and decision component is configured to

estimate the first object and the second object as respective three-dimensional geometric objects within a three-dimensional space represented by the three-dimensional image data, wherein the geometric objects respectively encapsulate the first object and the second object, and

determine a minimum distance between the geometric objects to yield an estimated distance between the first object and the second object.

5. The system of claim 3, wherein the hazard analysis and decision component is configured to

project the three-dimensional representations of the first object and the second object defined by the three-dimensional image data onto a two-dimensional plane to yield a first projection and a second projection;

and

determine a minimum distance between the first projection and the second projection to yield an estimated distance between the first object and the second object.

6. The system of claim 5, wherein the hazard analysis and decision component is further configured to

encapsulate the first projection and the second projection within respective geometrical objects, and

estimate the minimum distance as a distance between the geometrical objects.

7. The system of claim 1, wherein the hazard analysis and decision component is further configured to identify a location of the human relative to the motorized industrial equipment based on at least one of the 2D analysis or the 3D analysis.

8. The system of claim 1, further comprising a safety component configured to monitor one or more internal components of the one or more imaging sensor devices to detect a fault condition, and to at least one of generate a safety output or modify at least one of the results of the 2D analysis and the 3D analysis to compensate for the fault condition in response to detection of the fault condition based on one or more safety algorithms.

9. The system of claim 1, wherein the fault condition comprises at least one of a temperature condition, a power condition, a vibration condition, or a component fault condition.

10. The system of claim 2, wherein the correlated information comprises at least one of a location of the human relative to the motorized industrial equipment, a velocity of the human, a velocity of the motorized industrial equipment, an acceleration of the human, an acceleration of the motorized industrial equipment, a trajectory of the human, a trajectory of the motorized industrial equipment, or an anticipated intersection between the trajectory of the human and the trajectory of the motorized industrial equipment.

11. The system of claim 10, wherein the hazardous analysis and decision component is further configured to predict a future location of at least one of the human or a component of the motorized industrial equipment within the hazardous industrial area based the correlated information, and to generate the control output based on the future location.

12. The system of claim 1, further comprising an interface component configured to exchange data with an industrial controller over a network.

13. A method for controlling motorized industrial equipment, comprising:

collecting, by a safety system comprising at least one processor, image data from one or more imaging sensor devices that monitor a hazardous industrial zone;

performing two-dimensional (2D) imaging analysis on a set of pixels of the image data;

performing three-dimensional (3D) analysis on point cloud data comprising distance information for selected pixels of the image data calculated by the one or more imaging sensor devices;

classifying a first object in the image data as a human based on at least one of the 2D imaging analysis or the 3D analysis;

classifying a second object in the images data as motorized industrial equipment based on at least one of the 2D imaging analysis or the 3D analysis;

correlating results of the 2D imaging analysis and the 3D analysis to yield correlated information relating to a relationship between the human and the motorized equipment within the hazardous industrial zone; and

sending an instruction to alter operation of the motorized industrial equipment based on the correlated information.

14. The method of claim 13, further comprising identifying a location of the human relative to the motorized industrial equipment based on at least one of the 2D imaging analysis or the 3D analysis.

15. The method of claim 13, further comprising generating three-dimensional image data for the hazardous industrial zone based on aggregation of the point cloud data from the one or more imaging sensor devices.

16. The method of claim 13, further comprising:

monitoring one or more internal components of the one or more imaging sensor devices to detect a fault condition, and

generating a safety output in response to detection of the fault condition based on one or more safety algorithms.

17. The method of claim 13, wherein the correlating comprises determining, as the correlated information, at least one of a location of the human relative to the motorized industrial equipment, a velocity of the human, a velocity of the motorized industrial equipment, an acceleration of the human, an acceleration of the motorized industrial equipment, a trajectory of the human, a trajectory of the motorized industrial equipment, or an anticipated intersection between the trajectory of the human and the trajectory of the motorized industrial equipment.

18. The method of claim 13, further comprising selecting the selected pixels based on at least one of a location of the human relative to the motorized industrial equipment, a velocity of the motorized industrial equipment, or a trajectory of the motorized industrial equipment, and wherein the selected pixels comprise one of a single contiguous group of pixels or multiple non-contiguous groups of pixels.

19. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:

collecting image data from one or more imaging sensor devices monitoring a hazardous industrial area;

performing two-dimensional (2D) imaging analysis on one or more groups of pixels comprising the image data;

performing 3D (3D) analysis on point cloud data comprising distance information generated for selected pixels of the image data received from the one or more imaging sensor devices;

classifying a first object in the image data as a human based on at least one of the 2D imaging analysis or the 3D analysis;

classifying a second object in the image data as motorized industrial equipment based on at least one of the 2D imaging analysis or the 3D analysis;

correlating results of the 2D imaging analysis and the 3D analysis to yield correlated information relating to a relationship between the human and the motorized equipment within the hazardous industrial zone; and

sending an instruction to alter operation of the motorized industrial equipment based on the correlated information.

20. A non-transitory computer-readable medium of claim 19, wherein the correlated information comprises at least one of a location of the human relative to the motorized industrial equipment, a velocity of the human, a velocity of the motorized industrial equipment, an acceleration of the human, an acceleration of the motorized industrial equipment, a trajectory of the human, a trajectory of the motorized industrial equipment, or an anticipated intersection between the trajectory of the human and the trajectory of the motorized industrial equipment.