Integration of optical area monitoring with industrial machine control

• US 9,477,907 B2
• Filed: 11/17/2015
• Issued: 10/25/2016
• Est. Priority Date: 05/19/2014
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Galera, Richard, Bowlby, Anne, Jones, Derek W., Pradhan, Nilesh, Leard, Francis L.
• Primary Examiner(s): Liew, Alex
• Application Number:
  US14/943,246
• Publication Number:
  US 20160070991A1
• Time in Patent Office: 343 Days
• Field of Search: 382141-152
• US Class Current: 1/1
• CPC Class Codes: F16P 3/142 using image capturing devices
• CPC Class Codes: G05B 19/048 Monitoring; Safety
• CPC Class Codes: G05B 2219/14006 Safety, monitoring in general
• CPC Class Codes: G05B 9/03 with multiple-channel loop, i.e. redundant control systems
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An industrial safety system is provided that integrates optical safety monitoring with machine control. The safety system includes an imaging sensor device supporting pixel array processing functions that allow time-of-flight (TOF) analysis to be performed on selected portions of the pixel array, while two-dimensional imaging analysis is performed on the remaining portions of the array, reducing processing load and response time relative to performing TOF analysis for all pixels of the array. The portion of the pixel array designated for TOF analysis can be pre-defined through configuration of the imaging sensor device, or can be dynamically selected based on object detection and classification by the two-dimensional imaging analysis. The imaging sensor device can also implement a number of safety and redundancy functions to achieve a high degree of safety integrity.

Claims

1. An optical safety system, comprising:

a memory that stores executable instructions; and

a processor, communicatively coupled to the memory, that executes or facilitates execution of the executable instructions to at least;

capture image data representing an industrial area;

classify a first object represented in the image data as industrial equipment and a second object represented in the image data as a human based on a first result of a first type of analysis performed on a first subset of pixels of the image data and a second result of a second type of analysis performed on a second subset of the pixels;

correlate the first result and the second result to yield correlated information relating to a relationship between the human and the industrial equipment; and

control an output signal that controls the industrial equipment based on the correlated information.

2. The optical safety system of claim 1, wherein the first type of analysis comprises a two-dimensional (2D) analysis and the second type of analysis comprises a three-dimensional (3D) analysis.

3. The optical safety system of claim 2, wherein the processor further executes or facilitates execution of the executable instructions to at least, for one or more pixel arrays of the image data, group pixels of the one or more pixel arrays to yield the second subset of the pixels.

4. The optical safety system of claim 3, wherein the processor further executes or facilitates execution of the executable instructions to at least identify the second subset of the pixels based on a configuration profile that defines one or more portions of the one or more pixel arrays on which the 3D analysis is to be performed, and wherein the second subset of the pixels comprises one of a single contiguous group of pixels or multiple non-contiguous groups of pixels.

5. The optical safety system of claim 3, wherein the processor further executes or facilitates execution of the executable instructions to at least select the second subset of the pixels based on classifications of the first object and the second object and locations of the first object and the second object determined based on the 2D analysis, and wherein the second subset of the pixels comprises one of a single contiguous group of pixels or multiple non-contiguous groups of pixels.

6. The optical safety system of claim 3, wherein the processor further executes or facilitates execution of the executable instructions to at least:

generate three-dimensional image information for the industrial area based on aggregation of pixel-level distance information collected by one or more imaging sensor devices for the second subset of the pixels; and

perform the 3D analysis on the three-dimensional image information.

7. The optical safety system of claim 2, wherein the 2D analysis comprises at least one of object recognition, object classification, determination of an x-y location of at least one of the first object or the second object, correlation of pixel groups within the image data determined to belong to a common object, human recognition, or facial recognition.

8. The optical safety system of claim 2, wherein the correlated information comprises at least one of a three-dimensional location of the second object relative to the first object, a three-dimensional velocity of the second object relative to the first object, a three-dimensional acceleration of the second object relative to the first object, or a three-dimensional trajectory of the second object relative to the first object.

9. The optical safety system of claim 6, wherein the processor further executes or facilitates execution of the executable instructions to at least:

estimate the first object and the second object as respective three-dimensional geometric objects within a three-dimensional space represented by the three-dimensional image data, wherein the geometric objects respectively encapsulate the first object and the second object,

determine a minimum distance between the geometric objects to yield an estimated distance between the first object and the second object; and

control the output signal based at least in part on the minimum distance.

10. A method for controlling industrial equipment, comprising:

recording, by a safety system comprising at least one processor, image data representing an industrial area;

classifying, by the safety system, a first object recorded in the image data as an industrial machine and a second object recorded in the image data as a human based on a first result of a first type of analysis performed on a first subset of pixels of the image data and a second result of a second type of analysis performed on a second subset of the pixels;

correlating, by the safety system, the first result and the second result to yield correlated information defining a relationship between the human and the industrial machine; and

controlling, by the safety system, an output signal that controls the industrial machine based on the correlated information.

11. The method of claim 10, further comprising:

performing, as the first type of analysis, a two-dimensional (2D) analysis on the first subset of the pixels; and

performing, as the second type of analysis, a three-dimensional (3D) analysis on the second subset of the pixels.

12. The method of claim 10, wherein the controlling comprises at least one of initiating a removal of power from the industrial machine or initiating a change of an operating mode of the industrial machine.

13. The method of claim 10, further comprising, for one or more pixel arrays of the image data, grouping pixels of the one or more pixel arrays to yield the second subset of the pixels.

14. The method of claim 11, further comprising identifying, by the safety system, the second subset of the pixels based on configuration profile data that defines one or more portions of the one or more pixel arrays on which the 3D analysis is to be performed, wherein the second subset of the pixels comprises one of a single contiguous group of pixels or multiple non-contiguous groups of pixels.

15. The method of claim 11, further comprising:

determining, by the safety system, respective locations of the first object and the second object based on the first result of the 2D analysis; and

selecting, by the safety system, the second subset of the pixels based on the classifying and locations.

16. The method of claim 11, further comprising:

generating, by the safety system, three-dimensional image information for the industrial area based on aggregation of pixel-level distance information collected by one or more imaging sensor devices for the second subset of the pixels; and

performing, by the safety system, the 3D analysis on the three-dimensional image information.

17. The method of claim 11, further comprising performing, by the safety system as the 2D analysis, at least one of object recognition, object classification, determination of an x-y location of at least one of the first object or the second object, correlation of pixel groups within the image data determined to belong to a common object, human recognition, or facial recognition.

18. The method of claim 11, wherein the correlating comprises determining, as the correlated information, at least one of a three-dimensional location of the second object relative to the first object, a three-dimensional velocity of the second object relative to the first object, a three-dimensional acceleration of the second object relative to the first object, or a three-dimensional trajectory for the second object relative to the first object.

19. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:

collecting image data representing an industrial area;

classifying a first object recorded in the image data as industrial equipment and a second object recorded in the image data as a human based on a first result of a first type of analysis performed on a first subset of pixels of the image data and a second result of a second type of analysis performed on a second subset of the pixels;

generate correlated information based on one or more correlations between the first result and the second result, the correlated information defining a relationship between the human and the industrial equipment; and

controlling an output signal that controls the industrial equipment based on the correlated information.

20. The non-transitory computer-readable medium of claim 19, the operations further comprising:

performing, as the first type of analysis, a two-dimensional analysis; and

performing, as the second type of analysis, a three-dimensional analysis.