Detecting and classifying workspace regions for safety monitoring

• US 10,099,372 B2
• Filed: 02/06/2018
• Issued: 10/16/2018
• Est. Priority Date: 02/07/2017
• Status: Active Grant

• Current Assignee: Veo Robotics, Inc.
• Original Assignee: Veo Robotics, Inc.
• Inventors: Vu, Clara, Denenberg, Scott, Sobalvarro, Patrick, Barragan, Patrick, Moel, Alberto
• Primary Examiner(s): Park, Edward
• Application Number:
  US15/889,523
• Publication Number:
  US 20180222050A1
• Time in Patent Office: 252 Days
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01S 17/04 Systems determining the presence of a target
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01V 8/20 using multiple transmitters or receivers
• CPC Class Codes: G05B 2219/40202 Human robot coexistence
• CPC Class Codes: G05B 2219/40203 Detect position of operator, create non material barrier to protect operator
• CPC Class Codes: G05B 2219/40422 Force controlled velocity motion planning, adaptive
• CPC Class Codes: G05B 2219/40577 Multisensor object recognition
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: Y10S 901/47 Optical
• CPC Class Codes: Y10S 901/49 Protective device

Abstract

Systems and methods monitor a workspace for safety purposes using sensors distributed about the workspace. The sensors are registered with respect to each other, and this registration is monitored over time. Occluded space as well as occupied space is identified, and this mapping is frequently updated.

Claims

1. A safety system for identifying safe regions in a three-dimensional workspace including controlled machinery, the system comprising:

a plurality of sensors distributed about the workspace, each of the sensors being associated with a grid of pixels for recording images of a portion of the workspace within a sensor field of view, the workspace portions partially overlapping with each other;

a controller configured to;

register the sensors with respect to each other so that the images obtained by the sensors collectively represent the workspace;

generate a three-dimensional representation of the workspace as a plurality of volumes;

for each sensor pixel having an intensity level above a threshold value, preliminarily marking as unoccupied volumes intercepted by a line-of-sight ray path through the pixel and terminating at an estimated distance from the associated sensor of an occlusion, marking as occupied the volumes corresponding to a terminus of the ray path, and marking as unknown any volumes beyond the occlusion along the ray path;

for each sensor pixel having an intensity level below the threshold value, preliminarily marking as unknown all voxels intercepted by a line-of-sight ray path through the pixel and terminating at a boundary of the workspace;

finally marking as unoccupied volumes that have been preliminarily marked at least once as unoccupied; and

mapping one or more safe volumetric zones within the workspace, the volumetric zones being outside a safety zone of the machinery and including only volumes marked as unoccupied.

2. The safety system of claim 1, wherein the points are voxels.

3. The safety system of claim 1, wherein the safety zone is a 3D volume surrounding at least a portion of the machinery.

4. The safety system of claim 3, wherein the controller is responsive to real-time monitoring of the workspace by the sensors and is configured to alter operation of the machinery in response to an intrusion into the safety zone detected by the sensors.

5. The safety system of claim 4, wherein the safety zone is divided into a plurality of nested subzones within the 3D volume, a detected intrusion into each of the subzones resulting in a different degree of alteration of the operation of the machinery.

6. The safety system of claim 4, wherein the controller is configured to computationally extend the intrusion into the workspace in accordance with a model of human movement.

7. The safety system of claim 1, wherein the sensors are 3D sensors.

8. The safety system of claim 7, wherein at least some of the sensors are time-of-flight cameras.

9. The safety system of claim 7, wherein at least some of the sensors are 3D LIDAR sensors.

10. The safety system of claim 7, wherein at least some of the sensors are stereo vision cameras.

11. The safety system of claim 1, wherein the controller is configured to recognize a workpiece being handled by the machinery and treat the workpiece as a portion thereof in generating the safety zone.

12. A method of safely operating machinery in a three-dimensional workspace, the method comprising the steps of:

monitoring the workspace with a plurality of sensors distributed thereabout, each of the sensors being associated with a grid of pixels for recording images of a portion of the workspace within a sensor field of view, the workspace portions partially overlapping with each other;

registering the sensors with respect to each other so that the images obtained by the sensors collectively represent the workspace;

computationally generating a three-dimensional representation of the workspace stored in a computer memory;

for each sensor pixel having an intensity level above a threshold value, preliminarily marking as unoccupied, in the computer memory, volumes intercepted by a line-of-sight ray path through the pixel and terminating at an estimated distance from the associated sensor of an occlusion, marking as occupied the volumes corresponding to a terminus of the ray path, and marking as unknown any volumes beyond the occlusion along the ray path;

for each sensor pixel having an intensity level below the threshold value, preliminarily marking as unknown all volumes intercepted by a line-of-sight ray path through the pixel and terminating at a boundary of the workspace;

finally marking as unoccupied volumes that have been preliminarily marked at least once as unoccupied; and

computationally mapping one or more safe volumetric zones within the workspace, the volumetric zones being outside a safety zone of the machinery and including only volumes marked as unoccupied.

13. The method of claim 12, wherein the points are voxels.

14. The method of claim 12, wherein the safety zone is a 3D volume surrounding at least a portion of the machinery.

15. The method of claim 14, further comprising the step of responding to a detected intrusion in to the safety zone by altering operation of the machinery in response.

16. The method of claim 15, wherein the safety zone is divided into a plurality of nested subzones within the 3D volume, a detected intrusion into each of the subzones resulting in a different degree of alteration of the operation of the machinery.

17. The method of claim 15, further comprising the step of computationally extending the intrusion into the workspace in accordance with a model of human movement.

18. The method of claim 12, further comprising the step of computationally recognizing a workpiece being handled by the machinery and treating the workpiece as a portion thereof in generating the safety zone.

19. The method of claim 18, wherein the computational recognition step is performed using a neural network.

20. The method of claim 12, further comprising the step of generating safe action constraints for the machinery and controlling the machinery in accordance therewith.

21. The method of claim 12, wherein the machinery is at least one robot.