SAFETY MONITORING SYSTEM FOR HUMAN-MACHINE SYMBIOSIS AND METHOD USING THE SAME

• US 20150131896A1
• Filed: 03/25/2014
• Published: 05/14/2015
• Est. Priority Date: 11/11/2013
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Hu, Jhen-Jia, Wang, Hau-Wei, Huang, Chung-Ning
• Granted Patent: US 9,333,652 B2
• Time in Patent Office: Days
• US Class Current: 382/153
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A safety monitoring system for human-machine symbiosis is provided, including a spatial image capturing unit, an image recognition unit, a human-robot-interaction safety monitoring unit, and a process monitoring unit. The spatial image capturing unit, disposed in a working area, acquires at least two skeleton images. The image recognition unit generates at least two spatial gesture images corresponding to the at least two skeleton images, based on information of changes in position of the at least two skeleton images with respect to time. The human-robot-interaction safety monitoring unit generates a gesture distribution based on the at least two spatial gesture images and a safety distance. The process monitoring unit determines whether the gesture distribution meets a safety criterion.

Claims

1. A safety monitoring system for human-machine symbiosis, comprising:

a spatial image capturing unit, disposed in a working area, is configured to acquire at least two skeleton images;

an image recognition unit is configured to generate at least two spatial gesture images corresponding to the at least two skeleton images, based on information of changes in position of the at least two skeleton images with respect to time;

a human-robot-interaction safety monitoring unit is configured to generate a gesture distribution based on the at least two spatial gesture images and a safety distance; and

a process monitoring unit is configured to determine whether the gesture distribution meets a safety criterion.

2. The system according to claim 1, wherein the human-robot-interaction safety monitoring unit sends a warning signal to a robot controller when the gesture distribution does not meet the safety criterion.

3. The system according to claim 2, wherein the robot controller comprises a collision warning module and a trap warning module, wherein when the robot controller receives the warning signal, the collision warning module performs an instruction to avoid collision and the trap warning module performs an instruction to escape being trapped.

4. The system according to claim 1, further comprising an image database, coupled to the human-robot-interaction safety monitoring unit, is configured to store an image of the gesture distribution occurred within a preset time period.

5. The system according to claim 1, wherein the human-robot-interaction safety monitoring unit uses a contour map representing the at least two spatial gesture images, and the human-robot-interaction safety monitoring unit determines the gesture distribution, based on changes in vectors normal to a frontal surface of the contour map, and the frontal surface is an overlapping region of the at least two spatial gesture images.

6. The system according to claim 1, wherein the image recognition unit generates an expanded skeleton image.

7. The system according to claim 6, wherein the image recognition unit recognizes gestures of the head, hand(s), arm(s) of a human and gestures of arm(s) of a robot, based on the expanded skeleton image.

8. The system according to claim 6, wherein the human-robot-interaction safety monitoring unit includes a working-area dynamic monitoring unit for monitoring human and robot working areas, wherein the working area dynamic monitoring unit determines a protection boundary value with respect to human and robot, based on the expanded skeleton image.

9. The system according to claim 1, wherein the information includes a human movement path and a robot movement path.

10. The system according to claim 9, wherein the human movement path includes a human position path, a human velocity path, and a human acceleration path, and the robot movement path includes a robot position path, a robot velocity path, and a robot acceleration path;

and the human-robot-interaction safety monitoring unit determines whether the human movement path and the robot movement path overlap, based on the human position path, the human velocity path, the human acceleration path, the robot position path, the robot velocity path, and the robot acceleration path.

11. The system according to claim 10, wherein the human-robot-interaction safety monitoring unit determines that historical paths of points of the at least two spatial gesture images cross over at a same position, and further determines whether paths of the points of the at least two spatial gesture images cross over at a same time;

if so, the gesture distribution does not meet the safety criterion;

if not, the gesture distribution meets the safety criterion.

12. A safety monitoring method for human-machine symbiosis, comprising:

a spatial image capturing step for acquiring at least two skeleton images;

an image recognition step for generating at least two spatial gesture images corresponding to the at least two skeleton images, based on information of changes in position of the at least two skeleton images with respect to time;

a human-robot-interaction safety monitoring step for generating a gesture distribution based on the at least two spatial gesture images and a safety distance; and

a process monitoring step for determining whether the gesture distribution meets a safety criterion.

13. The method according to claim 12, wherein the human-robot-interaction safety monitoring step sends a warning signal to a robot controller when the gesture distribution does not meet the safety criterion.

14. The method according to claim 13, further comprising a collision warning step and a trap warning step, wherein after the warning signal is sent, the collision warning step performs a collision warning instruction and the trap warning step performs a trap warning instruction.

15. The method according to claim 12, wherein the image recognition step comprises:

acquiring images for recognizing a human skeleton, a human skeleton dilation region, a robot skeleton, and a robot skeleton dilation region, to generate the at least two spatial gesture images.

16. The method according to claim 15, wherein the at least two spatial gesture images is represented by a contour map, and the human-robot-interaction safety monitoring step determines the gesture distribution, based on changes in vectors normal to a frontal surface of the contour map, and the frontal surface is an overlapping region of the at least two spatial gesture images.

17. The method according to claim 16, wherein the human-robot-interaction safety monitoring step includes:

acquiring a human skeleton contour map, a robot skeleton contour map, a human-robot-interaction contour map, and a human-robot-interaction contour gradient map;

determining, based on the human-robot-interaction contour map, whether there is a frontal surface of the human-robot-interaction contour map at a same height, and comparing the height at which the frontal surface is with a preset value; and

determining, based on the human-robot-interaction contour gradient map, whether any contour gradients vector cross over.

18. The method according to claim 17, wherein if the contour map has the frontal surface existing at the same height, and the height of the frontal surface is equal to the preset value, it is determined that the gesture distribution does not meet the safety criterion.

19. The method according to claim 17, wherein the contour gradient vector crosses over, it is determined whether directional properties of velocity paths of points of the at least two spatial gesture images change or remain unchanged, so as to perform a collision warning step or a trap warning step.

20. The method according to claim 15, wherein the image recognition step further comprises performing an image pre-processing step for acquiring the information including a position, a continuous path, a velocity vector, a velocity path, an acceleration vector, and an acceleration path, with respect to a point in a skeleton space.

21. The method according to claim 20, wherein the human-robot-interaction safety monitoring step comprises:

determining, based on the continuous path with respect to the point of the skeleton space, whether historical paths of points of the at least two spatial gesture images cross over at a same position, and whether paths of the points of the at least two spatial gesture images cross over at a same time;

determining, based on the velocity path with respect to the point of the skeleton space, whether historical velocity paths of the points of the at least two spatial gesture images cross over, and whether directional properties of velocity paths of the points of the at least two spatial gesture images are parallel, non-parallel, or directionless; and

determining, based on the acceleration path with respect to the point of the skeleton space, whether historical acceleration paths of the points of the at least two spatial gesture images cross over, and whether acceleration paths of the points of the at least two spatial gesture images are greater than or equal to zero.

22. The method according to claim 21, wherein if the historical paths of the points of the at least two spatial gesture images cross over at the same position, and the paths of the points of the at least two spatial gesture images cross over at the same time, it is determined that the gesture distribution does not meet the safety criterion.

23. The method according to claim 21, wherein if the historical velocity paths of the points of the at least two spatial gesture images cross over and the directional properties of the velocity paths of the points of the at least two spatial gesture images are non-parallel, it is determined that the gesture distribution does not meet the safety criterion,

24. The method according to claim 21, wherein if the historical velocity paths of the points of the at least two spatial gesture images cross over, and the directional properties of the velocity paths of the points of the at least two spatial gesture images are directionless, it is further determined whether the acceleration paths of points of the at least two spatial gesture images are greater than or equal to zero.

25. The method according to claim 21, wherein if the historical velocity paths of the points of the at least two spatial gesture images cross over, and the directional properties of the velocity paths of the points of the at least two spatial gesture images are parallel, it is further determined whether the paths of the points of the at least two spatial gesture images cross over at the same time. 26, The method according to claim 21, wherein if the historical acceleration paths of the points of the at least two spatial gesture images cross over, it is further determined whether the acceleration paths of the points of the at least two spatial gesture images are greater than or equal to zero, so as to perform a collision warning step or a trap warning step, respectively.