Safety monitoring system for human-machine symbiosis and method using the same

• US 9,333,652 B2
• Filed: 03/25/2014
• Issued: 05/10/2016
• Est. Priority Date: 11/11/2013
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Hu, Jhen-Jia, Wang, Hau-Wei, Huang, Chung-Ning
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Zohrabyan, Narek
• Application Number:
  US14/224,476
• Publication Number:
  US 20150131896A1
• Time in Patent Office: 777 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A safety monitoring system for human-machine symbiosis is provided, including a spatial image capturing unit, an image recognition unit, a human-robot-interaction safety monitoring unit, and a process monitoring unit. The spatial image capturing unit, disposed in a working area, acquires at least two skeleton images. The image recognition unit generates at least two spatial gesture images corresponding to the at least two skeleton images, based on information of changes in position of the at least two skeleton images with respect to time. The human-robot-interaction safety monitoring unit generates a gesture distribution based on the at least two spatial gesture images and a safety distance. The process monitoring unit determines whether the gesture distribution meets a safety criterion.

Claims

1. A safety monitoring method for human-machine symbiosis, comprising:

a spatial image capturing step for acquiring at least two skeleton images;

an image recognition step for generating at least two spatial gesture images corresponding to the at least two skeleton images, based on information of changes in position of the at least two skeleton images with respect to time;

a human-robot-interaction safety monitoring step for generating a gesture distribution based on the at least two spatial gesture images and a safety distance; and

a process monitoring step for determining whether the gesture distribution meets a safety criterion;

wherein the image recognition step comprises;

acquiring images for recognizing a human skeleton, a human skeleton dilation region, a robot skeleton, and a robot skeleton dilation region, to generate the at least two spatial gesture images.

2. The method according to claim 1, wherein the human-robot-interaction safety monitoring step sends a warning signal to a robot controller when the gesture distribution does not meet the safety criterion.

3. The method according to claim 2, further comprising a collision warning step and a trap warning step, wherein after the warning signal is sent, the collision warning step performs a collision warning instruction and the trap warning step performs a trap warning instruction.

4. The method according to claim 1, wherein the at least two spatial gesture images is represented by a contour map, and the human-robot-interaction safety monitoring step determines the gesture distribution, based on changes in vectors normal to a frontal surface of the contour map, and the frontal surface is an overlapping region of the at least two spatial gesture images.

5. The method according to claim 4, wherein the human-robot-interaction safety monitoring step includes:

acquiring a human skeleton contour map, a robot skeleton contour map, a human-robot-interaction contour map, and a human-robot-interaction contour gradient map;

determining, based on the human-robot-interaction contour map, whether there is a frontal surface of the human-robot-interaction contour map at a same height, and comparing the height at which the frontal surface is with a preset value; and

determining, based on the human-robot-interaction contour gradient map, whether any contour gradient vectors cross over.

6. The method according to claim 5, wherein if the contour map has the frontal surface existing at the same height, and the height of the frontal surface is equal to the preset value, it is determined that the gesture distribution does not meet the safety criterion.

7. The method according to claim 5, wherein the contour gradient vectors cross over, it is determined whether directional properties of velocity paths of points of the at least two spatial gesture images change or remain unchanged, so as to perform a collision warning step or a trap warning step.

8. The method according to claim 1, wherein the image recognition step further comprises performing an image pre-processing step for acquiring the information including a position, a continuous path, a velocity vector, a velocity path, an acceleration vector, and an acceleration path, with respect to a point in a skeleton space.

9. The method according to claim 8, wherein the human-robot-interaction safety monitoring step comprises:

determining, based on the continuous path with respect to the point of the skeleton space, whether historical paths of points of the at least two spatial gesture images cross over at a same position, and whether paths of the points of the at least two spatial gesture images cross over at a same time;

determining, based on the velocity path with respect to the point of the skeleton space, whether historical velocity paths of the points of the at least two spatial gesture images cross over, and whether directional properties of velocity paths of the points of the at least two spatial gesture images are parallel, non-parallel, or directionless; and

determining, based on the acceleration path with respect to the point of the skeleton space, whether historical acceleration paths of the points of the at least two spatial gesture images cross over, and whether acceleration paths of the points of the at least two spatial gesture images are greater than or equal to zero.

10. The method according to claim 9, wherein if the historical paths of the points of the at least two spatial gesture images cross over at the same position, and the paths of the points of the at least two spatial gesture images cross over at the same time, it is determined that the gesture distribution does not meet the safety criterion.

11. The method according to claim 9, wherein if the historical velocity paths of the points of the at least two spatial gesture images cross over and the directional properties of the velocity paths of the points of the at least two spatial gesture images are non-parallel, it is determined that the gesture distribution does not meet the safety criterion.

12. The method according to claim 9, wherein if the historical velocity paths of the points of the at least two spatial gesture images cross over, and the directional properties of the velocity paths of the points of the at least two spatial gesture images are directionless, it is further determined whether the acceleration paths of points of the at least two spatial gesture images are greater than or equal to zero.

13. The method according to claim 9, wherein if the historical velocity paths of the points of the at least two spatial gesture images cross over, and the directional properties of the velocity paths of the points of the at least two spatial gesture images are parallel, it is further determined whether the paths of the points of the at least two spatial gesture images cross over at the same time.

14. The method according to claim 9, wherein if the historical acceleration paths of the points of the at least two spatial gesture images cross over, it is further determined whether the acceleration paths of the points of the at least two spatial gesture images are greater than or equal to zero, so as to perform a collision warning step or a trap warning step, respectively.