Controlling a robot in the presence of a moving object

• US 9,452,531 B2
• Filed: 02/04/2014
• Issued: 09/27/2016
• Est. Priority Date: 02/04/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kikkeri, Harshavardhana, Thibodeau, Bryan J.
• Primary Examiner(s): HOLLOWAY, JASON R
• Application Number:
  US14/172,230
• Publication Number:
  US 20150217455A1
• Time in Patent Office: 966 Days
• Field of Search: 700/246, 700/248, 700/250, 700/253, 700/255, 700/262, 901/15, 901/49
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: F16P 3/142 using image capturing devices
• CPC Class Codes: G05B 2219/37567 3-D vision, stereo vision, with two cameras
• CPC Class Codes: G05B 2219/37571 Camera detecting reflected light from laser
• CPC Class Codes: G05B 2219/39091 Avoid collision with moving obstacles
• CPC Class Codes: G05B 2219/40202 Human robot coexistence
• CPC Class Codes: G05B 2219/40203 Detect position of operator, create non material barrier to protect operator
• CPC Class Codes: G05B 2219/49152 Feedhold, stop motion if machine door is open, if operator in forbidden zone
• CPC Class Codes: G05B 2219/49158 On near collision reduce speed
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A method, system, and one or more computer-readable storage media for controlling a robot in the presence of a moving object are provided herein. The method includes capturing a number of frames from a three-dimensional camera system and analyzing a frame to identify a connected object. The frame is compared to a previous frame to identify a moving connected object (MCO). If an unexpected MCO is in the frame a determination is made if the unexpected MCO is in an actionable region. If so, the robot is instructed to take an action.

Claims

1. A method, performed by a computer processor, for controlling a robot in a presence of a moving object, comprising:

capturing a plurality of frames from a three-dimensional camera system;

analyzing a frame to identify a connected object comprising substantially proximate voxels;

comparing a location of the connected object in the frame to a location of the connected object in a previous frame to determine motion vectors for the substantially proximate voxels;

comparing the frame to the previous frame to identify a moving connected object (MCO) comprising the substantially proximate voxels, wherein the motion vectors for the substantially proximate voxels are consistent;

determining if an unexpected MCO is in the frame; and

, if so,

instructing the robot to take an action.

2. The method of claim 1, wherein analyzing a frame comprises forming a group comprising the substantially proximate voxels to form connected objects.

3. The method of claim 2, wherein comparing the frame to a previous frame comprises:

comparing a location of a group of voxels forming a connected object to a previous location of the group of voxels forming the connected object;

eliminating any voxels that have inconsistent motion vectors; and

defining remaining voxels in the group of voxels as the MCO.

4. The method of claim 1, comprising determining an expected motion of the robot from a robot controller to identify the robot as an expected MCO.

5. The method of claim 1, comprising removing an expected MCO from the frame before determining if the unexpected MCO is in the frame.

6. The method of claim 1, comprising determining if the unexpected MCO is in an actionable region.

7. The method of claim 6, comprising activating a visual warning, an audible warning, or both if the actionable region is a warning zone or a hazard zone.

8. The method of claim 6, comprising slowing a motion of the robot if the actionable region is a warning zone.

9. The method of claim 6, comprising stopping a motion of the robot if the unexpected MCO if the actionable region is a hazard zone.

10. The method of claim 6, comprising slowing a motion of the robot and moving the robot towards the unexpected MCO to allow interaction with the unexpected MCO.

11. The method of claim 1, comprising controlling a motion of auxiliary equipment based, at least in part, on a presence of an unexpected MCO.

12. A system for controlling a robot in a presence of a moving object, comprising:

a three dimensional camera;

a processor; and

a system memory comprising code that, when executed by the processor, is configured to;

collect a plurality of frames from the three dimensional camera;

analyze a frame to identify connected objects comprising substantially proximate voxels;

compare a location of the connected object in the frame to a location of the connected object in a previous frame to determine motion vectors for the substantially proximate voxels;

compare the frame to a previous frame to identify a moving connected object (MCO) comprising the substantially proximate voxels, wherein the motion vectors for the substantially proximate voxels are consistent;

determine if the MCO is not expected to be present; and

, if so,

send a signal to a robot controller, the robot controller causing the robot to take an action based on the signal.

13. The system of claim 12, comprising a module adapted to be placed in proximate to a robot and communicatively coupled to the robot controller.

14. The system of claim 12, comprising a robot.

15. The system of claim 12, wherein the system memory comprises code that, when executed by the processor, is configured to communicate a stop motion signal to the robot if an unexpected MCO enters a hazardous region.

16. The system of claim 12, wherein the system memory comprises code that, when executed by the processor, is configured to communicate a slow motion signal to the robot if an unexpected MCO enters a warning region.

17. The system of claim 12, comprising a visual warning system, an audible warning system, or both.

18. The system of claim 17, wherein the system memory comprises code that, when executed by the processor, is configured to activate the visual warning system, the audible warning system, or both, when an unexpected MCO enters a hazardous region or a warning region.

19. One or more computer-readable storage media for storing computer-readable instructions, the computer-readable instructions providing a system for controlling a robot in a presence of a moving object, the computer-readable instructions being executable by a computer processor to:

capture a plurality of frames from a three-dimensional camera system;

analyze a frame to identify a connected object comprising substantially proximate voxels;

compare a location of the connected object in the frame to a location of the connected object in a previous frame to determine motion vectors for the substantially proximate voxels;

compare the frame to a previous frame to identify a moving connected object (MCO) comprising the substantially proximate voxels, wherein the motion vectors for the substantially proximate voxels are having consistent motion vectors;

determine if an unexpected MCO is in the frame; and

, if so,

instruct a robot to take an action.

20. The one or more computer-readable storage media of claim 19, wherein the computer-readable instructions comprise code configured to stop a motion of the robot if the unexpected MCO is on a hazardous region.