Controlling a robot in the presence of a moving object

• US 10,195,741 B2
• Filed: 07/29/2016
• Issued: 02/05/2019
• Est. Priority Date: 02/04/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kikkeri, Harshavardhana, Thibodeau, Bryan J.
• Primary Examiner(s): Holloway, Jason
• Application Number:
  US15/224,109
• Publication Number:
  US 20160354927A1
• Time in Patent Office: 921 Days
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: F16P 3/142 using image capturing devices
• CPC Class Codes: G05B 2219/37567 3-D vision, stereo vision, with two cameras
• CPC Class Codes: G05B 2219/37571 Camera detecting reflected light from laser
• CPC Class Codes: G05B 2219/39091 Avoid collision with moving obstacles
• CPC Class Codes: G05B 2219/40202 Human robot coexistence
• CPC Class Codes: G05B 2219/40203 Detect position of operator, create non material barrier to protect operator
• CPC Class Codes: G05B 2219/49152 Feedhold, stop motion if machine door is open, if operator in forbidden zone
• CPC Class Codes: G05B 2219/49158 On near collision reduce speed
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A method, system, and one or more computer-readable storage media for controlling a robot in the presence of a moving object are provided herein. The method includes capturing a number of frames from a three-dimensional camera system and analyzing a frame to identify a connected object. The frame is compared to a previous frame to identify a moving connected object (MCO). If an unexpected MCO is in the frame a determination is made if the unexpected MCO is in an actionable region. If so, the robot is instructed to take an action.

Claims

1. A method, performed by a processor, for controlling a robot in the presence of a moving object, comprising:

analyzing a frame captured by a three-dimensional camera to identify a connected object comprising substantially proximate voxels;

determining motion vectors for the substantially proximate voxels based at least in part on the frame and a previous frame captured by the three-dimensional camera;

identifying a moving connected object (MCO) comprising the substantially proximate voxels, based at least in part on whether the motion vectors for the substantially proximate voxels are consistent;

determining if the MCO is an unexpected MCO based at least in part on an expected motion of the robot; and

instructing the robot to perform an action in response to determining the MCO is unexpected.

2. The method of claim 1, wherein determining if the MCO is unexpected is based at least in part on auxiliary objects within the frame.

3. The method of claim 1, wherein analyzing the frame comprises forming a group comprising the substantially proximate voxels to form connected objects.

4. The method of claim 1, comprising determining if the unexpected MCO is in an actionable region.

5. The method of claim 4, wherein the action taken is to activate an audible warning if the unexpected MCO is in the actionable region.

6. The method of claim 4, wherein the action taken is to activate a visual warning if the unexpected MCO is in the actionable region.

7. The method of claim 4, comprising slowing a motion of the robot if the unexpected MCO is in a warning zone.

8. The method of claim 4, comprising stopping a motion of the robot if the unexpected MCO is in a hazard zone.

9. The method of claim 8, comprising slowing a motion of the robot and moving the robot towards the unexpected MCO to allow interaction with the unexpected MCO.

10. The method of claim 2, wherein identifying the MCO comprises:

comparing a location of a group of voxels forming a connected object to a previous location of the group of voxels forming the connected object;

eliminating any voxels that have inconsistent motion vectors; and

defining remaining voxels in the group of voxels as the MCO.

11. A system for controlling a robot in a presence of a moving object, comprising:

a three dimensional camera;

a processor; and

a system memory comprising code that, based at least in part on execution by the processor, is configured to;

analyze a frame captured by a three-dimensional camera to identify an object in relative motion to the robot;

determine motion vectors for voxels representing the object based at least in part on the frame and a previous frame captured by the three-dimensional camera;

group proximate voxels having consistent motion vectors;

determine if the object is unexpected based at least in part on an evaluation of expected relative motion of the robot and the grouped proximate voxels; and

instruct the robot to take an action if the object is determined to be unexpected.

12. The system of claim 11, the robot being in motion.

13. The system of claim 11, the object being in motion.

14. The system of claim 11, the object being a moving connected object.

15. The system of claim 11, the three-dimensional camera being part of the robot.

16. The system of claim 11, wherein the system memory comprises code that, based at least in part on execution by the processor, is configured to activate a visual warning system, an audible warning system, or both, when the object enters a hazardous region or a warning region.

17. One or more computer-readable storage media for storing computer-readable instructions, the computer-readable instructions providing a system for controlling a robot in a presence of a moving object, the computer-readable instructions, based at least in part on execution by a processor, being configured to:

capture a plurality of frames from a three-dimensional camera system;

analyze one of the frames to identify a plurality of connected objects, each of the connected objects comprising substantially proximate voxels;

compare a location of each of the connected objects in the one frame to a location of each of the connected objects in a previous frame of the frames to determine motion vectors for the substantially proximate voxels;

compare the one frame to the previous frame to identify a moving connected object (MCO) comprising the substantially proximate voxels, wherein the motion vectors for the substantially proximate voxels are consistent;

determine if the MCO is an unexpected MCO based at least in part on an expected motion of the robot; and

instruct the robot to take an action.

18. The one or more computer-readable storage media of claim 17, wherein the computer-readable instructions, based at least in part on execution by the processor, are configured to stop a motion of the robot if the unexpected MCO is in a hazardous region.