Position-controlled robotic fleet with visual handshakes

• US 9,465,390 B2
• Filed: 11/11/2014
• Issued: 10/11/2016
• Est. Priority Date: 11/11/2014
• Status: Active Grant

• Current Assignee: Boston Dynamics, Inc. (SoftBank Group Corp.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mason, Julian, Konolige, Kurt
• Primary Examiner(s): Anwari, Maceeh
• Application Number:
  US14/538,047
• Publication Number:
  US 20160132059A1
• Time in Patent Office: 700 Days
• Field of Search: 701/28
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 1/1373 for fulfilling orders in warehouses
• CPC Class Codes: G05B 19/41815 characterised by the cooperation between machine tools, manipulators and conveyor or other workpiece supply system, workcell
• CPC Class Codes: G05B 2219/39129 One manipulator holds one piece, other inserts, screws other piece, dexterity
• CPC Class Codes: G05B 2219/39146 Swarm, multiagent, distributed multitask fusion, cooperation multi robots
• CPC Class Codes: G05B 2219/40605 Two cameras, each on a different end effector to measure relative position
• CPC Class Codes: G05D 1/0234 using optical markers or beacons optical beacons per se G01S1/70
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0287 involving a plurality of land vehicles, e.g. fleet or convoy travelling traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16
• CPC Class Codes: G05D 1/0291 Fleet control monitoring fleets in traffic control systems for road vehicles G08G1/127, G08G1/127
• CPC Class Codes: G05D 2201/0216 Vehicle for transporting goods in a warehouse, factory or similar
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

Example methods and systems may provide for a system that includes a control system communicatively coupled to a first robotic device and a second robotic device. The control system may identify a collaborative operation to be performed by a first robotic device and a second robotic device that is based on a relative positioning between the first robotic device and the second robotic device. The control system may also determine respective locations of the first robotic device and the second robotic device. The control system may further initiate a movement of the first robotic device along a path from the determined location of the first robotic device towards the determined location of the second robotic device. The first robotic device and the second robotic device may then establish a visual handshake that indicates the relative positioning between the first robotic device and the second robotic device for the collaborative operation.

Claims

1. A method comprising:

identifying, by a control system, a collaborative operation to be performed by a first robotic device and a second robotic device, wherein the collaborative operation is based on a relative positioning between the first robotic device and the second robotic device;

based on data representing a map of an environment in which the first robotic device and the second robotic device are operating, determining, by the control system, respective locations, relative to the map, of the first robotic device and the second robotic device, wherein the map is updated in response to movements of the first robotic device and the second robotic device relative to the environment;

initiating, by the control system, a movement of the first robotic device along a path from the determined respective location of the first robotic device relative to the map towards the determined respective location relative to the map of the second robotic device; and

after initiating the movement, causing, by the control system, the first robotic device and the second robotic device to establish a visual handshake that indicates the relative positioning between the first robotic device and the second robotic device for the collaborative operation.

2. The method of claim 1, wherein causing the first robotic device and the second robotic device to establish the visual handshake comprises:

causing the second robotic device to orient a camera along the path from the determined respective location of the first robotic device relative to the map towards the determined respective location of the second robotic device.

3. The method of claim 1, wherein causing the first robotic device and the second robotic device to establish the visual handshake comprises:

causing the first robotic device to orient a camera towards the determined respective location of the second robotic device.

4. The method of claim 1, wherein causing the first robotic device and the second robotic device to establish the visual handshake comprises:

causing a camera of the first robotic device to generate data representing images from a viewpoint that is oriented towards the determined respective location of the second robotic device relative to the map;

based on the data representing the images from the viewpoint that is oriented towards the determined respective location of the second robotic device relative to the map, detecting one or more particular images that depict a tag on an exterior surface of the second robotic device; and

based on respective sizes and positions of the tag within the one or more particular images, determining the relative positioning between the first robotic device and the second robotic device.

5. The method of claim 1, wherein the collaborative operation involves positioning the first robotic device at a particular distance from the second robotic device, the method further comprising:

causing at least one of the first robotic device and the second robotic device to move from the relative positioning indicated by the visual handshake to a relative positioning in which the first robotic device is at the particular distance from the second robotic device.

6. The method of claim 1, wherein the collaborative operation involves positioning the first robotic device at a particular orientation with respect to the second robotic device, the method further comprising:

causing at least one of the first robotic device and the second robotic device to move from the relative positioning indicated by the visual handshake to a relative positioning in which the first robotic device is at the particular orientation with respect to the second robotic device.

7. The method of claim 1, wherein the collaborative operation further involves positioning an end-effector of the first robotic device at a particular position relative to an end-effector of the second robotic device, and wherein the visual handshake further indicates a relative positioning between the end-effector of the first robotic device and the end-effector of the second robotic device.

8. The method of claim 1, wherein the collaborative operation involves navigating the first robotic device along a path that intersects a given proximity of the second robotic device, the method further comprising:

based on the relative positioning between the first robotic device and the second robotic device indicated by the visual handshake, navigating the first robotic device past the second robotic device such that the first robotic device is at least a threshold distance from the second robotic device during the navigation.

9. The method of claim 1, further comprising:

after establishing the visual handshake that indicates the relative positioning between the first robotic device and the second robotic device, transferring movement control of the first robotic device from the control system to a local control system of first robotic device.

10. A first robotic device comprising:

a communications interface;

one or more processors; and

data storage configured to store instructions, that when executed by the one or more processors, cause the first robotic device to;

receive instructions to perform a collaborative operation that is based on a relative positioning between the first robotic device and a second robotic device;

receive, over the communications interface, data indicating a path towards the second robotic device, wherein the path is determined based on data representing a map of an environment in which the first robotic device and the second robotic device are operating, and wherein the map is updated in response to movements of the first robotic device and the second robotic device relative to the environment;

responsive to receiving the data indicating the path towards the second robotic device, initiate movement of the first robotic device along the path determined based on data representing the map towards the second robotic device; and

after initiating the movement of the first robotic device along the path, establish a visual handshake with the second robotic device that indicates the relative positioning between the first robotic device and the second robotic device for the collaborative operation.

11. The first robotic device of claim 10, wherein the instructions further cause the first robotic device to:

receive data indicating a direction in which the second robotic device is located relative to the first robotic device; and

orient a camera in the direction in which the second robotic device is located relative to the first robotic device.

12. The first robotic device of claim 10, wherein the first robotic device further comprises a camera, and wherein the instructions further cause the first robotic device to:

cause the camera to generate data representing images from a viewpoint that is oriented in the direction in which the second robotic device is located relative to the first robotic device;

based on the data representing the images, detect one or more particular images that depict a tag on an exterior surface of the second robotic device; and

based on respective sizes of the tag within the one or more particular images, determine the relative positioning between the first robotic device and the second robotic device.

13. The first robotic device of claim 10, wherein establishing the visual handshake with the second robotic device comprises:

determining the relative positioning between the first robotic device and the second robotic device; and

sending data indicating the relative positioning between the first robotic device and the second robotic device to the second robotic device.

14. The first robotic device of claim 10, wherein establishing the visual handshake with the second robotic device comprises:

sending, to the second robotic device, data indicating that the first robotic device has detected a tag on the second robotic device; and

receiving data indicating that the second robotic device has detected a tag on the first robotic device.

15. A system comprising:

a computing system communicatively coupled to a first robotic device and a second robotic device, the computing system configured to;

identify a collaborative operation to be performed by a first robotic device and a second robotic device, wherein the collaborative operation is based on a relative positioning between the first robotic device and the second robotic device;

based on data representing a map of an environment in which the first robotic device and the second robotic device are operating, determine respective locations, relative to the map, of the first robotic device and the second robotic device, wherein the map is updated in response to movements of the first robotic device and the second robotic device relative to the environment; and

initiate a movement of the first robotic device along a path from the determined location of the first robotic device towards the determined respective location relative to the map of the second robotic device; and

the first robotic device and second robotic device configured to;

establish a visual handshake that indicates the relative positioning between the first robotic device and the second robotic device for the collaborative operation.

16. The system of claim 15, wherein the computing system is further configured to:

receive data indicating the relative positioning indicated by the visual handshake; and

determine that the map indicates a relative positioning between the first robotic device and the second robotic device that is offset from the relative positioning indicated by the visual handshake; and

update the map of the environment to correct the offset such that the map indicates the relative positioning indicated by the visual handshake.

17. The system of claim 15, wherein the computing system is further configured to:

initiate a movement of the second robotic device along a path from the determined location of the first robotic device towards a target location at which to perform the collaborative operation;

wherein determining the respective locations, relative to the map, of the first robotic device and the second robotic device comprises;

identifying the location of the second robotic device as the target location.

18. The system of claim 15, wherein:

the second robotic device comprises a robotic manipulator;

the collaborative operation involves transferring an object from the second robotic device to the first robotic device;

the visual handshake further indicates a target location on the first robotic device for the object; and

the second robotic device is configured to;

after establishing the visual handshake, cause the robotic manipulator of the second robotic device to transfer the object to the target location on the first robotic device.

19. The system of claim 15, wherein:

the first robotic comprises a container configured to hold one or more objects;

the second robotic device comprises a robotic manipulator;

the collaborative operation involves picking an object from the container of the first robotic device;

the visual handshake further indicates a location of the container on the first robotic device relative to the robotic manipulator of the second robotic device; and

the second robotic device is configured to;

after establishing the visual handshake, cause the robotic manipulator of the second robotic device to pick the object from the container of the first robotic device.

20. The system of claim 15, wherein the second robotic device comprises a camera that is mounted to a moveable end of a robotic manipulator, the second robotic device further configured to;

receive data indicating a particular direction in which the first robotic device is located; and

cause the robotic manipulator to orient the camera in the particular direction to detect a tag on an exterior surface of the first robotic device.