Methods and systems for distributing remote assistance to facilitate robotic object manipulation

• US 9,486,921 B1
• Filed: 03/26/2015
• Issued: 11/08/2016
• Est. Priority Date: 03/26/2015
• Status: Active Grant

• Current Assignee: Intrinsic Innovation LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Straszheim, Troy Donald, Zevenbergen, John William, Nusser, Stefan, Watts, Kevin William, Rublee, Ethan
• Primary Examiner(s): Figueroa, Jaime
• Application Number:
  US14/669,374
• Time in Patent Office: 593 Days
• Field of Search: 700/245, 700/250, 700/253, 700/257, 700/264
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1679 characterised by the tasks executed
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/40006 Placing, palletize, un palletize, paper roll placing, box stacking
• CPC Class Codes: G05B 2219/40298 Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/40513 Planning of vehicle and of its manipulator arm
• CPC Class Codes: G05B 2219/40518 Motion and task planning
• CPC Class Codes: G05B 2219/40629 Manipulation planning, consider manipulation task, path, grasping
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Methods and systems for distributing remote assistance to facilitate robotic object manipulation are provided herein. Regions of a model of objects in an environment of a robotic manipulator may be determined, where each region corresponds to a different subset of objects with which the robotic manipulator is configured to perform a respective task. Certain tasks may be identified, and a priority queue of requests for remote assistance associated with the identified tasks may be determined based on expected times at which the robotic manipulator will perform the identified tasks. At least one remote assistor device may then be requested, according to the priority queue, to provide remote assistance with the identified tasks. The robotic manipulator may then be caused to perform the identified tasks based on responses to the requesting, received from the at least one remote assistor device, that indicate how to perform the identified tasks.

Claims

1. A computing system for controlling a robotic manipulator, the computing system comprising:

at least one processor; and

data storage comprising instructions executable by the at least one processor to cause the computing system to perform operations comprising;

determining, from a model of a set of objects in an environment of the robotic manipulator, a plurality of regions of the model, wherein each region corresponds to a respective different subset of the set of objects, and wherein the robotic manipulator is configured to perform, for each subset of objects, a respective task involving a manipulation of the subset of objects;

identifying a plurality of the tasks for which the computing system will request remote assistance;

determining a priority queue of requests for remote assistance associated with the identified tasks based on expected times at which the robotic manipulator will perform the identified tasks for each subset of objects;

requesting, according to the priority queue and from at least one remote assistor device, remote assistance associated with the identified tasks;

receiving, from the at least one remote assistor device, a plurality of responses to the requesting, wherein each response includes information indicative of how to perform the manipulation of the respective subset of objects; and

causing the robotic manipulator to perform the identified tasks based at least in part on the plurality of responses received from the at least one remote assistor device.

2. The computing system of claim 1, wherein at least one of the identified tasks includes (i) identifying one or more locations on each object of the respective subset of objects and (ii) gripping the respective subset of objects at the one or more locations on each object of the respective subset of objects.

3. The computing system of claim 1, wherein at least one of the identified tasks includes determining a sequenced order in which to perform the manipulation of each object of the respective subset of objects.

4. The computing system of claim 1, wherein at least one of the identified tasks includes (i) determining a segmentation of the respective subset of objects and (ii) performing the manipulation of the respective subset of objects based on the determined segmentation.

5. The computing system of claim 1, wherein at least one of the identified tasks includes (i) identifying a virtual boundary line separating two adjacent objects of the respective subset of objects and (ii) performing the manipulation of the respective subset of objects based on the identified virtual boundary line.

6. The computing system of claim 1, wherein the set of objects comprises a set of boxes, and wherein the model comprises a two-dimensional (2D) image of a faç

ade of the set of boxes.

7. The computing system of claim 1, the operations further comprising:

determining a confidence score associated to each region, wherein the confidence score indicates a level of confidence in how to perform the respective task involving the manipulation of the subset of objects of the respective region,

wherein identifying the plurality of the tasks for which the computing system will request remote assistance is based at least in part on the determined confidence scores.

8. The computing system of claim 1, wherein the at least one remote assistor device includes a plurality of remote assistor devices,

wherein determining the priority queue of requests for remote assistance comprises assigning, to each remote assistor device of the plurality of remote assistor devices, at least one respective request based on a predetermined remote assistor rating of the remote assistor device,

wherein the predetermined remote assistor rating indicates a history of whether the robotic manipulator has correctly performed previous tasks based on at least one response that the computing system has previously received from the remote assistor device, and

wherein requesting, according to the priority queue and from the at least one remote assistor device, remote assistance associated with the identified tasks comprises sending the requests to the plurality of remote assistor devices based on the assigning.

9. The computing system of claim 8, wherein the assigning is based on a determination that the predetermined remote assistor rating exceeds a predetermined threshold.

10. A method comprising:

determining, from a model of a set of objects in an environment of a robotic manipulator, a plurality of regions of the model, wherein each region corresponds to a respective different subset of the set of objects, and wherein the robotic manipulator is configured to perform, for each subset of objects, a respective task involving a manipulation of the subset of objects;

identifying a plurality of the tasks for which the computing system will request remote assistance;

determining a priority queue of requests for remote assistance associated with the identified tasks based on expected times at which the robotic manipulator will perform the identified tasks for each subset of objects;

requesting, according to the priority queue and from at least one remote assistor device, remote assistance associated with the identified tasks;

receiving, from the at least one remote assistor device, a plurality of responses to the requesting, wherein each response includes information indicative of how to perform the manipulation of the respective subset of objects; and

causing the robotic manipulator to perform the identified tasks based at least in part on the plurality of responses received from the at least one remote assistor device.

11. The method of claim 10, further comprising:

in response to causing the robotic manipulator to perform the identified tasks based at least in part on the plurality of responses, determining task performance statistics associated to each of the at least one remote assistor device, wherein the task performance statistics indicate whether the robotic manipulator correctly performed each of the identified tasks based on the information included in each of the plurality of responses; and

providing, to each of the at least one remote assistor device, a notification of the determined task performance statistics.

12. The method of claim 11, wherein the determined task performance statistics indicate one or more of:

(i) a percentage of each type of identified task the robotic manipulator has correctly performed based on information included in responses received from the remote assistor device and (ii) an amount of time taken by the robotic manipulator to perform each identified task based on the information included in the responses received from the remote assistor device.

13. The method of claim 10, wherein the at least one remote assistor device includes a plurality of different remote assistor devices,

wherein requesting, according to the priority queue and from the at least one remote assistor device, remote assistance associated with the identified tasks comprises sending a given request to each remote assistor device of the plurality of remote assistor devices,

wherein receiving the plurality of responses from the at least one remote assistor device comprises receiving respective responses to the given request from each remote assistor device of the plurality of remote assistor devices, and

wherein causing the robotic manipulator to perform the identified tasks based at least in part on the plurality of responses received from the at least one remote assistor device comprises causing the robotic manipulator to perform the identified tasks based on an aggregation of the respective responses to the given request received from each remote assistor device.

14. The method of claim 10, wherein requesting, according to the priority queue and from the at least one remote assistor device, remote assistance associated with the identified tasks comprises sending, to the at least one remote assistor device, the requests for remote assistance with the identified tasks, wherein each request includes an indication of the respective region associated to the respective identified task.

15. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing system configured to control a robotic manipulator, cause the computing system to perform operations comprising:

determining, from a model of a set of objects in an environment of the robotic manipulator, a plurality of regions of the model, wherein each region corresponds to a respective different subset of the set of objects, and wherein the robotic manipulator is configured to perform, for each subset of objects, a respective task involving a manipulation of the subset of objects;

identifying a plurality of the tasks for which the computing system will request remote assistance;

determining a priority queue of requests for remote assistance associated with the identified tasks based on expected times at which the robotic manipulator will perform the identified tasks for each subset of objects;

requesting, according to the priority queue and from at least one remote assistor device, remote assistance associated with the identified tasks;

receiving, from the at least one remote assistor device, a plurality of responses to the requesting, wherein each response includes information indicative of how to perform the manipulation of the respective subset of objects; and

causing the robotic manipulator to perform the identified tasks based at least in part on the plurality of responses received from the at least one remote assistor device.

16. The non-transitory computer readable medium of claim 15, wherein at least one of the identified tasks includes (i) identifying one or more locations on each object of the respective subset of objects and (ii) gripping the respective subset of objects at the one or more locations on each object of the respective subset of objects, the operations further comprising:

determining a confidence score associated to each region, wherein the confidence score indicates a level of confidence in how to identify the one or more locations on each object of the respective subset of objects,

wherein identifying the plurality of the tasks for which the computing system will request remote assistance is based at least in part on the determined confidence scores.

17. The non-transitory computer readable medium of claim 15, wherein at least one of the identified tasks includes determining a sequenced order in which to perform the manipulation of each object of the respective subset of objects, the operations further comprising:

determining a confidence score associated to each region, wherein the confidence score indicates a level of confidence in how to determine the sequenced order in which to perform the manipulation of each object of the respective subset of objects,

wherein identifying the plurality of the tasks for which the computing system will request remote assistance is based at least in part on the determined confidence scores.

18. The non-transitory computer readable medium of claim 15, wherein at least one of the identified tasks includes (i) determining a segmentation of the respective subset of objects and (ii) performing the manipulation of the respective subset of objects based on the determined segmentation, the operations further comprising:

determining a confidence score associated to each region, wherein the confidence score indicates a level of confidence in how to determine a segmentation that distinguishes boundaries of the subset of objects of the respective region,

wherein identifying the plurality of the tasks for which the computing system will request remote assistance is based at least in part on the determined confidence scores.

19. The non-transitory computer readable medium of claim 15, wherein determining the priority queue of requests for remote assistance associated with the identified tasks based on expected times at which the robotic manipulator will perform the identified tasks for each subset of objects comprises rank-ordering the requests in order of the expected times at which the robotic manipulator will perform each of the identified tasks so as to define a particular order in which to send the requests, and

wherein requesting, according to the priority queue and from the at least one remote assistor device, remote assistance associated with the identified tasks comprises sending, in the particular order, the requests to the at least one remote assistor device.

20. The non-transitory computer readable medium of claim 15, wherein the at least one remote assistor device includes a plurality of remote assistor devices, the operations further comprising:

in response to causing the robotic manipulator to perform the identified tasks based at least in part on the plurality of responses, determining, for each remote assistor device of the plurality, task performance statistics indicating (i) a percentage of each type of identified task the robotic manipulator has correctly performed based on information included in responses received from the remote assistor device and (ii) an amount of time taken by the robotic manipulator to correctly perform each identified task based on the information included in responses received from the remote assistor device;

determining a ranking of the plurality of remote assistor devices based on the task performance statistics determined for each remote assistor device; and

providing, to each remote assistor device of the plurality, a notification of the determined ranking.