Robot implemented item manipulation

• US 10,360,531 B1
• Filed: 12/19/2016
• Issued: 07/23/2019
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Stallman, Timothy, Garber, Menashe, Sweeney, Andrew M.
• Primary Examiner(s): Savusdiphol, Paultep
• Application Number:
  US15/384,139
• Time in Patent Office: 946 Days
• Field of Search: 235385, 235454, 23546201, 235375
• CPC Class Codes: G06K 19/06028 using bar codes
• CPC Class Codes: G06K 19/07758 arrangements for adhering the record carrier to further objects or living beings, functioning as an identification tag
• CPC Class Codes: G06K 7/1413 1D bar codes
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

A technology is provided for robot implemented item manipulation. In one example, a method includes analyzing one or more items including a target item for manipulation, using machine vision rules, to identify the target item. When a determination is made that the robot is unable to perform the manipulation, a request may be transmitted requesting human assistance. An electronic response may be received in response to the request which includes an instruction for manipulating the target item. The robot may manipulate the target item using the instruction.

Claims

1. A non-transitory computer-readable medium comprising computer-executable instructions which, when executed by a processor, implement a method for human directed robotic pick operations, the method comprising:

obtaining a first image of one or more items including a target item to be picked and placed at a destination;

machine analyzing the first image to identify the target item, how to pick the target item, and the destination;

determining a failure condition exists responsive to a robot being unable to pick and place the target item;

transmitting, during a pick and place operation by the robot, the first image to a remote human assistance device and requesting human assistance based in part on the first image;

receiving a second image from the remote human assistance device including image data having an electronic marking made to the first image with the remote human assistance device describing picking or placing the target item;

analyzing the second image to identify the electronic marking; and

picking and placing, by the robot, the target item at the destination using an instruction associated with the electronic marking.

2. The computer-readable medium of claim 1, further comprising analyzing the electronic marking to interpret the instruction.

3. The computer-readable medium of claim 1, wherein the instruction includes at least one of an identification of a portion of the image representing the target item, identification of an end effector to use to pick up the target item, identification of where to pick up the target item, or identification of the destination.

4. The computer-readable medium of claim 1, wherein:

obtaining the image further comprises capturing a machine vision image using a first camera and capturing a human-use image using a second camera in a different position relative to the robot than the first camera;

machine analyzing the image further comprises machine analyzing the machine vision image; and

transmitting the image further comprises transmitting the human-use image.

5. The computer-readable medium of claim 1, wherein the image used in the machine analyzing and the transmitting is one image captured by one camera operated by the robot.

6. A robot implemented item manipulation method, comprising:

analyzing one or more items including a target item for manipulation, using machine vision analysis, to identify the target item;

determining that a robot is unable to perform the manipulation;

transmitting, during a pick and place operation by the robot, a request for assistance to a remote human assistance device;

receiving an electronic response to the request from the remote human assistance device, the response including image data of an electronic marking made using the remote human assistance device to provide an instruction for manipulating the target item;

interpreting the electronic marking in the image data of the response into a robot-usable instruction; and

manipulating, by the robot, the target item using the robot-usable instruction.

7. The method of claim 6, wherein manipulating the target item comprises placing the target item at a destination autonomously using the instruction.

8. The method of claim 6, further comprising determining from the instruction an identification of a robotic arm end effector for use in picking up the target item.

9. The method of claim 6, further comprising:

marking an image of the target item electronically to identify at least one of;

a portion of the first image representing the target item, how to pick up the target item, where to pick up the target item, a destination, or what the item is; and

transmitting the image to the remote human assistance device with the request.

10. The method of claim 9, further comprising identifying additional markings on the image when received with the electronic response from the human assistance device, wherein the additional markings indicate at least one of:

correcting, replacing, or adding to the markings made to the image prior to the transmitting to the remote human assistance device.

11. The method of claim 6, wherein the robot is a flying robot and transmitting the request comprises transmitting an aerial image of a customer drop off zone with the request to a customer, the request further requesting the customer to identify the destination within the aerial image of the customer drop off zone, wherein manipulating the target item comprises releasing the target item at the destination.

12. The method of claim 6, further comprising:

embedding first metadata into a first image of the target item to identify at least one of;

a portion of the image representing the target item, how to pick up the target item, where to pick up the target item, a destination, or what the item is;

transmitting the first image to the remote human assistance device with the request for assistance; and

identifying second metadata in a second image received with the electronic response, wherein the second metadata indicates at least one of;

correcting, replacing, or adding to the first metadata embedded in the first image.

13. A robotic picking system, comprising:

a robot with an arm having at least one end effector configured to pick up items;

an imaging device configured to capture images of the items accessible to the robot with the arm;

a network adapter to enable electronic communications between the robot and a remote human assistance device over a network;

a processor of the robot;

a memory of the robot in electronic communication with the processor; and

computer instructions stored in the memory, the computer instructions being executable by the processor to;

capture a first image of a target item using the imaging device;

transmit, during a pick and place operation by the robot, the first image to the remote human assistance device using the network adapter;

receive a second image from the human assistance device using the network adapter;

analyze image data in the second image to identify image markup made on the first image using the remote human assistance device to provide pick and place instructions for picking and placing the target item; and

actuate the arm to pick up the target item and to place the target item at a destination based on the pick and place instructions in the second image.

14. The robotic picking system of claim 13, wherein the pick and place instructions include at least one of:

the image markup providing an outline of the target item on the second image, the image markup providing a line designating an orientation of the at least one end effector for picking up the target item, the image markup identifying a destination for placing the target item, the image markup identifying an orientation of the target item for placing at the destination, the image markup identifying where on the target item to pick up the target item, or the image markup identifying which of a plurality of available end effectors to use to pick up the target item.

15. The robotic picking system of claim 13, wherein the pick and place instructions are non-alphanumeric and include one or more lines or shapes.

16. The robotic picking system of claim 13, wherein the computer instructions are further configured to provide a graphical user interface to the remote human assistance device, the graphical user interface enabling drawing operations for drawing on the first image and for enabling drag and drop operations for dragging and dropping portions of the first image identified by the drawing to generate the image markup in the image data of the second image.

17. The robotic picking system of claim 13, wherein the computer instructions further comprise machine learning code configured to learn from the pick and place instructions to at least one of segment the target item from other items in the first image, place the target item at the destination or manipulate the target item in order to avoid requests to the human assistance device for future operations.

18. The robotic picking system of claim 13, wherein the pick and place instructions indicate that the robot cannot pick up the item and instruct the robot to request local human assistance.

19. The robotic picking system of claim 13, wherein the pick and place instructions instruct the robot to at least one of jostle, shake, bump or tip the target item or a structure supporting the target item.

20. The robotic picking system of claim 13, wherein the first image comprises streaming video or a three dimensional model.