Item illumination based on image recognition

• US 9,171,278 B1
• Filed: 09/25/2013
• Issued: 10/27/2015
• Est. Priority Date: 09/25/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Rao, Mohan Krishna, Kong, Cheuk-Man, Abbott, Morley Duncan, Maniymaran, Balasubramaneyam
• Primary Examiner(s): Logan, Klye
• Application Number:
  US14/036,657
• Time in Patent Office: 762 Days
• Field of Search: 700/216
• US Class Current: 1/1
• CPC Class Codes: B65G 1/137 with arrangements or automatic control means for selecting which articles are to be removed
• CPC Class Codes: B65G 1/1373 for fulfilling orders in warehouses
• CPC Class Codes: B65G 2209/04 Indication location means
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

An inventory item to be picked may be selected along with an inventory location (e.g., storage bin) storing a particular unit of the item among a units of heterogeneous items also stowed in the inventory location. One or more images of the inventory location may be obtained and analyzed to determine a position of the unit of the item. Instructions may be sent to direct a device to illuminate the determined position of the unit of the item, and pick instructions may be sent to the same or a different the device. A device may receive the position instructions from a control system, and illuminate the determined position of the unit of the item within the inventory location. The pick instructions may direct an agent or mobile storage bin to a particular location and direct the agent to pick the unit of the item from the illuminated position.

Claims

1. A system, comprising:

a fulfillment center with a plurality of storage bins;

one or more computing devices configured to implement a control system configured to;

select an item to be picked based on an order for the item and select an identified storage bin storing a particular unit of the item among a plurality of units of heterogeneous items also stowed together in the identified storage bin;

obtain one or more images of the identified storage bin;

analyze the one or more images of the identified storage bin to determine a position of the unit of the item among the plurality of units of heterogeneous items also stowed in the identified storage bin, wherein to perform said analyze the control system is further configured to process at least one of the one or more images of the identified storage bin in real-time with a picking process for the unit of the item, wherein the one or more images of the identified storage bin that are processed in real-time are captured during the pick process for the unit of the item, before the unit of the item is picked;

send one or more position instructions to direct a laser-enabled device to point the laser at the determined position of the unit of the item; and

send one or more pick instructions to a laser-enabled device, wherein the one or more pick instructions direct an agent or direct a mobile inventory location containing the identified storage bin to a particular location and direct the agent to pick a unit of the item from the identified storage bin at the particular location;

the laser-enabled device configured to;

receive the one or more position instructions from the control system; and

illuminate with a laser the determined position of the unit of the item within the identified storage bin; and

wherein the control system is further configured to;

obtain a current position that the laser is actually illuminating from one or more real-time images of the identified storage bin, wherein the one or more real-time images capture a current position of the laser illumination in real-time with the picking process of the unit of the item; and

instruct adjustment of a direction that the laser is pointed, wherein said adjustment is based on feedback determined from comparing the current position the laser is actually illuminating with the determined position the laser is intended to be illuminating, wherein adjusting the direction the laser is pointed maintains illumination of the intended position when the laser-enabled device projecting the laser is moved, or corrects a position of the illumination when the current position is not the same as the intended position.

2. The system of claim 1, wherein to analyze the one or more images of the storage bin to determine a position, the control system is further configured to pre-process at least one of the one or more images of the identified storage bin before sending said one or more location-based pick instructions to the device, wherein to pre-process the at least one image the control system is configured to compare an identifying image of the item to the at least one image of the identified storage bin to determine the position of the unit of the item in the identified storage bin, wherein the at least one image is a most recent image of the identified storage bin.

3. A method, comprising:

performing, by one or more computing devices having one or more processors;

receiving a before image of a respective inventory location before a respective unit of the same item is stowed to or picked from the respective inventory location, wherein the respective inventory location is the inventory location or another inventory location;

receiving an after image of the respective inventory location after the respective unit of that item is stowed to or picked from the respective inventory location;

determining an identifying image of the item based on a comparison of the before image and the after image;

obtaining one or more images of an inventory location comprising a unit of an item, wherein the inventory location comprises a plurality of heterogeneous items that are stowed together in the inventory location;

analyzing the one or more images of the inventory location to determine a position of the unit of the item among the plurality of heterogeneous items stowed in the inventory location, wherein said analyzing comprises performing an image recognition technique based on the identifying image of the item to locate the item within the one or more images of the inventory location; and

instruct light-based illumination of the determined position of the unit of the item in the inventory location among the plurality of heterogeneous items stowed in the inventory location, such that the determined position that is illuminated distinguishes the unit of the item from other ones of the plurality of heterogeneous items in the inventory location.

4. The method of claim 3, wherein said analyzing the one or more images of the inventory location to determine the position further comprises:

pre-processing at least one of the one or more images of the inventory location before the unit of the item is selected for picking to fulfill an order for the item, wherein said pre-processing comprises comparing the identifying image of the item to the at least one image of the inventory location to determine the position of the unit of the item in the inventory location.

5. The method of claim 3, further comprising:

obtaining one or more other images of the inventory location comprising a unit of another item, wherein the inventory location comprises a plurality of heterogeneous items that are stowed together in the inventory location;

analyzing the one or more other images of the inventory location to determine a position of the unit of the other item among the plurality of heterogeneous items stowed in the inventory location; and

instructing light-based illumination of the determined position of the unit of the other item in the inventory location among the plurality of heterogeneous items, such that the position that is illuminated distinguishes the unit of the other item from other ones of the plurality of heterogeneous items in the inventory location and is different from the position that was illuminated for the unit of the item.

6. The method of claim 3, wherein said analyzing the one or more images of the inventory location to determine a position further comprises:

processing at least one of the one or more images of the inventory location in real-time with a picking process for the item, wherein the at least one image of the inventory location is captured during the picking process for the item, before the unit of the item is picked.

7. The method of claim 6, further comprising:

obtaining a current position illuminated by the light-based illumination from real-time imagery of the light-based illumination when the light-based illumination is illuminating the current position; and

adjusting a direction that the light-based illumination is pointed, wherein said adjusting is based on feedback determined from comparing the current position the light-based illumination is illuminating with the determined position the light-based illumination is intended to be illuminating, wherein said adjusting the direction the light-based illumination is pointed maintains illumination of the intended position when a device projecting the light-based illumination is moved.

8. The method of claim 3, wherein said illuminating, via light-based illumination, the determined position of the unit of the item in the inventory location comprises illuminating the determined position with a laser.

9. The method of claim 8, further comprising:

for a period of time, continuously analyzing imagery in an area surrounding a path of the laser, wherein analyzing imagery surrounding the path of the laser comprises analyzing the imagery to recognize one or more facial feature in the imagery;

discontinuing projection of the laser when the one or more facial features are recognized in the imagery.

10. A non-transitory computer-readable medium storing program instructions that when executed by a computer perform:

obtaining one or more images of an inventory location for a unit of an item, wherein the inventory location comprises a plurality of heterogeneous items that are stowed together in the inventory location;

analyzing the one or more images of the inventory location to determine a position for the unit of the item among the plurality of heterogeneous items stowed in the inventory location, wherein the inventory location comprises one or more graphical indicators associated with one or more known dimensions that are captured in the one or more images;

determining a ratio between the one or more known dimensions and one or more dimensions represented in the one or more images of the inventory location based on the one or more graphical indicators captured in the one or more images;

determining a reference position of the unit of the item represented in the one or more images of the inventory location with respect to the one or more graphical indicators; and

generating, based at least in part on the ratio and the reference position, instructions for illuminating, via light-based illumination, the determined position for the unit of the item in the inventory location.

11. The non-transitory computer-readable medium of claim 10, wherein said analyzing the one or more images of the inventory location to determine a position further comprises:

pre-processing at least one of the one or more images of the inventory location before the unit of the item is selected for picking to fulfill an order for the item, wherein said pre-processing comprises comparing an identifying image of the item to the at least one image of the inventory location to determine the position in the inventory location.

12. The non-transitory computer-readable medium of claim 10, wherein the program instructions are further executable to perform:

processing at least one of the one or more images of the inventory location in real-time with a picking process for the item, wherein the at least one image is captured during the picking process for the item, before the unit of the item is picked.

13. The non-transitory computer-readable medium of claim 12, wherein the program instructions are further executable to perform, prior to said illuminating, via light-based illumination, the determined position of the unit of the item in the inventory location:

projecting the light-based illumination into a field of view captured by a real-time image of the inventory location that is captured in real-time with the picking process;

adjusting a direction of the light-based illumination, wherein said adjusting is based on feedback determined from comparing a current position illuminated by the light-based illumination with the determined position the light-based illumination is intended to be illuminating, wherein said adjusting brings the light-based illumination upon the determined position.

14. The non-transitory computer-readable medium of claim 10, wherein the program instructions are further executable to perform sending the generated instructions for illuminating the determined position to a laser-enabled device configured to illuminate the determined position with a laser in response to receiving the instructions.

15. The non-transitory computer-readable medium of claim 14, wherein the program instructions are further executable to perform:

analyzing one or more images of a path of the laser to recognize one or more facial features in the one or more images of the path of the laser; and

generating instructions to discontinue projection of the laser based on recognizing the one or more facial features in the one or more images of the path of the laser.

16. A system, comprising:

a laser-enabled device; and

one or more computing devices configured to implement a control system configured to;

obtain one or more images of an inventory location for a unit of an item, wherein the inventory location comprises a plurality of heterogeneous items that are stowed together in the inventory location; and

analyze the one or more images of the inventory location to determine a position for the unit of the item among the plurality of heterogeneous items stowed in the inventory location;

for a period of time, continuously analyze imagery in an area surrounding a path of a laser of the laser-enabled device, wherein continuously analyze imagery comprises analyze the imagery to recognize one or more facial features in the imagery; and

instruct the laser-enabled device to discontinue projection of a respective laser when the one or more facial features are recognized in the imagery;

wherein the laser-enabled device is configured to discontinue projection of a laser in response to receipt of the instruction to discontinue projection of the laser.