Capturing packaging image via scanner

• US 9,505,554 B1
• Filed: 09/24/2013
• Issued: 11/29/2016
• Est. Priority Date: 09/24/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kong, Cheuk-Man, Rao, Mohan Krishna, Abbott, Morley Duncan, Maniymaran, Balasubramaneyam
• Primary Examiner(s): CUMBESS, YOLANDA RENEE
• Application Number:
  US14/035,671
• Time in Patent Office: 1,162 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B65G 2209/04 Indication location means
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]

Abstract

Methods of and devices and systems for obtaining and using images of items as the items appear in an inventory area (e.g., storage bin) are disclosed. Images of inventory areas are captured before and after stow or pick operations. A difference between the before and after images is determined and extracted as an image of the item that was stowed or picked. The process may be repeated for numerous units of the item and one or more definitive item images are selected. The definitive item image may be displayed to pickers directly to facilitate picking, or may be used to determine the position of a unit of the item in a particular inventory area such that the position of the item to be picked may be indicated to the picker. Definitive item images may also be used for inventory validation, enforcement of pick and stow etiquette.

Claims

1. A system, comprising:

one or more computing devices configured to implement;

a control module configured to direct a stowing or picking process of a unit of an item to or from, respectively, a storage bin in a materials handling facility, wherein the unit of the item is stowed among or picked from among one or more units of one or more other items in the storage bin; and

an item image module configured to;

receive a before image of the storage bin before the unit of the item is stowed to or picked from the storage bin;

receive an after image of the storage bin after the unit of the item is stowed to or picked from the storage bin; and

determine an image of the item based on a comparison of the before and after images of the storage bin, wherein the determined image of the item distinguishes the item from the one or more other items in the storage bin.

2. The system of claim 1, further comprising receiving an image of another storage bin and using the determined image of the item based on the comparison to determine a position of another unit of the item in the another storage bin.

3. The system of claim 2, wherein subsequent to the determination of the position of the other unit of the item in the other storage bin, the control module is further configured to send a recent image of the other storage bin to a device for display on the device, wherein the position of the item is highlighted in the displayed recent image to facilitate picking of the other unit of the item from the other storage bin.

4. A method, comprising:

performing by one or more computing devices having one or more processors;

directing one or more agents to stow an item to or pick an item from a storage location of a materials handling facility, wherein a given unit of the item is stowed among or picked from among one or more units of other items in the storage location;

receiving a before image of the storage location before the unit of the item is stowed to or picked from the storage location;

receiving an after image of the storage location after the unit of the item is stowed to or picked from the storage location; and

determining an image of the item based on a comparison of the before and after images, wherein the determined image of the item distinguishes the item from the other items in the storage location.

5. The method of claim 4, further comprising scanning an identifier (ID) of the storage location, wherein scanning the ID of the storage location triggers capturing the before image or the after image.

6. The method of claim 4, further comprising:

repeating the steps of directing, receiving a before image, receiving an after image and determining, for a plurality of units of the item at a plurality of other storage locations such that a plurality of images of the item are determined; and

applying one or more analysis techniques to the plurality of images to select one or more identifying images of the item from the plurality of images of the item.

7. The method of claim 6, further comprising:

obtaining a recent image of another storage location of another unit of the item;

comparing one or more identifying images of the item to the recent image of the another storage location to determine an image-based inventory of units of the item for the another storage location;

determining a discrepancy between the determined image-based inventory and an inventory record associated with the another storage location; and

based at least in part on a determination of a discrepancy, triggering an inventory check on the another storage location.

8. The method of claim 7, wherein the inventory check comprises recording the discrepancy in a discrepancy record, sending an instruction for a manual inspection of the image of the another storage location, or sending an instruction for a manual inspection of the another storage location.

9. The method of claim 4, further comprising:

identifying another unit of the item for picking in another storage location;

obtaining a recent image of the another storage location;

determining a position of the another unit of the item in the another storage location; and

displaying the recent image of the another storage location, wherein the position of the another unit of the item in the image of the another storage location is highlighted.

10. The method of claim 4, further comprising;

analyzing at least the after image of the item in the storage location; and

observing, in the after image, a breach of stow etiquette, wherein the breach in stow etiquette comprises failure to comply with a stow etiquette rule stored in a stow etiquette data store.

11. The method of claim 4, wherein the storage location comprises one or more graphic indicators that are captured in the before image or the after image, wherein one or more known physical dimensions are associated with the one or more graphic indicators, the method further comprising:

determining a ratio between the one or more known physical dimensions and one or more dimensions represented in the before or after image based on the one or more graphic indicators captured in the image; and

using the ratio to determine one or more physical dimensions of the unit of the item represented in the image of the storage location.

12. The method of claim 4, further comprising:

receiving an image of another storage location;

determining a dimension scale for items represented in the image of another storage location based upon known dimensions for one or more graphical indicators represented in the image or based upon known dimension data of one or more items represented in the image; and

identifying another unit of the item based at least in part on using the dimension scale for items represented in the image and at least one known dimension of the item.

13. A non-transitory computer-readable medium, storing program instructions that when executed by a computer perform:

directing one or more agents to stow an item to or pick the item from a storage location of a materials handling facility, wherein a given unit of the item is stowed among or picked from among one or more units of other items in the storage location;

receiving a before image of the storage location before the unit of the item is stowed to or picked from the storage location;

receiving an after image of the storage location after the unit of the item is stowed to or picked from the storage location; and

determining an image of the unit of the item based on a comparison of the before and after images, wherein the determined image of the item distinguishes the item from the other items in the storage location.

14. The non-transitory computer-readable medium of claim 13, wherein the program instructions are further executable to perform:

repeating the directing, receiving the before image, receiving the after image and determining to determine a plurality of images of an item; and

selecting one or more identifying images of the item from the plurality of determined images wherein the selecting comprises applying machine learning techniques to the plurality of images to select the one or more identifying images of the item.

15. The non-transitory computer-readable medium of claim 14, wherein the program instructions are further executable to perform:

obtaining an image of another storage location of another unit of the item;

comparing one or more identifying images of the item to the image of the another storage location to determine an inventory of units of the item for the another storage location;

determining a discrepancy between the determined inventory and an inventory record associated with units of the item for the another storage location; and

based at least in part on a determination of a discrepancy, triggering an inventory check on the another storage location.

16. The non-transitory computer-readable medium of claim 15, wherein the inventory check comprises recording the discrepancy in a discrepancy record, sending an instruction for a virtual inspection of the another storage location, or sending an instruction for a physical inspection of the another storage location.

17. The non-transitory computer-readable medium of claim 13, wherein the storage location comprises one or more graphical indicators of a known dimension that are captured during said capturing the before image or the after image, and wherein the program instructions are further executable to perform:

determining a ratio between the known dimension and a dimension represented in the before image or the after image based on the one or more graphic indicators represented in the image; and

using the ratio to determine one or more dimensions of the unit of the item represented in the image of the storage location.

18. The non-transitory computer-readable medium of claim 13, wherein the program instructions are further executable to perform:

identifying another unit of the item for picking;

obtaining an image of another storage location of the another unit of the item; and

displaying the image of the another storage location of the another unit of the item, wherein the another unit of the item in the image of the another storage location is highlighted.

19. The non-transitory computer-readable medium of claim 13, wherein the program instructions are further executable to perform:

obtaining an image of another storage location subsequent to a picking of a unit of the item from, or a stow of the item to the another storage location;

determining a prior position of the picked or a current position of the stowed item in the storage location; and

analyzing the determined prior or current position of the item in the storage location to determine a breach of stow etiquette, wherein the breach in stow etiquette comprises failure to comply with a stow etiquette rule stored in a stow etiquette data store.

20. The non-transitory computer-readable medium of claim 13, wherein the program instructions are further executable to perform:

analyzing a positional relationship between a plurality of graphical indicators represented in the before image or in the after image to determine a perspective distortion of the before image or the after image of the storage location;

performing an alignment process, wherein the alignment process includes perspective correction to produce a corrected image of the storage location; and

identifying the item in the corrected image of the storage location.