Detecting item interaction and movement

• US 10,268,983 B2
• Filed: 06/26/2013
• Issued: 04/23/2019
• Est. Priority Date: 06/26/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kumar, Dilip, Kornfield, Elie Micah, Prater, Alexander Clark, Boyapati, Sridhar, Ren, Xiaofeng, Yuan, Chang
• Primary Examiner(s): Masud, Rokib
• Application Number:
  US13/928,345
• Publication Number:
  US 20150019391A1
• Time in Patent Office: 2,127 Days
• Field of Search: 705418, 705 28, 705 16, 382103
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G07G 1/0036 Checkout procedures

Abstract

This disclosure describes a system for tracking removal or placement of items at inventory locations with a materials handling facility. In some instances, a user may remove an item from an inventory location and the inventory management system may detect that removal and update a user item list associated with the user to include an item identifier representative of the removed item. Likewise, if the user places an item at an inventory location, the inventory management system may detect that placement and update the user item list to remove an item identifier representative of the placed item.

Claims

1. A system, comprising:

a plurality of cameras positioned within a materials handling facility, each camera of the plurality of cameras configured to generate images of objects located within the materials handling facility;

one or more processors; and

a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to at least;

determine a location of a user within the materials handling facility, wherein the location is near an inventory location;

retrieve, from a data store, a user item list associated with the user, the user item list indicating at least one of an item to be picked by the user, an item picked by the user, an item to be stowed by the user, or an item stowed by the user,

receive, from at least one of the plurality of cameras, a first image of at least a portion of a hand of the user;

determine that the first image was captured prior to the hand of the user moving into the inventory location;

receive, from at least one of the plurality of cameras, a second image of at least a portion of the hand of the user;

determine that the second image was captured after the hand of the user is removed from the inventory location;

compare the first image and the second image to determine that the second image includes a representation of an object held in the hand of the user;

in response to a determination that the second image includes a representation of an object held in the hand of the user, identify an item of inventory associated with the inventory location;

determine that the object corresponds with the item of inventory; and

add to the user item list, an item identifier representative of the item.

2. The system of claim 1, wherein the program instructions that when executed by the one or more processors cause the one or more processors to compare the first image and the second image further include program instructions that when executed by the one or more processors cause the one or more processor to at least:

identify in at least one of the first image or the second image at least a portion of the hand of the user based at least in part on a skin color; and

identify in the second image at least a portion of the object as held by the hand of the user and having a color different than a skin color of the hand of the user.

3. The system of claim 1, wherein the program instructions that when executed by the one or more processors cause the one or more processors to compare the first image and the second image further include program instructions that cause the one or more processors to identify a difference in objects included in the first image and the second image.

4. The system of claim 1, wherein the item of inventory associated with the inventory location is known to the system based on a prior placement of the item of inventory at the inventory location.

5. The system of claim 1, wherein the program instructions that when executed by the one or more processors further cause the one or more processors to at least:

receive a third image including a representation of the inventory location prior to the hand of the user moving into the inventory location;

receive a fourth image including a representation of the inventory location after the hand of the user is removed from the inventory location; and

wherein the program instructions to determine that the user is holding the object further include instructions to determine that the user is holding the object based at least in part on a comparison of the third image and the fourth image to identify that an object has been removed from the inventory location.

6. The system of claim 1, wherein the determination that the object corresponds with the item of inventory includes considering at least one of a weight of the item received from a scale located at the inventory location, a shape of at least a portion of the object received from a pressure sensor at the inventory location, or a feature of the object identified from the second image.

7. The system of claim 1, wherein the program instructions that when executed by the one or more processors further causing the one or more processors to at least:

receive an input from at least one of input device, the input indicating that the object has been removed from the inventory location; and

wherein the input device is at least one of a pressure sensor, a camera, an infrared sensor, a scale, a volume displacement sensor, or a light curtain.

8. The system of claim 1, wherein the program instructions that when executed by the one or more processors to cause the one or more processors to determine that the first image was captured prior to the hand of the user moving into the inventory location further include program instructions that when executed by the one or more processors cause the one or more processors to at least:

determine that the first image was captured prior to the hand of the user moving into the inventory location based at least in part on a determination that the image was captured in response to a detection of an object approaching the inventory location.

9. The system of claim 8, wherein it is determined that the hand of the user is approaching the inventory location based on one or more of:

image processing of images captured of the inventory location, or detection of an object penetrating a light curtain positioned adjacent a vertical plane in front of the inventory location.

10. The system of claim 8, wherein the program instructions that when executed by the one or more processors to cause the one or more processors to determine that the second image was captured after the hand of the user is removed from the inventory location further include program instructions that when executed by the one or more processors cause the one or more processors to at least:

determine that the second image was captured after the hand of the user is removed from the inventory location based at least in part on the second image being an image of a series of images that make up a video.

11. A method comprising:

under control of one or more computing devices configured with executable instructions,

determining a location of a user within a materials handling facility;

retrieving from a data store a user item list associated with the user, the user item list indicating at least one of an item to be picked by the user, an item picked by the user, an item to be stowed by the user, or an item stowed by the user;

receiving, from at least one of a plurality of cameras positioned within the materials handling facility, each camera of the plurality of cameras configured to generate images of objects located within the materials handling facility, a first image of the location;

determining that the first image was obtained prior to the user placing a hand into an inventory location near the location within the materials handling facility;

receiving, from at least one of the plurality of cameras, a second image of the location;

determining that the second image was obtained subsequent to the user placing the hand into the inventory location;

comparing the first image and the second image to determine that the user has placed an object at the inventory location;

determining an identity of the object placed at the inventory location; and

updating the user item list to remove an item identifier for the object placed at the inventory location from the user item list.

12. The method of claim 11, wherein determining the identity of the object placed at the inventory location includes:

processing at least one of the first image or the second image to obtain a representation of at least a portion of the object;

comparing the at least a portion of the object with stored representations of items associated with items identified on the user item list to determine a correlation between the representation of the at least a portion of the object and each of the stored representations of items; and

wherein updating the user item list includes removing the item identifier associated with the item having a highest confidence score.

13. The method of claim 11, wherein determining the identity of the object placed at the inventory location further includes:

computing a confidence score for a highest correlation; and

determining that the confidence score exceeds a threshold; and

wherein updating the user item list is done automatically and without human input.

14. The method of claim 11, wherein determining the identity of the object placed at the inventory location further includes:

computing a confidence score for a highest correlation;

determining that the confidence score does not exceed a threshold;

sending a message to an associate to identify the object; and

receiving from the associate an identity of the object.

15. The method of claim 11, the method further comprising:

determining that the object is to remain at the inventory location; and

updating an item stock quantity associated with the inventory location.

16. The method of claim 11, the method further comprising:

determining that the object is not to remain at the inventory location; and

sending a message to an associate to remove the object from the inventory location.

17. A system, comprising:

a plurality of cameras positioned within a materials handling facility, each camera of the plurality of cameras configured to generate images of objects located within the materials handling facility;

one or more processors; and

a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to at least;

determine a location of a user within the materials handling facility, wherein the location is near an inventory location;

retrieve, from a data store, a user item list associated with the user, the user item list indicating at least one of an item to be picked by the user, an item picked by the user, an item to be stowed by the user, or an item stowed by the user;

receive, from at least one of the plurality of cameras, a first image of at least a portion of a user;

determine that the first image was captured prior to the user reaching into the inventory location;

receive, from at least one of the plurality of cameras, a second image of at least a portion of the user;

determine that the second image was captured after the user is no longer reaching into the inventory location;

compare the first image and the second image to determine that the second image includes a representation of an object held by the user;

in response to a determination that the second image includes a representation of an object held by the user, identify an item of inventory associated with the inventory location;

determine that the object corresponds with the item of inventory; and

add to the user item list, an item identifier representative of the item.

18. The system of claim 17, wherein the program instructions that when executed by the one or more processors cause the one or more processors to compare the first image and the second image and further include program instructions that when executed by the one or more processors cause the one or more processors to at least:

identify in at least one of the first image or the second image at least a portion of the user based at least in part on a skin color; and

identify in the second image at least a portion of the object as held by the user and having a color different than a skin color of the user.

19. The system of claim 17, wherein the program instructions that when executed by the one or more processors cause the one or more processors to compare the first image and the second image and further include program instructions that cause the one or more processors to identify a difference in objects included in the first image and the second image.

20. The system of claim 17, wherein the item of inventory associated with the inventory location is known to the system based on a prior placement of the item of inventory at the inventory location.

21. The system of claim 17, wherein the program instructions that when executed by the one or more processors further cause the one or more processors to at least:

receive a third image including a representation of the inventory location prior to the user reaching into the inventory location;

receive a fourth image including a representation of the inventory location after the user is no longer reaching into the inventory location; and

wherein the program instructions to determine that the user is holding the object further include instructions to determine that the user is holding the object based at least in part on a comparison of the third image and the fourth image to identify that an object has been removed from the inventory location.

22. The system of claim 17, wherein the determination that the object corresponds with the item of inventory includes considering at least one of a weight of the item received from a scale located at the inventory location, a shape of at least a portion of the object received from a pressure sensor at the inventory location, or a feature of the object identified from the second image.

23. The system of claim 17, wherein the program instructions that when executed by the one or more processors further causing the one or more processors to at least:

receive an input from at least one input device, the input from the input device indicating that the object has been removed from the inventory location; and

wherein the input device is at least one of a pressure sensor, a camera, an infrared sensor, a scale, a volume displacement sensor, or a light curtain.

24. The system of claim 17, wherein the at least a portion of the user includes at least a portion of a hand of the user.

25. The system of claim 17, wherein the object is held in a hand of the user.

26. A system, comprising:

a plurality of cameras, each camera of the plurality of cameras configured to generate images of objects within a field of view of the camera;

one or more processors; and

a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to at least;

receive, from at least one of the plurality of cameras, a first image of at least a portion of a user;

receive, from at least one of the plurality of cameras, a second image of at least a portion of the user;

compare the first image and the second image to determine that the second image includes a representation of an object held by the user;

in response to a determination that the second image includes a representation of an object held by the user, determine an item identifier corresponding to the object; and

add to a user item list corresponding to the user, the item identifier.

27. The system of claim 26, wherein the program instructions that when executed by the one or more processors further cause the one or more processors to at least:

determine a plurality of inventory items associated with an inventory location in a field of view of the at least one of the plurality of cameras; and

wherein the item identifier corresponds to one of the plurality of inventory items.

28. The system of claim 26, wherein the program instructions that when executed by the one or more processors further cause the one or more processors to at least:

receive an input from at least one input device, the input from the input device indicating a movement of the object.