Determining an item involved in an event

• US 10,438,277 B1
• Filed: 12/23/2014
• Issued: 10/08/2019
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Jiang, Hao, Asmi, Yasser Baseer, Desai, Nishitkumar Ashokkumar, Maldonado, Emilio Ian, Chinoy, Ammar, Bibireata, Daniel, Raghavan, Sudarshan Narasimha
• Primary Examiner(s): Hayles, Ashford S
• Application Number:
  US14/582,000
• Time in Patent Office: 1,750 Days
• Field of Search: 705 28
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 30/0635 Processing of requisition or of purchase orders
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Described is a multiple-camera system and process for determining an item involved in an event. For example, when a user picks an item or places an item at an inventory location, image information for the item may be obtained and processed to identify the item involved in the event and associate that item with the user.

Claims

1. A computer implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

receiving an event notification that includes an event location and an event time for an event;

determining an event time window that includes the event time;

determining a first item type associated with an inventory location near the event location;

determining a user pattern within a defined distance of the event location during the event time window; and

processing a plurality of images that each include a corresponding representation of the event location to at least;

determine an object connecting the user pattern to a touch point near the event location;

determine that an item was picked from an inventory location at the touch point;

determine a probability that the item picked from the inventory location is of the first item type;

determine a second probability that the item picked was of a second item type; and

determine a confidence score that the item picked was of the first item type, wherein the confidence score is based at least in part on a difference between the probability and the second probability.

2. The computer implemented method of claim 1, further comprising:

determining that the confidence score does not exceed a threshold;

processing a second plurality of images obtained after the event time to determine a third probability that the item is of the first item type, wherein the plurality of images include a representation of the item after the item is placed in a tote;

determining a second confidence score that the item picked was of the first item type, wherein the second confidence score is based at least in part on a difference between the third probability and the second probability;

determining that the second confidence score exceeds the threshold; and

identifying the item as the first item type.

3. The computer implemented method of claim 1, wherein determining the probability includes:

determining at least one of a size, a shape or a color of the item.

4. A computer implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

receiving an event notification that includes an event location and an event time for an event;

determining a defined distance around the event location;

determining a user pattern located within the defined distance of the event location;

determining a first item and a second item within the defined distance;

processing a first plurality of images of an area within the defined distance to determine a first probability that the first item is involved in the event;

processing the first plurality of images to determine a second probability that the second item is involved in the event;

determining a first confidence score representative of a first confidence that the first item is involved in the event, wherein the first confidence score is based at least in part on a difference between the first probability and the second probability;

determining if the first confidence score exceeds a first threshold; and

in response to determining that the first confidence score exceeds the first threshold, associating the first item with the event.

5. The computer implemented method of claim 4, further comprising:

in response to determining that the first confidence score does not exceed the first threshold;

processing a second plurality of images to identify an object released by a user associated with the user pattern;

determining a third probability that the object released by the user corresponds to the first item;

determining a second confidence score representative of a second confidence that the first item is involved in the event, wherein the second confidence score is based at least in part on a difference between the third probability and the second probability;

determining that the second confidence score exceeds a second threshold; and

in response to determining that the second confidence score exceeds the second threshold, associating the first item with the event.

6. The computer implemented method of claim 5, wherein:

the first plurality of images are obtained from a first camera positioned above the event location; and

the second plurality of images are obtained from a second camera.

7. The computer implemented method of claim 5, wherein the object is released into a tote associated with the user pattern.

8. The computer implemented method of claim 4, further comprising:

identifying a user associated with the user pattern; and

adjusting at least one of the first probability or the second probability based at least in part on a past pick history of the user.

9. A computing system, comprising:

one or more processors; and

a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to at least;

receive an event notification that includes an event location and an event time for an event;

determine a user pattern within a defined distance of the event location;

process a plurality of images to determine at least;

a first probability that a first item is involved in the event; and

a second probability that a second item is involved in the event;

determine a first confidence score representative of a first confidence that the first item is involved in the event, wherein the first confidence score is based at least in part on a difference between the first probability and the second probability;

determine that the first confidence score exceeds a first threshold; and

in response to a determination that the first confidence score exceeds the first threshold, associate the first item with the event.

10. The computing system of claim 9, wherein the program instructions that when executed by the one or more processors to process the plurality of images to determine a first probability further include instructions that when executed by the one or more processors cause the one or more processors to at least:

determine a distance between a touch point and the first item.

11. The computing system of claim 9, wherein the first probability is further determined based at least in part on a distance between the first item and the event location.

12. The computing system of claim 9, wherein the program instructions that when executed by the one or more processors further cause the one or more processors to at least:

obtain event location information from an input component within the defined distance of the event location; and

wherein the first probability is further determined based at least in part on the event location information obtained from the input component.

13. The computing system of claim 12, wherein the input component is at least one of:

a camera;

a radio frequency identification (RFID) reader;

a visual identification reader;

or

a load cell.

14. The computing system of claim 12, wherein the event location information is at least one of:

an identification of a radio frequency identification (RFID) tag that is no longer detected;

an identification of an RFID tag newly detected by an RFID reader near the event location;

a decrease in a weight at the event location;

an identification of a visual tag that is near the event location;

or

an increase in a weight at the event location.

15. The computing system of claim 9, wherein the program instructions that when executed by the one or more processors further include instructions that when executed by the one or more processors cause the one or more processors to at least:

adjust at least one of the first probability or the second probability based at least in part on whether a user associated with the user pattern has historically picked the first item or the second item.

16. The computing system of claim 9, wherein the event location corresponds to an inventory location within a materials handling facility.

17. The computing system of claim 9, wherein the first item is an inventory item associated with an inventory location within the defined distance.

18. A method, comprising:

receiving an event notification of an event;

determining an area corresponding to the event;

processing a data corresponding to the area to determine a first probability that a first item is involved in the event;

processing the data corresponding to the area to determine a second probability that a second item is involved in the event;

determining a confidence score representative of a confidence that the first item is involved in the event, wherein the confidence score is based at least in part on a difference between the first probability and the second probability;

determining that the confidence score exceeds a threshold; and

in response to determining that the confidence score exceeds the first threshold, associating the first item with the event.

19. The method of claim 18, further comprising:

determining a user at the area; and

adjusting at least one of the first probability or the second probability based at least in part on information about the user.