Disambiguating between multiple users

• US 10,552,750 B1
• Filed: 12/23/2014
• Issued: 02/04/2020
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Raghavan, Sudarshan Narasimha, Maldonado, Emilio Ian, Smith, David Allen, Xu, Min, Desai, Nishitkumar Ashokkumar, Bibireata, Daniel, Lai, Kevin Kar Wai, Dalal, Pahal Kamlesh
• Primary Examiner(s): Starks, Wilbert L
• Application Number:
  US14/581,995
• Time in Patent Office: 1,869 Days
• Field of Search: None
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G06N 7/00 Computing arrangements based on specific mathematical models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 30/0639 Item locations

Abstract

Described is a multiple-camera system and process for disambiguating between multiple users and identifying which of the multiple users performed an event. For example, when an event is detected, user patterns near the location of the event are determined, along with touch points at the location of the event. User pattern orientation and/or arm trajectories between the event location and the user patterns may be determined and processed to disambiguate between multiple users and determine which user pattern is involved in the event.

Claims

1. A computer implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

in response to a detection, by one or more input components, of an occurrence of an event at an event location within a materials handling facility receiving;

an event notification that includes an indication of the event location and an event time indicating a time of the occurrence of the event;

a first user pattern location corresponding to a first user pattern at the event time; and

a second user pattern location corresponding to a second user pattern at the event time;

specifying, based on the event time, an event time window that includes the event time;

determining, based at least in part on the first user pattern location, that the first user pattern is located within a defined distance of the event location during the event time window;

determining, based at least in part on the second user pattern location, that the second user pattern is located within the defined distance of the event location during the event time window;

processing a plurality of images generated during the event time window to at least;

identify a first object positioned between the first user pattern and the event location;

determine a first trajectory information of the first object between the first user pattern and the event location;

identify a second object positioned between the second user pattern and the event location; and

determine a second trajectory information of the second object between the second user pattern and the event location;

determining, based at least in part on the first trajectory information, a first probability that the first user pattern is to be associated with the event;

determining, based at least in part on the second trajectory information, a second probability that the second user pattern is to be associated with the event;

determining, based at least in part on a difference between the first probability and the second probability, a confidence score representative of a confidence that the first user pattern is to be associated with the event;

determining that the confidence score exceeds a threshold; and

in response to determining that the confidence score exceeds the threshold, associating the event with the first user pattern.

2. The computer implemented method of claim 1, wherein the first object includes a first segment and a second segment;

wherein the second object includes a third segment and a fourth segment;

wherein the first object crosses over the second object;

the method further comprising;

processing the plurality of images generated during the event time window to at least determine a third trajectory information for the first segment and the third segment, wherein the third trajectory information identifies a depth jump between a first segment and the third segment; and

determining, based at least in part on the third trajectory information, a third probability that the first user pattern is to be associated with the event.

3. The computer implemented method of claim 1, further comprising:

determining a second event within the defined distance and during the event time window;

determining that the second user pattern is associated with the second event; and

in response to determining that the second user pattern is associated with the second event, adjusting at least one of the second probability, the first probability, or the confidence score.

4. The computer implemented method of claim 1, wherein determining the first probability includes:

determining that at least one of a curvature of the first object or a length of the first object corresponds with an expected arm curvature or an expected arm length.

5. A computing system, comprising:

a processor; and

a memory coupled to the processor and storing program instructions that when executed by the processor cause the processor to at least;

in response to a detection, by one or more input components, of an occurrence of an event at an event location within, a materials handling facility, receive;

an event notification that includes an indication of the event location;

a first user pattern location corresponding to a first user pattern; and

a second user pattern location corresponding to a second user pattern;

determine, based at least in part on the first user pattern location, that the first user pattern is within a defined distance of the event location;

determine, based at least in part on the second user pattern location, that the second user pattern is within the defined distance of the event location;

process a plurality of images that include a representation of the event location to determine an object between the event location and at least one of the first user pattern or the second user pattern;

determine, based at least in part on the object and the first user pattern location, a first probability that the object is a first arm extending from the first user pattern to the event location;

determine, based at least in part on the object and the second user pattern location, a second probability that the object is a second arm extending from the second user pattern to the event location; and

associate the first user pattern with the event based at least in part on a determination that the first probability is greater than the second probability.

6. The computing system of claim 5, wherein the first probability is further determined based at least in part on one or more of:

a number of segments of the object;

a curvature of the object;

a length of the object;

a length of a first segment of the object;

a connection point between the object and the first user pattern;

an orientation of the first user pattern;

a position of the object with respect to the first user pattern;

a color of at least a portion of the object and a color of at least a portion of the first user pattern;

or

a position of the object with respect to the event location.

7. The computing system of claim 5, wherein the program instructions that when executed by the processor further cause the processor to at least:

process the plurality of images to determine a second object between the event location and at least one of the first user pattern and the second user pattern;

determine, based at least in part on the second object and the first user pattern location, a third probability that the second object is the first arm extending from the first user pattern to the event location;

determine, based at least in part on the second object and the second user pattern location, a fourth probability that the second object is the second arm extending from the second user pattern to the event location; and

wherein the first user pattern is associated with the event based at least in part on a determination that the first probability is greater than the second probability, the third probability, and the fourth probability.

8. The computing system of claim 7, wherein the program instructions that when executed by the processor further cause the processor to at least:

determine a touch point at an inventory location within the defined distance of the event location;

process the plurality of mages to determine that the second object connects with the touch point;

process the plurality of images to determine that the second object connects with the second user pattern; and

wherein the second probability is further determined based at least in part on the determination that the second object connects with the touch point and the second user pattern.

9. The computing system of claim 7, wherein the program instructions that when executed by the processor further cause the processor to at least:

determine that the object is connected to the first user pattern;

determine that the second object is connected to the second user pattern;

determine a first touch point between the object and a first inventory location within the defined distance of the event location, wherein the first touch point has a first touch point duration and a first touch point size;

determine a second touch point between the second object and a second inventory location within the defined distance of the event location, wherein the second touch point has a second touch point duration and a second touch point size; and

wherein the first user pattern is associated with the event based at least in part on a determination that at least one of;

the first touch point duration is representative of the event;

the first touch point size is representative of the event;

the second touch point duration is not representative of the event;

or

the second touch point size is not representative of the event.

10. The computing system of claim 5, wherein:

the program instructions that when executed by the processor to determine the first probability further include instructions that cause the processor to at least;

determine the first probability based at least in part on depth information corresponding to a position of the object; and

the program instructions further cause the processor to at least;

determine, based at least in part on the depth information, at least one of a curvature of the object, a length of the object, a connection of the object with the first user pattern, or a position of the object with respect to the event location.

11. The computing system of claim 10, wherein the program instructions that when executed by the processor further cause the processor to at least:

compare the length of the object with an expected am length; and

the expected arm length is at least one of;

a known arm length of a user associated with the first user pattern;

or

a stored arm length.

12. The computing system of claim 5, wherein the program instructions that when executed by the processor further cause the processor to at least:

process the plurality of images to determine an orientation of a first user associated with the first user pattern; and

wherein the first probability is further based at least in part on the orientation of the first user.

13. The computing system of claim 5, wherein the program instructions that when executed by the processor further cause the processor to at least:

process the plurality of images to determine;

first user depth information for a first user associated with the first user pattern; and

object depth information for the object;

determine a connection depth at which the object connects with the first user; and

determine that the connection depth corresponds with a depth of a shoulder of the first user.

14. The computing system of claim 5, wherein the program instructions that when executed by the processor further cause the processor to at least:

determine for the first user pattern a confidence score that a first user associated with the first user pattern was involved in the event, wherein the confidence score is based at least in part on a difference between the first probability and the second probability;

determine that the confidence score does not exceed a threshold; and

provide the plurality of images to a second user for manual review; and

wherein the first user pattern is associated with the event based at least in part on a response received from the second user.

15. A computer implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

in response to a detection, by one or more input components, of an occurrence of an event at an event location within a materials handling facility, receiving;

an event notification that includes an indication of the event location for the event and an event time indicating a time of the occurrence of the event;

a first user pattern location corresponding to a first user pattern at the event time; and

a second user pattern location corresponding to a second user pattern the event time;

specifying, based on the event time, an event time window that begins before the event time and ends after the event time;

determining, based at least in part on the, first user pattern location, that the first user pattern is located within a defined distance of the event location during the event time window;

determining, based at least in part on the second user pattern location, that the second user pattern is located within the defined distance of the event location during the event time window;

processing a plurality of images received from a first camera positioned above an area within the defined distance to at least identify a first object positioned between the event location and at least one of the first user pattern or the second user pattern, wherein the plurality of images are received from a first camera positioned above the area;

determining, based at least in part on the first user pattern location, a first probability that the first object is a first arm extending from the first user pattern to the event location;

determining, based at least in part on the second user pattern location, a second probability that the first object is a second arm extending from the second user pattern to the event location; and

associating the first user pattern with the event based at least in part on a determination that the first probability is greater than the second probability.

16. The computer implemented method of claim 15, further comprising:

determining that at least a portion of the area within the defined distance is blocked from a first field of view of the first camera;

obtaining images from a second camera wherein a least a portion of a second field of view of the second camera includes at least a portion of the area that is blocked from the first field of view of the first camera; and

determining that the at least a portion of the area that is blocked from the first field of view of the first camera is viewable from the images obtained from the second camera.

17. The computer implemented method of claim 15, further comprising:

determining that at least a portion of the area within the defined distance is blocked from a first field of view of the first camera;

obtaining a second plurality of images from a second camera wherein a least a portion of a second field of view of the second camera includes at least a portion of the area that is blocked from the first field of view of the first camera;

determining that the at least a portion of the area that is blocked from the first field of view of the first camera is blocked from a field of view of the second camera; and

sending the plurality of images to a user for manual review.

18. The computer implemented method of claim 17, wherein the second camera is positioned on an inventory shelf within the defined distance.

19. The computer implemented method of claim 15, wherein the event time window identifies a period of time that begins before the event time and extends after the event time.

20. The computer implemented method of claim 15, wherein the first probability is further based at least in part on a determination that the first object is connected to a user associated with the first user pattern.