Camera hierarchy for monitoring large facilities

• US 10,291,862 B1
• Filed: 12/23/2014
• Issued: 05/14/2019
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Liberato, Frank Florian, Raghavan, Sudarshan Narasimha, Maldonado, Emilio Ian, Koka, Muralidhar, Chinoy, Ammar, Bibireata, Daniel, Asmi, Yasser Baseer, Jiang, Hao, Kim, Jaechul, Desai, Nishitkumar Ashokkumar
• Primary Examiner(s): Navas, Jr., Edemio
• Application Number:
  US14/581,986
• Time in Patent Office: 1,603 Days
• Field of Search: None
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Described is a multiple-camera system for use in capturing images of users within a materials handling facility and processing those images to monitor the movement of users. For large materials handling facilities, a large number of cameras may be required to monitor the facility. Processing of the data generated from a large number of cameras becomes difficult. The implementations described herein include a hierarchy that allows image data from any number of cameras within a materials handling facility to be processed without substantially increasing the processing time needed or sacrificing processing capabilities.

Claims

1. A vision based monitoring system, comprising:

a first camera cluster positioned in a first area of a materials handling facility, the first camera cluster including a first plurality of cameras;

a second camera cluster positioned in a second area of the materials handling facility, the second camera cluster including a second plurality of cameras;

a first camera cluster processing system remote from the materials handling facility, the first camera cluster processing system including;

a first one or more processors; and

a first memory storing first program instructions that when executed by the first one or more processors cause the first one or more processors to at least;

receive, from each camera of the first plurality of cameras of the first camera cluster, image data representative of at least a portion of an image obtained from a respective camera of the first plurality of cameras;

process the image data received from each respective camera of the first plurality of cameras of the first camera cluster to determine a first user pattern representative of a first user positioned in a first field of view of a first camera of the first plurality of cameras of the first camera cluster and in a second field of view of a second camera of the first plurality of cameras of the first camera cluster, wherein determination of the first user pattern includes;

associating first image data of the first camera representative of the first user with a respective physical location of the materials handling facility in the first field of view of the first camera;

associating second image data of the second camera representative of the first user with a respective physical location of the materials handling facility in the second field of view of the second camera;

determining that a first portion of the first image data is associated with a first location;

determining that a second portion of the second image data is associated with the first location, wherein the first location is within the first field of view that at least partially overlaps with the second field of view; and

including, in the first user pattern, a first color data and a first depth information corresponding to the first portion of the first image data as representative of the first location, without including a second color data and a second depth information corresponding to the second portion of the second image data in the first user pattern;

assign a first identifier to the first user pattern, wherein the first identifier identifies the first user pattern; and

send to a cluster aggregation system, the first identifier and first location information corresponding to the first location, without sending to the cluster aggregation system the first image data or the second image data;

a second camera cluster processing system remote from the materials handling facility, the second camera cluster processing system including;

a second one or more processors; and

a second memory storing second program instructions that when executed by the second one or more processors further cause the second one or more processors to at least;

receive, from each camera of the second plurality of cameras of the second camera cluster, image data representative of at least a portion of an image obtained from a respective camera of the second plurality of cameras of the second camera cluster,

process the image data received from each respective camera of the second plurality of cameras of the second camera cluster to determine a second user pattern representative of a second user positioned in a third field of view of a third camera of the second plurality of cameras of the second camera cluster and in a fourth field of view of a fourth camera of the second plurality of cameras of the second camera cluster, wherein determination of the second user pattern includes;

associating third image data of the third camera representative of the second user with a respective physical location of the materials handling facility in the third field of view of the third camera;

associating fourth image data of the fourth camera representative of the second user with a respective physical location of the materials handling facility in the fourth field of view of the fourth camera;

determining that a third portion of the third image data is associated with a second location;

determining that a fourth portion of the fourth image data is associated with the second location, wherein the second location is within the third field of view that at least partially overlaps with the fourth field of view; and

including, in the second user pattern, a third color data and a third depth information corresponding to the third portion of the third image data as representative of the second location, without including a fourth color data and a fourth depth information corresponding to the fourth portion of the fourth image data in the second user pattern; and

assign a second identifier to the second user pattern, wherein the second identifier identifies the second user pattern;

send to the cluster aggregation system, the second identifier and second location information corresponding to the second location of the second user pattern, without sending to the cluster aggregation system the third image data or the fourth image data;

the cluster aggregation system remote from the materials handling facility, the cluster aggregation system including;

a third one or more processors; and

a third memory storing third program instructions that when executed by the third one or more processors cause the third one or more processors to at least;

receive from the first camera cluster processing system the first identifier and the first location information, without receiving the image data received from each respective camera of the first plurality of cameras;

receive from the second camera cluster processing system the second identifier and the second location information, without receiving the image data received from each respective camera of the second plurality of cameras;

determine an overlap between at least a portion of the first location information and the second location information;

in response to determination of an overlap;

determine, based at least in part on the first location information and a first prior location information for the first user, a first trajectory of the first user;

determine, based at least in part on the second location information and a second prior location information for the second user, a second trajectory of the second user; and

determine, based at least in part on the first trajectory and the second trajectory, that the first user and the second user are a same user; and

associate at least one of the first location information, the second location information, the first identifier, or the second identifier with a user identifier representative of the same user.

2. The vision based monitoring system of claim 1, further comprising:

at least one computing component coupled to at least one of the first plurality of cameras, the at least one computing component including;

a fourth one or more processors; and

a fourth memory storing fourth program instructions that when executed by the fourth one or more processors cause the fourth one or more processors to at least;

compare obtained image data corresponding to an image obtained by the at least one of the first plurality of cameras with baseline image data associated with the at least one camera to determine obtained image data that is different than the baseline image data; and

establish the image data to include only a portion of the obtained image data that is different than the baseline image data.

3. The vision based monitoring system of claim 1, wherein:

the first plurality of cameras of the first camera cluster is between one-hundred and five-hundred cameras; and

the second plurality of cameras of the second camera cluster is between one-hundred and five-hundred cameras.

4. The vision based monitoring system of claim 1, wherein the third program instructions of the cluster aggregation system are further configured to at least:

receive, from each of a plurality of camera cluster processing systems, location information corresponding to locations of user patterns determined by the camera cluster processing system; and

aggregate the location information to determine a location of each of a plurality of users within the materials handling facility, wherein;

the first location information received from the first camera cluster processing system and the second location information received from the second camera cluster processing system is determined to correspond to the first user; and

utilize either the first location information or the second location information as representative of a location of the first user.

5. A system, comprising:

a plurality of cluster processing systems, each of the plurality of cluster processing systems including a respective one or more processors and a respective memory storing respective program instructions that when executed by the respective one or more processors cause the respective one or more processors to at least;

receive image data from each of a plurality of cameras associated with a respective cluster of cameras located in a materials handling facility;

process the received image data to determine a user pattern representative of a user located in a field of view of a first camera of the plurality of cameras of the respective cluster of cameras and a second camera of the plurality of cameras of the respective cluster of cameras, wherein determination of the user pattern includes;

associating first image data of the first camera representative of the user with a respective physical location of the materials handling facility in the first field of view of the first camera;

associating second image data of the second camera representative of the user with a respective physical location of the materials handling facility in the second field of view of the second camera;

determining that a first portion of the first image data is associated with a first location;

determining that a second portion of the second image data is associated with the first location, wherein the first location is within the first field of view that at least partially overlaps with the second field of view; and

including, in the user pattern, a first color data and a first depth information corresponding to the first portion of the first image data as representative of the first location, without including a second color data and a second depth information corresponding to the second portion of the second image data in the user pattern;

assign location information representative of the first location of the user pattern in the materials handling facility;

assign a session identifier to the user pattern, the session identifier identifying the user pattern without identifying a user;

send the location information and the session identifier to a cluster aggregation system, without sending to the cluster aggregation system the image data received from the plurality of cameras associated with the respective cluster of cameras;

the cluster aggregation system including;

a first one or more processors; and

a memory storing first program instructions that when executed by the first one or more processors cause the first one or more processors to at least;

receive from each of the plurality of cluster processing systems the corresponding location information and corresponding session identifiers, without receiving the image data received from the plurality of cameras associated with the respective cluster of cameras;

determine that the location information of a first session identifier and the location information of a second session identifier at least partially overlap;

in response to determination of an overlap;

determine, based at least in part on the location information of the first session identifier and a first prior location information for the first session identifier, a first trajectory;

determine, based at least in part on the location information of the second session identifier and a second prior location information for the second session identifier, a second trajectory; and

determine, based at least in part on the first trajectory and the second trajectory, that the first session identifier and the second session identifier correspond to a same user identifier; and

store in a data store a mapping between the same user identifier and at least one of the first session identifier or the second session identifier.

6. The system of claim 5, wherein the image data is processed to determine the user pattern by at least:

determine based at least in part on depth information included in the image data a shape represented in the image data; and

determine a movement of the shape over a period of time.

7. The system of claim 5, wherein the first program instructions of the cluster aggregation system further cause the first one or more processors to at least:

determine that at least a portion of the location information received from a first cluster processing system corresponds to a same location as at least a portion of the location information received from a second cluster processing system;

determine that the at least a portion of the location information received from the first cluster processing system and the at least a portion of the location information received from the second cluster processing system correspond to a same user pattern; and

select the at least a portion of the location information received from the first cluster processing system as representative of the user pattern.

8. The system of claim 7, wherein the at least a portion of the location information received from the first cluster processing system corresponds to a location:

included in a field of view of a first camera associated with a first cluster; and

included in a field of view of a second camera associated with a second cluster.

9. The system of claim 5, wherein:

each of the plurality of cluster processing systems receive reduced image data that does not include image data matching baseline image data associated with a camera of the plurality of cameras associated with a respective cluster.

10. The system of claim 5, wherein each of the respective program instructions further cause the respective one or more processors to at least:

process the received image data to reduce the data sent to the cluster aggregation system by at least;

process the received image data to determine a plurality of objects represented in the image data that do not correspond to a baseline image; and

reduce the image data to include only image data corresponding to the plurality of objects.

11. The system of claim 5, wherein the first program instructions of the cluster aggregation system further cause the first one or more processors to at least:

determine a trajectory for a user pattern based at least in part on the first location information and the session identifier sent by a cluster processing system for the user pattern and a prior location information corresponding to the session identifier.

12. The system of claim 5, wherein each of the plurality of cluster processing systems and the cluster aggregation system is remote from the materials handling facility.

13. The system of claim 12, wherein the first program instructions of the cluster aggregation system further cause the first one or more processors to send a respective user identifier and corresponding first location information to at least one system that provides a service for the materials handling facility.

14. The system of claim 12, wherein a user identifier and corresponding first location information is utilized to monitor a location of the user within the materials handling facility.

15. A system, comprising:

a plurality of cameras positioned at respective locations within a materials handling facility, each camera of the plurality of cameras associated with one cluster of a plurality of clusters, each cluster associated with a sub-plurality of cameras of the plurality of cameras;

a plurality of cluster processing systems, each of the plurality of cluster processing systems associated with one cluster of the plurality of clusters and including;

a respective one or more processors; and

a respective memory storing respective program instructions that when executed by the respective one or more processors cause the respective one or more processors to at least;

receive image data from each of a first camera and a second camera associated with the respective cluster;

process the image data to determine a user pattern included in the image data, wherein determination of the user pattern includes;

associating first image data of the first camera representative of a user with a respective physical location of the materials handling facility in a first field of view of the first camera;

associating second image data of the second camera representative of the user with a respective physical location of the materials handling facility in a second field of view of the second camera;

determining that a first portion of the first image data is associated with a first location;

determining that a second portion of the second image data is associated with the first location, wherein the first location is within the first field of view that at least partially overlaps with the second field of view; and

including, in the user pattern, a first color data and a first depth information corresponding to the first portion of the first image data as representative of the first location, without including a second color data and a second depth information corresponding to the second portion of the second image data in the user pattern;

assign an identifier to the user pattern; and

send to a cluster aggregation system;

location information corresponding to the location of the user pattern; and

the respective identifiers assigned to the user patterns, without sending to the cluster aggregation system the image data;

the cluster aggregation system associated with each of the plurality of cluster processing systems and including;

a first one or more processors; and

a first memory storing first program instructions that when executed by the first one or more processors cause the first one or more processors to at least;

receive, from each of the plurality of cluster processing systems, location information corresponding to the respective user pattern, and the respective identifier assigned to the user pattern, without receiving the image data; and

determine an overlap between at least a portion of a first location information and at least a portion of a second location information;

in response to determination of an overlap;

determine, based at least in part on the first location information and a first prior location information for a first user pattern associated with the first location information, a first trajectory;

determine, based at least in part on the second location information and a second prior location information for a second user pattern associated with the second location information, a second trajectory; and

determine, based at least in part on the first trajectory and the second trajectory, that the first user pattern and the second user pattern represent a same user; and

store in a data store a mapping between a user identifier for the same user and at least one of a first identifier corresponding to the first location or a second identifier corresponding to the second location.

16. The system of claim 15, wherein each of the cameras provide image data that includes color information for an image and depth information for objects represented in the image.

17. The system of claim 15, wherein the cameras are mounted at defined positions above a surface of the materials handling facility, at defined orientations with respect to the materials handling facility, and at defined angles with respect to the surface.

18. The system of claim 17, where a field of view of at least two cameras overlap above the surface of the materials handling facility.

19. The system of claim 15, wherein the image data of at least one camera is reduced to exclude image data that corresponds to a baseline image data associated with the at least one camera.

20. The system of claim 15, wherein the location information corresponding to each of the plurality of clusters, when combined, is representative of all users located in the materials handling facility.