Autonomous store system that analyzes camera images to track people and their interactions with items

• US 10,373,322 B1
• Filed: 07/16/2018
• Issued: 08/06/2019
• Est. Priority Date: 07/16/2018
• Status: Active Grant

• Current Assignee: Accel Robotics Corporation
• Original Assignee: Accel Robotics Corporation
• Inventors: Buibas, Marius, Quinn, John, Feigum, Kaylee, Petre, Csaba
• Primary Examiner(s): Fujita, Katrina R
• Application Number:
  US16/036,754
• Time in Patent Office: 386 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

An autonomous store system that analyzes camera images to track people and their interactions with items using a processor that obtains a 3D model of a store that contains items and item storage areas. Receives images from cameras captured over a time period and analyzes the images and the 3D model of the store to detect a person in the store based on the images, calculates a trajectory of the person, identifies an item storage area proximal to the trajectory of the person during an interaction time period, analyzes two or more images to identify an item within the item storage area that is moved during the interaction time period. The images are captured within or proximal in time to the interaction time period, and the images contain views of the item storage area, and attribute motion of the item to the person. Enables calibration and placement algorithms for cameras.

Claims

1. An autonomous store system that analyzes camera images to track people and their interactions with items, comprising:

a processor configured to

obtain a 3D model of a store that contains items and item storage areas;

receive a respective time sequence of images from cameras in said store, wherein said time sequence of images is captured over a time period; and

,

analyze said time sequence of images from each camera of said cameras, and said 3D model of said store to

detect a person in said store based on said time sequence of images;

track movement of said person through said store;

calculate a trajectory of said person across said time period;

calculate a 3D field of influence volume of said person around said person at points of time during said time period,

wherein said 3D field of influence volume indicates a region in which said person could potentially move said items;

calculate a trajectory of the 3D field of influence volume through said store;

identify an item storage area of said item storage areas that is at or proximal to said trajectory of said person and proximal to said 3D field of influence volume during an interaction time period within said time period;

identify said item storage area comprising a 3D location that intersects said 3D field of influence volume of said person around said person during said interaction time period,

wherein said 3D location comprises a location of said item within said item storage area that moved during said interaction time period;

analyze two or more images of said time sequence of images to identify an item of said items within said item storage area that moves during said interaction time period, wherein

said two or more images are captured within or proximal in time to said interaction time period, and

said two or more images contain views of said item storage area; and

,

attribute motion of said item to said person, wherein said trajectory of said 3D field of influence volume shows that said 3D field of influence of volume of said person around said person intersected said 3D location of said item during a time interval proximal in time to said motion.

2. The system of claim 1 wherein said processor is further configured to interface with a point of sale computer and charge an amount associated with said item to said person without a cashier and wherein optionally a description of said item is sent to a mobile device associated with said person and wherein said processor or point of sale computer is configured to accept a confirmation from said mobile device that said item is correct or in dispute.

3. The system of claim 1 wherein

each image of said time sequence of images is a 2D image;

and,

said calculate said trajectory of said person comprises calculate a 3D location and orientation of said person and at least one body landmark from two or more 2D projections of said person in said time sequence of images.

4. The system of claim 1 wherein said calculate said 3D field of influence volume of said person around said person comprises:

calculate a spatial probability distribution for multiple landmarks on said person at said points of time during said time period, wherein each landmark of said multiple landmarks corresponds to a location on a body part of said person.

5. The system of claim 4 wherein said 3D field of influence volume of said person around said person comprises points comprising a distance to a closest landmark of said multiple landmarks that is less than or equal to a threshold distance.

6. The system of claim 4 wherein said 3D field of influence volume of said person around said person comprises a union of probable zones for each landmark of said multiple landmarks, wherein each probable zone of said probable zones contains a threshold probability of said spatial probability distribution for a corresponding landmark.

7. The system of claim 4 wherein said calculate said spatial probability distribution for multiple landmarks on said person at said points of time during said time period comprises:

calculate a predicated spatial probability distribution for said multiple landmarks at one or more points of time during said time period based on a physics model; and

,

calculate a corrected spatial probability distribution at one or more points of time during said time period based on observations of one or more of said multiple landmarks in said time sequence of images.

8. The system of claim 4 wherein

said landmarks comprise eyes of said person;

and,

said processor is further configured to

calculate a field of view of said person based on a location of said eyes of said person; and

detect that said person is looking at an item of said items when said item is in said field of view.

9. The system of claim 1 wherein said processor is further configured to analyze said two or more images of said time sequence of images to classify said motion of said item as a type of motion comprising taking, putting or moving.

10. The system of claim 1 wherein said analyze two or more images of said time sequence of images to identify an item of said items within said item storage area that moves during said interaction time period comprises:

obtain a neural network trained to recognize items from changes across images;

set an input layer of said neural network to said two or more images; and

,

calculate a probability associated with said item based on an output layer of said neural network.

11. The system of claim 10 wherein said neural network is further trained to classify an action performed on an item into classes comprising taking, putting, or moving.

12. The system of claim 10 further comprising:

a verification system configured to accept input confirming or denying that said person is associated with said motion of said item;

a machine learning system configured to

receive said input confirming or denying that said person is associated with said motion of said item; and

update said neural network based on said input.

13. The system of claim 1 wherein said processor is further configured to

identify one or more distinguishing characteristics of said person by analyzing a first subset of said time sequence of images;

and,

recognize said person in a second subset of said time sequence of images using said one or more distinguishing characteristics.

14. The system of claim 13 wherein said recognize said person in said second subset occurs without determination of an identity of said person.

15. The system of claim 13 wherein said second subset of said time sequence of images contains images of said person and images of a second person.

16. The system of claim 13 wherein said one or distinguishing characteristics comprise one or more of

shape or size of one or more body segments of said person;

shape, size, color, or texture of one or more articles of clothing worn by said person; and

,

gait pattern of said person.

17. The system of claim 1 wherein said processor is further configured to:

obtain camera calibration data for each camera of said cameras in said store; and

,

analyze said time sequence of images from each camera of said cameras using said camera calibration data.

18. The system of claim 17 wherein said processor is further configured to:

obtain calibration images from each camera of said cameras; and

,

calculate said camera calibration data from said calibration images.

19. The system of claim 18 wherein

said calibration images comprise images captured of one or more synchronization events;

and,

said camera calibration data comprises temporal offsets among said cameras.

20. The system of claim 18 wherein

said calibration images comprise images captured of one or more markers placed in said store at locations defined relative to said 3D model;

and,

said camera calibration data comprises position and orientation of said cameras with respect to said 3D model.

21. The system of claim 18 wherein

said calibration images comprise images captured of one or more color calibration targets located in said store;

said camera calibration data comprises color mapping data between each camera of said cameras and a standard color space.

22. The system of claim 21 wherein said camera calibration processor is further configured to recalculate said color mapping data when lighting conditions change in said store.

23. The system of claim 1 further comprising a camera placement optimization processor configured to:

obtain said 3D model of said store; and

,

calculate a recommended number of said cameras in said store and a recommended location and orientation of each camera of said cameras in said store.

24. The system of claim 23 wherein said calculate a recommended number of said cameras in said store and a recommended location and orientation of each camera of said cameras in said store comprises:

obtain a set of potential camera locations and orientations in said store;

obtain a set of item locations in said item storage areas; and

,

iteratively update a proposed number of cameras and a proposed set of camera locations and orientations to obtain a minimum number of cameras and a location and orientation for each camera of said minimum number of cameras such that each item location of said set of item locations is visible to at least two of said minimum number of cameras.

25. The system of claim 1 further comprising said cameras, wherein said cameras are coupled with said processor.

26. The system of claim 1 wherein said processor is further configured to detect shoplifting when said person leaves said store without paying for said item.

27. The system of claim 1 wherein said computer processor is further configured to detect that said person is looking at an item of said items.

28. An autonomous store system that analyzes camera images to track people and their interactions with items, comprising:

a processor configured to

obtain a 3D model of a store that contains items and item storage areas;

receive a respective time sequence of images from cameras in said store, wherein said time sequence of images is captured over a time period;

obtain camera calibration data for each camera of said cameras, said camera calibration data comprising

temporal offsets among said cameras;

position and orientation of said cameras with respect to said 3D model; and

color mapping data between each camera of said cameras and a standard color space; and

,

analyze said time sequence of images from each camera of said cameras, said camera

calibration data, and said 3D model of said store to

detect a person in said store based on said time sequence of images;

track movement of said person through said store;

identify one or more distinguishing characteristics of said person by analyzing a first subset of said time sequence of images, wherein said one or more distinguishing characteristics comprise one or more of

 

shape or size of one or more body segments of said person;

 

shape, size, color, or texture of one or more articles of clothing worn by said person; and

,

 

gait pattern of said person;

recognize said person in a second subset of said time sequence of images using said one or more distinguishing characteristics;

calculate a spatial probability distribution for multiple landmarks on said person at points of time during said time period, wherein each landmark of said multiple landmarks corresponds to a location on a body part of said person;

calculate a 3D field of influence volume of said person around said person at said points of time during said time period based on said spatial probability distribution for said multiple landmarks;

 

wherein said 3D field of influence volume indicates a region in which said person could potentially move said items;

calculate a trajectory of the 3D field of influence volume through said store;

identify an item storage area of said item storage areas comprising a 3D location that intersects said 3D field of influence volume of said person around said person during an interaction time period within said time period,

 

wherein said 3D location comprises a location of said item within said item storage area that moved during said interaction time period;

obtain a neural network trained to recognize items from changes across images;

set an input layer of said neural network to two or more images of said time sequence of images, wherein

 

said two or more images are captured within or proximal in time to said interaction time period, and

 

said two or more images contain views of said item storage area;

identify an item of said items within said item storage area that moves during said interaction time period based on a probability associated with said item obtained from an output layer of said neural network; and

,

attribute motion of said item to said person, wherein said trajectory shows that said 3D field of influence of volume of said person around said person intersected said 3D location of said item during a time interval proximal in time to said motion.