Autonomous store tracking system

• US 10,282,852 B1
• Filed: 01/23/2019
• Issued: 05/07/2019
• Est. Priority Date: 07/16/2018
• Status: Active Grant

• Current Assignee: Accel Robotics Corporation
• Original Assignee: Accel Robotics Corporation
• Inventors: Buibas, Marius, Quinn, John, Feigum, Kaylee, Petre, Csaba, Maseda, Michael Brandon, Cseh, Martin Alan
• Primary Examiner(s): Fujita, Katrina R
• Application Number:
  US16/254,776
• Time in Patent Office: 104 Days
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A system that analyzes camera images to track a person in an autonomous store, and to determine when a tracked person takes or moves items in the store. The system may associate a field of influence volume around a person'"'"'s location; intersection of this volume with an item storage area, such as a shelf, may trigger the system to look for changes in the items on the shelf. Items that are taken from, placed on, or moved on a shelf may be determined by a neural network that processes before and after images of the shelf. Person tracking may be performed by analyzing images from fisheye ceiling cameras projected onto a plane horizontal to the floor. Projected ceiling camera images may be analyzed using a neural network trained to recognize shopper locations. The autonomous store may include modular ceiling and shelving fixtures that contain cameras, lights, processors, and networking.

Claims

1. An autonomous store tracking system, comprising

a processor configured to

obtain a 3D model of a store that contains items and item storage areas;

receive a time sequence of images from each camera of a plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period;

analyze said time sequence of images and said 3D model of said store to

determine a sequence of locations of a person in said store during said time period; and

calculate a field of influence volume around each location of said sequence of locations;

when said field of influence volume intersects an item storage area of said item storage areas during an interaction time period within said time period,

receive a first image from a camera in said store oriented to view said item storage area, wherein said first image is captured before or at the beginning of said interaction time period;

receive a second image from said camera in said store oriented to view said item storage area, wherein said second image is captured after or at the end of said interaction time period;

set an input of a neural network to said first image and said second image,

wherein said neural network outputs

a probability that each item of said items is moved during said interaction time period, and

a probability that each action of a set of actions is performed during said interaction time period;

select an item from said items with a highest probability of being moved during said time period in an output of said neural network;

select an action from said set of actions with a highest probability of being performed during said time period in said output of said neural network, and,

attribute said action and said item to said person.

2. The autonomous store tracking system of claim 1, wherein

said plurality of cameras in said store comprise a plurality of ceiling cameras mounted on a ceiling of said store;

said analyze said time sequence of images and said 3D model of said store comprises

project said time sequence of images from each ceiling camera onto a plane parallel to a floor of said store, to form a time sequence of projected images corresponding to each ceiling camera;

analyze said time sequence of projected images corresponding to each ceiling camera, and said 3D model of said store to

determine said sequence of locations of a person in said store during said time period; and

calculate said field of influence volume around each location of said sequence of locations.

3. The autonomous store tracking system of claim 2, wherein said each ceiling camera of said plurality of ceiling cameras is a fisheye camera.

4. The autonomous store tracking system of claim 2, wherein

said field of influence volume around each location of said sequence of locations is a translated copy of a standardized shape.

5. The autonomous store tracking system of claim 4, wherein said standardized shape comprises a cylinder.

6. The autonomous store tracking system of claim 2, wherein said each location of said sequence of locations comprises a point.

7. The autonomous store tracking system of claim 2, wherein said determine said sequence of locations of a person in said store during said time period comprises

for each time in said time sequence of projected images corresponding to each ceiling camera,

subtract a store background image from each projected image of said projected images captured at said each time to form a corresponding plurality of masks at said each time;

combine said plurality of masks at said each time to form a combined mask; and

,

identify a location of said person at said each time as a high intensity location in said combined mask.

8. The autonomous store tracking system of claim 2, wherein said determine said sequence of locations of a person in said store during said time period comprises

for each time in said time sequence of projected images corresponding to each ceiling camera,

input said projected images captured at said each time into a machine learning system that outputs an intensity map, wherein

said intensity map comprises a likelihood at each location that a person is at said location.

9. The autonomous store tracking system of claim 8, wherein said machine learning system comprises a neural network.

10. The autonomous store tracking system of claim 9, wherein said neural network comprises a fully convolutional network.

11. The autonomous store tracking system of claim 10, wherein said fully convolutional network comprises

a first half subnetwork comprising a plurality of copies of a feature extraction network, each copy of said plurality of copies corresponding to a ceiling camera of said plurality of ceiling cameras, wherein said each copy comprises an input layer comprising a corresponding projected image of said projected images;

a feature merging layer coupled to said first half subnetwork, wherein said feature merging layer averages outputs of said plurality of copies of said feature extraction network; and

,

a second half subnetwork coupled to said feature merging layer, wherein an output layer of said second half subnetwork comprises said intensity map.

12. The autonomous store tracking system of claim 11, wherein said input layer of said each copy further comprises

a position map corresponding to the ceiling camera corresponding to said copy, wherein a value of said position map at a location is a function of a distance between said location on said plane and said each ceiling camera.

13. The autonomous store tracking system of claim 8, wherein said determine said sequence of locations of a person in said store during said time period further comprises

input into said machine learning system a position map corresponding to each ceiling camera of said plurality of ceiling cameras, wherein a value of said position map at a location is a function of a distance between said location on said plane and said each ceiling camera.

14. The autonomous store tracking system of claim 2, further comprising

a modular ceiling comprising

a longitudinal rail mounted to said ceiling of said store;

one or more transverse rails, wherein each transverse rail of said one or more transverse rails is mounted to said longitudinal rail;

one or more integrated lighting-camera modules mounted to said each transverse rail, wherein each integrated lighting-camera module of said one or more integrated lighting-camera modules comprises

a lighting element surrounding a center area; and

,

two or more ceiling-mounted cameras of said plurality of ceiling-mounted cameras mounted in said center area.

15. The autonomous store tracking system of claim 14, wherein

a position of said each transverse rail along said longitudinal rail is adjustable;

a position of said each integrated lighting-camera module along said each transverse rail is adjustable;

said center area comprises a camera module comprising at least one slot into which said two or more ceiling-mounted cameras are attached; and

,

a position of each ceiling-mounted camera of said two or more ceiling-mounted cameras in said at least one slot is adjustable.

16. The autonomous store tracking system of claim 1, further comprising

one or more modular shelves, each modular shelf of said one or more modular shelves comprising

at least one camera module mounted on a bottom side of said each modular shelf, wherein each camera module of said at least one camera module comprises two or more downward-facing cameras;

at least one lighting module, wherein each lighting module of said at least one lighting module comprises a downward-facing light;

a right-facing camera mounted on or proximal to a left edge of said each modular shelf;

a left-facing camera mounted on or proximal to a right edge of said each modular shelf;

a processor; and

,

a network switch.

17. The autonomous store tracking system of claim 16, wherein

said item storage areas comprise said one or more modular shelves;

said camera in said store oriented to view said item storage area comprises a downward-facing camera of said downward-facing cameras of a modular shelf of said one or more modular shelves located above said item storage area.

18. The autonomous store tracking system of claim 16, wherein

said each modular shelf comprises a front rail and a back rail onto which said each camera module and said each lighting module are attached;

a position of said each camera module along said front rail and said back rail is adjustable; and

;

a position of said each lighting module along said front rail and said back rail is adjustable.

19. The autonomous store tracking system of claim 16, wherein

said each camera module comprises at least one slot into which said two or more downward-facing cameras are attached;

and,

a position of each downward-facing camera of said two or more downward-facing cameras in said at least one slot is adjustable.

20. An autonomous store tracking system, comprising:

a modular ceiling in a store comprising

a longitudinal rail mounted to a ceiling of said store;

one or more transverse rails, wherein each transverse rail of said one or more transverse rails is mounted to said longitudinal rail;

one or more integrated lighting-camera modules mounted to said each transverse rail, wherein each integrated lighting-camera module of said one or more integrated lighting-camera modules comprises

a lighting element surrounding a center area; and

two or more ceiling-mounted cameras of a plurality of ceiling-mounted cameras of said store mounted in said center area;

wherein

a position of said each transverse rail along said longitudinal rail is adjustable;

a position of said each integrated lighting-camera module along said each transverse rail is adjustable;

said center area comprises a camera module comprising at least one slot into which said two or more ceiling-mounted cameras are attached; and

a position of each ceiling-mounted camera of said two or more ceiling-mounted cameras in said at least one slot is adjustable;

one or more modular shelves in said store, each modular shelf of said one or more modular shelves comprising

at least one camera module mounted on a bottom side of said each modular shelf, wherein each camera module of said at least one camera module comprises two or more downward-facing cameras;

at least one lighting module, wherein each lighting module of said at least one lighting module comprises a downward-facing light;

a right-facing camera mounted on or proximal to a left edge of said each modular shelf;

a left-facing camera mounted on or proximal to a right edge of said each modular shelf;

a processor; and

a network switch;

wherein

said each modular shelf is an item storage area for one or more items in said store;

said each modular shelf comprises a front rail and a back rail onto which said each camera module and said each lighting module are attached;

a position of said each camera module along said front rail and said back rail is adjustable;

a position of said each lighting module along said front rail and said back rail is adjustable;

said each camera module comprises at least one slot into which said two or more downward-facing cameras are attached; and

a position of each downward-facing camera of said two or more downward-facing cameras in said at least one slot is adjustable;

a processor configured to

obtain a 3D model of said store;

receive a time sequence of images from each camera of said plurality of ceiling-mounted cameras, wherein said time sequence of images from each camera is captured over a time period;

project said time sequence of images from each ceiling camera onto a plane parallel to a floor of said store, to form a time sequence of projected images corresponding to each ceiling camera;

analyze said time sequence of projected images corresponding to each ceiling camera, and said 3D model of said store to

determine a sequence of locations of a person in said store during said time period; and

calculate a field of influence volume around each location of said sequence of locations;

when said field of influence volume intersects an item storage area of said item storage areas during an interaction time period within said time period,

receive a first image from a camera in said store oriented to view said item storage area, wherein said first image is captured before or at the beginning of said interaction time period;

receive a second image from said camera in said store oriented to view said item storage area, wherein said second image is captured after or at the end of said interaction time period;

set an input of a neural network to said first image and said second image,

wherein said neural network outputs

a probability that each item of said items is moved during said interaction time period, and

a probability that each action of a set of actions is performed during said interaction time period;

select an item from said items with a highest probability of being moved during said time period in an output of said neural network;

select an action from said set of actions with a highest probability of being performed during said time period in said output of said neural network, and,

attribute said action and said item to said person.