Smart shelf system that integrates images and quantity sensors

• US 10,586,208 B2
• Filed: 07/16/2019
• Issued: 03/10/2020
• Est. Priority Date: 07/16/2018
• Status: Active Grant

• Current Assignee: Accel Robotics Corporation
• Original Assignee: Accel Robotics Corporation
• Inventors: Buibas, Marius, Quinn, John, Feigum, Kaylee, Petre, Csaba, Piekniewski, Filip, Bapst, Aleksander, Yousefisahi, Soheyl, Kuo, Chin-Chang
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US16/513,509
• Publication Number:
  US 20200019921A1
• Time in Patent Office: 238 Days
• CPC Class Codes: G01B 7/18 using change in resistance
• CPC Class Codes: G01G 19/14 for weighing suspended loads G01G3/00 takes precedence; incorporation of weighing devices in cranes B66C1/40, B66C13/16
• CPC Class Codes: G01G 19/4144 for controlling weight of goods in commercial establishments, e.g. supermarket, P.O.S. systems
• CPC Class Codes: G01G 19/42 for counting by weighing G01G19/387 takes precedence
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 2210/21 Collision detection, intersection
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/292 : Multi-camera tracking
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06T 7/97 : Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A system that integrates camera images and quantity sensors to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed may then be attributed to a shopper near the area. Shelves may be divided into storage zones, such as bins or lanes, and a quantity sensor may measure the item quantity in each zone. Quantity changes indicate that a shopper has taken or placed items in the zone. Distance sensors, such as LIDAR, may be used for shelves that push items towards the front. Strain gauges may be used for bins or hanging rods. Quantity changes may trigger analysis of camera images of the shelf to identify the items taken or replaced. Images from multiple cameras that view a shelf may be projected to a vertical plane at the front of the shelf to simplify analysis.

Claims

1. A smart shelf system that integrates images and quantity sensors, comprising:

a plurality of quantity sensors, each corresponding to a storage zone of a plurality of storage zones of an item storage area, wherein

each quantity sensor of said plurality of quantity sensors is configured to generate a quantity signal that is correlated with a quantity of items contained in the storage zone corresponding to said each quantity sensor;

a processor coupled to

said plurality of quantity sensors, and to

a plurality of cameras oriented to view said item storage area;

wherein said processor is configured to

analyze said quantity signal from said plurality of quantity sensors to

identify an affected zone of said plurality of storage zones within which a shopper added or removed at least one item;

determine an action time at which said shopper added or removed said at least one item; and

determine an item quantity change in said affected zone;

obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said action time;

obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said action time;

project said plurality of before images onto a plane in said item storage area to generate a plurality of projected before images;

project said plurality of after images onto said plane to generate a plurality of projected after images;

analyze

said plurality of projected before images, and

said plurality of projected after images, to

identify said at least one item added to or removed from said affected zone at said action time;

associate said at least one item and said item quantity change with said shopper;

obtain a 3D model of a store that contains said item storage area;

receive a time sequence of images from each camera of a second plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period;

analyze said time sequence of images and said 3D model of said store to

determine a sequence of locations of a person in said store during said time period; and

calculate a field of influence volume around each location of said sequence of locations; and

,

when said field of influence volume intersects said item storage area,

identify said shopper as said person; and

,

associate said at least one item and said item quantity change with said person.

2. The system of claim 1, wherein said plane in said item storage area comprises a vertical plane along or proximal to a front face of said item storage area.

3. The system of claim 2, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises

analyze regions of said plurality of projected before images and said plurality of projected after images that correspond to said affected zone.

4. The system of claim 3, wherein said analyze regions of said plurality of projected before images and said plurality of projected after images comprises

when said item quantity change is positive,

input said regions of said plurality of projected after images into a classifier;

when said item quantity change is negative,

input said regions of said plurality of projected before images into said classifier; and

,

identify said at least one item as an output of said classifier.

5. The system of claim 4, wherein said classifier comprises a neural network trained to identify items based on images of said items.

6. The system of claim 1, wherein

each storage zone of said plurality of storage zones comprises a bin comprising a moveable back,

wherein

when said shopper removes an item from said bin, said moveable back moves towards a front end of said bin;

and

when said shopper adds an item to said bin, said moveable back moves away from said front end of said bin; and

,

said quantity signal associated with said each storage zone varies with a position of said moveable back.

7. The system of claim 6, wherein said each quantity sensor associated with said each storage zone comprises a distance sensor configured to measure a distance to said moveable back.

8. The system of claim 7, wherein said distance sensor comprises a LIDAR sensor.

9. The system of claim 8, wherein a resolution of said LIDAR sensor consists of one pixel.

10. The system of claim 7, wherein said distance sensor comprises an ultrasonic range finder.

11. The system of claim 1, wherein

each storage zone of said plurality of storage zones comprises a hanging mount configured to hold items suspended from said hanging mount;

and,

said quantity signal associated with said each storage zone varies with a weight of said items suspended from said hanging mount.

12. The system of claim 11, wherein said each quantity sensor associated with said each storage zone comprises two or more strain gauges.

13. The system of claim 1, wherein

each storage zone of said plurality of storage zones comprises a bin configured to contain items;

and,

said quantity signal associated with said each storage zone varies with a weight of said items in said bin.

14. The system of claim 13, wherein said each quantity sensor associated with said each storage zone comprises a weight scale.

15. The system of claim 1, wherein said plane in said item storage area comprises a vertical plane along or proximal to a front face of said item storage area.

16. The system of claim 15, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises

select a subset of said plurality of projected before images and a subset of said plurality of projected after images wherein a view of said affected zone is not occluded by said field of influence volume;

and,

analyze regions of said subset of said plurality of projected before images and said subset of said plurality of projected after images that correspond to said affected zone.

17. The system of claim 16, wherein said analyze regions of said subset of said plurality of projected before images and said subset of said plurality of projected after images comprises

when said item quantity change is positive,

input said regions of said subset of said plurality of projected after images into a classifier that performs visual identification of items based on images of said items;

when said item quantity change is negative,

input said regions of said subset of said plurality of projected before images into said classifier; and

,

identify said at least one item as an output of said classifier.

18. The system of claim 17, wherein said classifier comprises one or more of

an engineered classifier;

a trained classifier.

19. The system of claim 18, wherein said trained classifier comprises a neural network.

20. A smart shelf system that integrates images and quantity sensors, comprising:

a plurality of quantity sensors, each corresponding to a storage zone of a plurality of storage zones of an item storage area, wherein

each quantity sensor of said plurality of quantity sensors is configured to generate a quantity signal that is correlated with a quantity of items contained in the storage zone corresponding to said each quantity sensor;

a processor coupled to

said plurality of quantity sensors, and to

a plurality of cameras oriented to view said item storage area;

wherein said processor is configured to

analyze said quantity signal from said plurality of quantity sensors to

identify an affected zone of said plurality of storage zones within which a shopper added or removed at least one item;

determine an action time at which said shopper added or removed said at least one item; and

determine an item quantity change in said affected zone;

wherein

 

each storage zone of said plurality of storage zones comprises a bin comprising a moveable back, wherein

 

when said shopper removes an item from said bin, said moveable back moves towards a front end of said bin; and

 

when said shopper adds an item to said bin, said moveable back moves away from said front end of said bin;

 

said quantity signal associated with said each storage zone varies with a position of said moveable back;

 

said each quantity sensor associated with said each storage zone comprises a distance sensor configured to measure a distance to said moveable back;

 

said distance sensor comprises a LIDAR sensor; and

,

 

a resolution of said LIDAR sensor consists of one pixel;

obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said action time;

obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said action time;

project said plurality of before images onto a plane in said item storage area to generate a plurality of projected before images;

project said plurality of after images onto said plane to generate a plurality of projected after images;

analyze

said plurality of projected before images, and

said plurality of projected after images, to

identify said at least one item added to or removed from said affected zone at said action time; and

,

associate said at least one item and said item quantity change with said shopper.

21. A smart shelf system that integrates images and quantity sensors, comprising:

a plurality of quantity sensors, each corresponding to a storage zone of a plurality of storage zones of an item storage area, wherein

each quantity sensor of said plurality of quantity sensors is configured to generate a quantity signal that is correlated with a quantity of items contained in the storage zone corresponding to said each quantity sensor;

a processor coupled to

said plurality of quantity sensors, and to

a plurality of cameras oriented to view said item storage area;

wherein said processor is configured to

analyze said quantity signal from said plurality of quantity sensors to

identify an affected zone of said plurality of storage zones within which a shopper added or removed at least one item;

determine an action time at which said shopper added or removed said at least one item; and

determine an item quantity change in said affected zone;

wherein

 

each storage zone of said plurality of storage zones comprises a hanging mount configured to hold items suspended from said hanging mount;

 

said quantity signal associated with said each storage zone varies with a weight of said items suspended from said hanging mount; and

,

 

said each quantity sensor associated with said each storage zone comprises two or more strain gauges;

obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said action time;

obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said action time;

project said plurality of before images onto a plane in said item storage area to generate a plurality of projected before images;

project said plurality of after images onto said plane to generate a plurality of projected after images;

analyze

said plurality of projected before images, and

said plurality of projected after images, to

identify said at least one item added to or removed from said affected zone at said action time; and

,

associate said at least one item and said item quantity change with said shopper.