Smart shelf system that integrates images and quantity sensors
First Claim
1. A smart shelf system that integrates images and quantity sensors, comprising:
- a plurality of quantity sensors, each corresponding to a storage zone of a plurality of storage zones of an item storage area, whereineach quantity sensor of said plurality of quantity sensors is configured to generate a quantity signal that is correlated with a quantity of items contained in the storage zone corresponding to said each quantity sensor;
a processor coupled tosaid plurality of quantity sensors, and toa plurality of cameras oriented to view said item storage area;
wherein said processor is configured toanalyze said quantity signal from said plurality of quantity sensors toidentify an affected zone of said plurality of storage zones within which a shopper added or removed at least one item;
determine an action time at which said shopper added or removed said at least one item; and
determine an item quantity change in said affected zone;
obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said action time;
obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said action time;
project said plurality of before images onto a plane in said item storage area to generate a plurality of projected before images;
project said plurality of after images onto said plane to generate a plurality of projected after images;
analyzesaid plurality of projected before images, andsaid plurality of projected after images, toidentify said at least one item added to or removed from said affected zone at said action time;
associate said at least one item and said item quantity change with said shopper;
obtain a 3D model of a store that contains said item storage area;
receive a time sequence of images from each camera of a second plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period;
analyze said time sequence of images and said 3D model of said store todetermine a sequence of locations of a person in said store during said time period; and
calculate a field of influence volume around each location of said sequence of locations; and
,when said field of influence volume intersects said item storage area,identify said shopper as said person; and
,associate said at least one item and said item quantity change with said person.
1 Assignment
0 Petitions
Accused Products
Abstract
A system that integrates camera images and quantity sensors to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed may then be attributed to a shopper near the area. Shelves may be divided into storage zones, such as bins or lanes, and a quantity sensor may measure the item quantity in each zone. Quantity changes indicate that a shopper has taken or placed items in the zone. Distance sensors, such as LIDAR, may be used for shelves that push items towards the front. Strain gauges may be used for bins or hanging rods. Quantity changes may trigger analysis of camera images of the shelf to identify the items taken or replaced. Images from multiple cameras that view a shelf may be projected to a vertical plane at the front of the shelf to simplify analysis.
49 Citations
21 Claims
-
1. A smart shelf system that integrates images and quantity sensors, comprising:
-
a plurality of quantity sensors, each corresponding to a storage zone of a plurality of storage zones of an item storage area, wherein each quantity sensor of said plurality of quantity sensors is configured to generate a quantity signal that is correlated with a quantity of items contained in the storage zone corresponding to said each quantity sensor; a processor coupled to said plurality of quantity sensors, and to a plurality of cameras oriented to view said item storage area; wherein said processor is configured to analyze said quantity signal from said plurality of quantity sensors to identify an affected zone of said plurality of storage zones within which a shopper added or removed at least one item; determine an action time at which said shopper added or removed said at least one item; and determine an item quantity change in said affected zone; obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said action time; obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said action time; project said plurality of before images onto a plane in said item storage area to generate a plurality of projected before images; project said plurality of after images onto said plane to generate a plurality of projected after images; analyze said plurality of projected before images, and said plurality of projected after images, to identify said at least one item added to or removed from said affected zone at said action time; associate said at least one item and said item quantity change with said shopper; obtain a 3D model of a store that contains said item storage area; receive a time sequence of images from each camera of a second plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period; analyze said time sequence of images and said 3D model of said store to determine a sequence of locations of a person in said store during said time period; and calculate a field of influence volume around each location of said sequence of locations; and
,when said field of influence volume intersects said item storage area, identify said shopper as said person; and
,associate said at least one item and said item quantity change with said person. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
-
-
20. A smart shelf system that integrates images and quantity sensors, comprising:
-
a plurality of quantity sensors, each corresponding to a storage zone of a plurality of storage zones of an item storage area, wherein each quantity sensor of said plurality of quantity sensors is configured to generate a quantity signal that is correlated with a quantity of items contained in the storage zone corresponding to said each quantity sensor; a processor coupled to said plurality of quantity sensors, and to a plurality of cameras oriented to view said item storage area; wherein said processor is configured to analyze said quantity signal from said plurality of quantity sensors to identify an affected zone of said plurality of storage zones within which a shopper added or removed at least one item; determine an action time at which said shopper added or removed said at least one item; and determine an item quantity change in said affected zone; wherein
each storage zone of said plurality of storage zones comprises a bin comprising a moveable back, wherein
when said shopper removes an item from said bin, said moveable back moves towards a front end of said bin; and
when said shopper adds an item to said bin, said moveable back moves away from said front end of said bin;
said quantity signal associated with said each storage zone varies with a position of said moveable back;
said each quantity sensor associated with said each storage zone comprises a distance sensor configured to measure a distance to said moveable back;
said distance sensor comprises a LIDAR sensor; and
,
a resolution of said LIDAR sensor consists of one pixel;obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said action time; obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said action time; project said plurality of before images onto a plane in said item storage area to generate a plurality of projected before images; project said plurality of after images onto said plane to generate a plurality of projected after images; analyze said plurality of projected before images, and said plurality of projected after images, to identify said at least one item added to or removed from said affected zone at said action time; and
,associate said at least one item and said item quantity change with said shopper.
-
-
21. A smart shelf system that integrates images and quantity sensors, comprising:
-
a plurality of quantity sensors, each corresponding to a storage zone of a plurality of storage zones of an item storage area, wherein each quantity sensor of said plurality of quantity sensors is configured to generate a quantity signal that is correlated with a quantity of items contained in the storage zone corresponding to said each quantity sensor; a processor coupled to said plurality of quantity sensors, and to a plurality of cameras oriented to view said item storage area; wherein said processor is configured to analyze said quantity signal from said plurality of quantity sensors to identify an affected zone of said plurality of storage zones within which a shopper added or removed at least one item; determine an action time at which said shopper added or removed said at least one item; and determine an item quantity change in said affected zone; wherein
each storage zone of said plurality of storage zones comprises a hanging mount configured to hold items suspended from said hanging mount;
said quantity signal associated with said each storage zone varies with a weight of said items suspended from said hanging mount; and
,
said each quantity sensor associated with said each storage zone comprises two or more strain gauges;obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said action time; obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said action time; project said plurality of before images onto a plane in said item storage area to generate a plurality of projected before images; project said plurality of after images onto said plane to generate a plurality of projected after images; analyze said plurality of projected before images, and said plurality of projected after images, to identify said at least one item added to or removed from said affected zone at said action time; and
,associate said at least one item and said item quantity change with said shopper.
-
Specification