Projected image item tracking system

• US 10,535,146 B1
• Filed: 05/06/2019
• Issued: 01/14/2020
• Est. Priority Date: 07/16/2018
• Status: Active Grant

• Current Assignee: Accel Robotics Corporation
• Original Assignee: Accel Robotics Corporation
• Inventors: Buibas, Marius, Quinn, John, Feigum, Kaylee, Petre, Csaba, Maseda, Michael Brandon, Cseh, Martin Alan
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US16/404,667
• Publication Number:
  US 20200020112A1
• Time in Patent Office: 253 Days
• CPC Class Codes: G01G 19/4144 for controlling weight of goods in commercial establishments, e.g. supermarket, P.O.S. systems
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 20/208 Input by product or record sensing, e.g. weighing or scanner processing
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/21 Collision detection, intersection
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G07F 9/026 for alarm, monitoring and auditing in vending machines or means for indication, e.g. when empty
• CPC Class Codes: G07G 1/0036 Checkout procedures
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A projected image item tracking system that analyzes projected camera images to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed on them may then be attributed to a shopper near the area. Projected images may be combined to generate a 3D volume difference between the state of the area before and after shopper interaction. The volume difference may be calculated using plane-sweep stereo, or using convolutional neural networks. Because these methods may be computationally intensive, the system may first localize a change volume where items appear to have been displaced, and then generate a volume difference only within that change volume. This optimization results in significant savings in power consumption and in more rapid identification of items. The 3D volume difference may also indicate the quantity of items displaced, for example from a vertical stack.

Claims

1. A projected image item tracking system comprising:

a processor coupled to

a sensor configured to generate

an enter signal when a shopper reaches into or towards an item storage area in a store, wherein said item storage area contains items; and

,

an exit signal when said shopper retracts from said item storage area; and

,

a plurality of cameras oriented to view said item storage area;

wherein said processor is configured to

obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said enter signal;

obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to said camera of said plurality of cameras, wherein said each after image is captured at a time after said exit signal;

project said plurality of before images onto two or more planes in said item storage area to generate a plurality of projected before images corresponding to each combination of a plane of said two or more planes and said camera of said plurality of cameras;

project said plurality of after images onto said two or more planes to generate a plurality of projected after images corresponding to said each combination of said plane of said two or more planes and said camera of said plurality of cameras;

analyze

said plurality of projected before images, and

said plurality of projected after images, to

identify an item of said items taken from or put into said item storage area between said enter signal and said exit signal; and

,

associate said item with said shopper; and

,

wherein said analyze said plurality of projected before images and said plurality of projected after images comprises

calculate a 3D volume difference between

contents of said item storage area at said time before said enter signal and

contents of said item storage area at said time after said exit signal;

when said 3D volume difference indicates that said contents of said item storage area at said time after said exit signal is smaller than said contents of said item storage area at said time before said enter signal,

input at least a portion of one or more of said plurality of projected before images that intersects said 3D volume difference into a classifier;

when said 3D volume difference indicates that said contents of said item storage area at said time after said exit signal is larger than said contents of said item storage area at said time before said enter signal,

input at least a portion of one or more of said plurality of projected after images that intersects said 3D volume difference into said classifier; and

,

identify said item of said items taken from or put into said item storage area as an output of said classifier.

2. The system of claim 1, wherein said classifier comprises a neural network trained to recognize images of said items.

3. The system of claim 1, wherein said processor is further configured to

calculate a quantity of said item of said items taken from or put into said item storage area based on said 3D volume difference;

and,

associate said quantity with said shopper and with said item.

4. The system of claim 3, wherein said calculate said quantity of said item comprises

obtain a size of said item;

and,

compare said 3D volume difference with said size of said item to determine said quantity of said item.

5. The system of claim 1, wherein said processor is further configured to:

when said 3D volume difference indicates that said contents of said item storage area at said time after said exit signal is smaller than said contents of said item storage area at said time before said enter signal,

associate a take action with said shopper and with said item; and

,

when said 3D volume difference indicates that said contents of said item storage area at said time after said exit signal is larger than said contents of said item storage area at said time before said enter signal,

associate a put action with said shopper and with said item.

6. The system of claim 1, wherein said calculate said 3D volume difference comprises

calculate a before 3D surface of said contents of said item storage area from said plurality of projected before images;

calculate an after 3D surface of said contents of said item storage area from said plurality of projected after images; and

,

calculate said 3D volume difference as a volume between said before 3D surface and said after 3D surface.

7. The system of claim 6, wherein

said calculate said before 3D surface comprises apply a plane sweep stereo algorithm to said plurality of projected before images;

and,

said calculate said after 3D surface comprises apply said plane sweep stereo algorithm to said plurality of projected after images.

8. The system of claim 7, wherein said two or more planes are parallel to a surface of said item storage area.

9. The system of claim 7, wherein at least one plane of said two or more planes is not parallel to a surface of said item storage area.

10. The system of claim 6, wherein said processor is further configured to

compare said plurality of projected before images with said plurality of projected after images to calculate a change region of each plane of said two or more planes;

combine said change region of each plane of said two or more planes to calculate a change volume in said item storage area;

calculate said before 3D surface and said after 3D surface only in said change volume.

11. The system of claim 10, wherein said compare said plurality of projected before images to said plurality of projected after images comprises

calculate an image difference between

each projected before image of said plurality of projected before images, and

a corresponding projected after image of said plurality of projected after images that is associated

with a same camera of said plurality of cameras and

with a same plane of said two or more planes;

and,

for each plane of said two or more planes, combine the image difference associated with each camera of said plurality of cameras and associated with said each plane to form said change region of said each plane.

12. The system of claim 10, wherein said combine the image difference associated with each camera of said plurality of cameras and associated with said each plane comprises

weight each pixel in said image difference based on a distance between a point in said each plane corresponding to said pixel and a location of said each camera, to form a weighted image difference;

calculate an average weighted image difference for said each plane as an average of said weighted image difference corresponding to said each camera; and

,

calculate said change region of said each plane based on said average weighted image difference.

13. The system of claim 10, wherein said calculate said image difference comprises

calculate an absolute value of a difference between a pixel value of said each projected before image and a corresponding pixel value of said corresponding projected after image.

14. The system of claim 10, wherein said calculate said image difference comprises

input said each projected before image and said corresponding projected after image into a neural network trained to identify image differences between pairs of images.

15. The system of claim 1, further comprising

a modular shelf comprising

said plurality of cameras oriented to view said item storage area;

a right-facing camera mounted on or proximal to a left edge of said modular shelf;

a left-facing camera mounted on or proximal to a right edge of said modular shelf;

a shelf processor; and

,

a network switch;

wherein said processor comprises a network of computing devices, said computing devices comprising

a store processor; and

,

said shelf processor.

16. The system of claim 15, wherein

said sensor comprises said right-facing camera, said left-facing camera;

and, said processor is further configured to

analyze images from said right-facing camera and said left-facing camera to detect when said shopper reaches into or towards said item storage area, and to generate said enter signal; and

,

analyze images from said right-facing camera and said left-facing camera to detect when said shopper retracts from said item storage area, and to generate said exit signal.

17. The system of claim 15, wherein

said shelf processor comprises or is coupled to a memory;

and,

said shelf processor is configured to

receive images from said plurality of cameras and store said images in said memory;

when said shelf processor receives or generates said enter signal, retrieve said plurality of before images from said memory.

18. The system of claim 17, wherein said shelf processor is further configured to

when said shelf processor receives or generates said enter signal, transmit said plurality of before images from said memory to said store processor;

and,

when said shelf processor receives or generates said exit signal, receive said plurality of after images from said plurality of cameras or obtain said plurality of after images from said memory and transmit said plurality of after images to said store processor.

19. The system of claim 1, wherein said analyze said plurality of projected before images and said plurality of projected after images further comprises

input at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images into a neural network trained to output said item of said items taken from or put into said item storage area between said enter signal and said exit signal.

20. The system of claim 19, wherein said neural network is further trained to output an action that indicates whether said item of said items is taken from or is put into said item storage area between said enter signal and said exit signal.

21. The system of claim 20, wherein said neural network comprises

a feature extraction layer applied to each of said at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images, wherein said feature extraction layer outputs image features;

a differencing layer applied to each pair of said image features associated with a portion of a projected before image from said camera and said image features associated with a portion of a projected after image from said camera, wherein said differencing layer outputs feature differences associated with said camera;

one or more convolutional layers applied to said feature differences associated with each camera of said plurality of cameras;

an item classifier layer applied to an output of said one or more convolutional layers; and

,

an action classifier layer applied to said output of said one or more convolutional layers.

22. A projected image item tracking system comprising:

a processor coupled to

a sensor configured to generate

an enter signal when a shopper reaches into or towards an item storage area in a store, wherein said item storage area contains items; and

,

an exit signal when said shopper retracts from said item storage area; and

,

a plurality of cameras oriented to view said item storage area;

wherein said processor is configured to

obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said enter signal;

obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to said camera of said plurality of cameras, wherein said each after image is captured at a time after said exit signal;

project said plurality of before images onto two or more planes in said item storage area to generate a plurality of projected before images corresponding to each combination of a plane of said two or more planes and said camera of said plurality of cameras;

project said plurality of after images onto said two or more planes to generate a plurality of projected after images corresponding to said each combination of said plane of said two or more planes and said camera of said plurality of cameras;

analyze

said plurality of projected before images, and

said plurality of projected after images, to

identify an item of said items taken from or put into said item storage area between said enter signal and said exit signal; and

,

associate said item with said shopper; and

,

wherein said analyze said plurality of projected before images and said plurality of projected after images comprises

input at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images into a neural network trained to output said item of said items taken from or put into said item storage area between said enter signal and said exit signal,

wherein said neural network is further trained to output an action that indicates whether said item of said items is taken from or is put into said item storage area between said enter signal and said exit signal; and

,

wherein said neural network comprises

a feature extraction layer applied to each of said at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images, wherein said feature extraction layer outputs image features;

a differencing layer applied to each pair of said image features associated with a portion of a projected before image from said camera and said image features associated with a portion of a projected after image from said camera, wherein said differencing layer outputs feature differences associated with said camera;

one or more convolutional layers applied to said feature differences associated with each camera of said plurality of cameras;

an item classifier layer applied to an output of said one or more convolutional layers; and

,

an action classifier layer applied to said output of said one or more convolutional layers.