Realtime inventory tracking using deep learning

• US 10,445,694 B2
• Filed: 01/24/2019
• Issued: 10/15/2019
• Est. Priority Date: 08/07/2017
• Status: Active Grant

• Current Assignee: Standard Cognition Corp.
• Original Assignee: Standard Cognition Corp.
• Inventors: Fisher, Jordan E., Fischetti, Daniel L., Locascio, Nicholas J.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US16/256,904
• Publication Number:
  US 20190156276A1
• Time in Patent Office: 264 Days
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition

Abstract

Systems and techniques are provided for tracking inventory items in an area of real space including inventory display structures. A plurality of cameras are disposed above the inventory display structures. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A memory stores a map of the area of real space identifying inventory locations on inventory display structures. The system is coupled to a plurality of cameras and uses the sequences of images produced by at least two cameras in the plurality of cameras to find a location of an inventory event in three dimensions in the area of real space. The system matches the location of the inventory event with an inventory location.

Claims

1. A system for tracking inventory events, such as puts and takes, in an area of real space, comprising:

a processing system configured to receive a plurality of sequences of images of corresponding fields of view in the real space including inventory display structures, the field of view of each sensor overlapping with the field of view of at least one other sensor in the plurality of sensors, and having access to memory storing a store inventory for the area of real space including a log data structure in memory identifying locations of inventory display locations in the area of real space, the log data structure including item identifiers and their respective quantities for items identified on inventory display locations, the processing system including;

logic that uses the sequences of images produced by at least two sensors in the plurality of sensors to find a location of an inventory event, to identify item associated with the inventory event, and to attribute the inventory event to a customer,

logic that uses the sequences of images produced by the plurality of sensors to detect departure of the customer from the area of real space, and in response to update the store inventory in the memory for items associated with inventory events attributed to the customer, and

logic to match location of the inventory event with an inventory location and update the log data structure using a procedure including calculating a distance from the location of the inventory event to inventory locations on inventory display structures and matching the inventory event with an inventory location based on the calculated distance.

2. The system of claim 1, the processing system including logic that uses the sequences of images to track locations of a plurality of customers in the area of real space, and in which the logic to attribute the inventory event to the customer matches the location of the inventory event to a location of one of the customers in the plurality of customers.

3. The system of claim 1, wherein the inventory event is one of a put and a take of an inventory item.

4. The system of claim 1, wherein the processing system includes logic that updates the log data structure in response to inventory events at locations matching an inventory location in the log data structure.

5. The system of claim 4, wherein the log data structure includes item identifiers and their respective quantities for items identified on inventory display locations.

6. The system of claim 1, wherein logic that uses the sequences of images produced by at least two sensors to find a location of an inventory event, creates a data structure including an item identifier, a put or take indicator, three dimensional coordinates for the inventory event in the area of real space and a timestamp.

7. The system of claim 1, wherein the logic that processes the sequences of images comprises image recognition engines which generate data sets representing elements in the images corresponding to hands, and executes analysis of the data sets from sequences of images from at least two sensors to determine locations of inventory events in three dimensions.

8. The system of claim 7, wherein the image recognition engines comprise convolutional neural networks.

9. The system of claim 1, further including a planogram identifying positions of inventory locations in the area of real space and items positioned on the inventory locations, the processing system including logic to determine misplaced items if the inventory event is matched with an inventory location that does not match the planogram.

10. A method of tracking inventory events, such as puts and takes, in an area of real space, the method including:

using a plurality of sequences of images of corresponding fields of view in the real space, including inventory display structures, the field of view of each sequence of images overlapping with the field of view of at least one other sequence of images in the plurality of sequences of images;

identifying locations of inventory display locations in the area of real space in a log data structure including item identifiers and their respective quantities for items identified on inventory display locations;

finding a location of an inventory event using at least two sequences of images in the plurality of sequences of images;

identifying an item associated with the inventory event;

attributing the inventory event to a customer;

matching location of the inventory event with an inventory location using a procedure including calculating a distance from the location of the inventory event to inventory locations on inventory display structures and matching the inventory event with an inventory location based on the calculated distance; and

detecting departure of the customer from the area of real space using the sequences of images produced by the plurality of cameras and in response updating a store inventory for items associated with inventory events attributed to the customer.

11. The method of claim 10, further including using the sequences of images to track locations of a plurality of customers in the area of real space, and attributing the inventory event to the customer by matching the location of the inventory event to a location of one of the customers in the plurality of customers.

12. The method of claim 10, wherein the inventory event is one of a put and a take of an inventory item.

13. The method of claim 10, further including

updating the log data structure in response to inventory events at a location matching an inventory location in the log data structure.

14. The method of claim 13, wherein the log data structure includes item identifiers and their respective quantities for items identified on inventory display locations.

15. The method of claim 10, wherein the finding a location of an inventory event using at least two sequences of images in the plurality of sequences of images, includes creating a data structure including an item identifier, a put or take indicator, three dimensional coordinates of the inventory event in the area of real space and a time stamp.

16. The method of claim 10, wherein the identifying an item associated with the inventory event includes processing the sequences of images using image recognition engines which generate data sets representing elements in the images corresponding to hands, and executes analysis of the data sets from at least two sequences of images to determine locations of inventory events in three dimensions.

17. The method of claim 16, wherein the image recognition engines comprise convolutional neural networks.

18. The method of claim 10, further including a planogram identifying positions of inventory locations in the area of real space and items positioned on the inventory locations, the method including, determining misplaced items if the inventory event is matched with an inventory location that does not match the planogram.

19. A non-transitory computer readable storage medium impressed with computer program instructions to track inventory events, such as puts and takes, in an area of real space, the instructions, when executed on a processor, implement a method comprising:

using a plurality of sequences of images of corresponding fields of view in the real space, including inventory display structures, the field of view of each sequence of images overlapping with the field of view of at least one other sequence of images in the plurality of sequences of images;

identifying locations of inventory display locations in the area of real space in a log data structure including item identifiers and their respective quantities for items identified on inventory display locations;

finding a location of an inventory event using at least two sequences of images in the plurality of sequences of images;

identifying an item associated with the inventory event;

attributing the inventory event to a customer;

matching location of the inventory event with an inventory location using a procedure including calculating a distance from the location of the inventory event to inventory locations on inventory display structures and matching the inventory event with an inventory location based on the calculated distance; and

detecting departure of the customer from the area of real space using the sequences of images and in response updating a store inventory for items associated with inventory events attributed to the customer.

20. The non-transitory computer readable storage medium of claim 19, implementing the method further comprising, using the sequences of images, to track locations of a plurality of customers in the area of real space, and attributing the inventory event to the customer by matching the location of the inventory event to a location of one of the customers in the plurality of customers.

21. The non-transitory computer readable storage medium of claim 19, wherein the inventory event is one of a put and take of an inventory item.

22. The non-transitory computer readable storage medium of claim 19, the method further comprising

updating the log data structure in response to inventory events at a location matching an inventory location in the log data structure.

23. The non-transitory computer readable storage medium of claim 22, wherein the log data structure includes item identifiers and their respective quantities for items identified on inventory display locations.

24. The non-transitory computer readable storage medium of claim 19, wherein the finding a location of an inventory event using at least two sequences of images in the plurality of sequences of images, includes creating a data structure including an item identifier, a put or take indicator, three dimensional coordinates of the inventory event in the area of real space and a timestamp.

25. The non-transitory computer readable storage medium of claim 19, wherein the identifying an item associated with the inventory event includes processing the sequences of images using image recognition engines which generate data sets representing elements in the images corresponding to hands, and executes analysis of the data sets from at least two sequences of images to determine locations of inventory events in three dimensions.

26. The non-transitory computer readable storage medium of claim 25, wherein the image recognition engines comprise convolutional neural networks.

27. The non-transitory computer readable storage medium of claim 19, the method further comprising, a planogram identifying positions of inventory locations in the area of real space and items positioned on the inventory locations, the method including, determining misplaced items if the inventory event is matched with an inventory location that does not match the planogram.