Systems and methods for deep learning-based notifications

• US 10,474,993 B2
• Filed: 01/24/2019
• Issued: 11/12/2019
• Est. Priority Date: 08/07/2017
• Status: Active Grant

• Current Assignee: Standard Cognition Corp.
• Original Assignee: Standard Cognition Corp.
• Inventors: Fisher, Jordan E., Fischetti, Daniel L., Ogle, Brandon L., Novak, John F., Dorman, Kyle E., Kihara, Kenneth S., Lasheras, Juan C.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US16/256,361
• Publication Number:
  US 20190156275A1
• Time in Patent Office: 292 Days
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Systems and techniques are provided for tracking puts and takes of inventory items by subjects in an area of real space. A plurality of cameras with overlapping fields of view produce respective sequences of images of corresponding fields of view in the real space. In one embodiment, the system includes first image processors, including subject image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The first image processors process images to identify subjects represented in the images in the corresponding sequences of images. The system includes second image processors, including background image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images. Following this, the second image processors process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

Claims

1. A system for tracking inventory events, in an area of real space, comprising:

a processing system receiving a plurality of sequences of images of respective fields of view in the real space, the processing system including;

logic to process the sequences of images to identify inventory events in the area of real space, and create data structures for the inventory events including a subject identifier identifying the subject performing the inventory event in the area of real space, an item identifier, a classification confidence score for the item, a location represented by positions in three dimensions of the area of real space and a timestamp; and

logic to track identified subjects in the area of real space and to display images representing the identified subjects, and in response to comparison of a classification confidence score for the item identified in the data structure created for a given inventory event with a threshold to change a color of the display image of the subject identified in the data structure.

2. The system of claim 1, wherein the logic to track identified subjects is performed without use of personal identifying biometric information.

3. The system of claim 1, wherein the fields of view of images in the sequences of images overlap with the field of view of at least one other image in the sequences of images.

4. The system of claim 1, including first image recognition engines receiving the sequences of images, which process images to generate first data sets that identify subjects and locations of the identified subjects in the real space.

5. The system of claim 4, wherein the first image recognition engines comprise convolutional neural networks.

6. The system of claim 4, including logic to process the first data sets to specify bounding boxes which include images of hands of the identified subjects in images in the sequences of images.

7. The system of claim 6, including second image recognition engines, receiving the sequences of images, which process the specified bounding boxes in the images to detect the inventory events and to generate the item identifiers for the detected inventory events.

8. The system of claim 1, including logic to determine a movement of the identified subject in the area of real space towards an exit from the area of real space and generate notifications when the identified subject is within a predetermined distance to the exit.

9. The system of claim 8, the processing system including logic that generates notifications in response to a payment status for items identified in inventory events for the identified subject.

10. A method for tracking inventory events in an area of real space, the method including:

identifying inventory events in the area of real space using sequences of images of corresponding fields of view in the real space, and creating data structures for the inventory events including a subject identifier identifying the subject performing the inventory event in the area of real space, an item identifier, a classification confidence score for the item, a location represented by positions in three dimensions of the area of real space and a timestamp; and

tracking identified subjects in the area of real space, and changing a color of display image representing the identified subject in a given inventory event in response to comparisons of classification confidence for the item identified in the data structure created for the given inventory event with a threshold.

11. The method of claim 10, wherein the

tracking identified subjects is performed without the use of personal identifying biometric information.

12. A non-transitory computer readable storage medium impressed with computer program instructions to track inventory events in an area of real space, the instructions when executed on a processor, implement a method comprising:

identifying inventory events in the area of real space using sequences of images of corresponding fields of view in the real space, and creating data structures for the inventory events including a subject identifier identifying the subject performing the inventory event in the area of real space, an item identifier, a classification confidence score for the item, a location represented by positions in three dimensions of the area of real space and a timestamp; and

tracking identified subjects in the area of real space, and changing a color of display image representing the identified subject in a given inventory event in response to comparisons of classification confidence for the item identified in the data structure created for the given inventory event with a threshold.

13. The non-transitory computer readable storage medium of claim 12, wherein the

tracking identified subjects is performed without the use of personal identifying biometric information.

14. The non-transitory computer readable storage medium of claim 12, the method further comprising:

receiving the sequences of images of corresponding fields of view in the real space, and processing the images using image recognition engines to identify subjects and locations of the identified subjects in the real space.