Predicting inventory events using foreground/background processing

• US 10,474,988 B2
• Filed: 04/04/2018
• Issued: 11/12/2019
• Est. Priority Date: 08/07/2017
• Status: Active Grant

• Current Assignee: Standard Cognition Corp.
• Original Assignee: Standard Cognition Corp.
• Inventors: Fisher, Jordan E., Fischetti, Daniel L., Ogle, Brandon L., Novak, John F., Dorman, Kyle E., Kihara, Kenneth S., Lasheras, Juan C.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US15/945,473
• Publication Number:
  US 20190043003A1
• Time in Patent Office: 587 Days
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Systems and techniques are provided for tracking puts and takes of inventory items by subjects in an area of real space. A plurality of cameras with overlapping fields of view produce respective sequences of images of corresponding fields of view in the real space. In one embodiment, the system includes first image processors, including subject image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The first image processors process images to identify subjects represented in the images in the corresponding sequences of images. The system includes second image processors, including background image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images. Following this, the second image processors process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

Claims

1. A system for tracking changes in an area of real space, comprising:

a plurality of cameras, cameras in the plurality of cameras producing respective sequences of images of corresponding fields of view in the real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;

a processing system coupled to the plurality of cameras, the processing system including;

first image processors, including subject image recognition engines, receiving corresponding sequences of images from the plurality of cameras, which process images to identify subjects represented in the images in the corresponding sequences of images;

second image processors, including background image recognition engines, receiving corresponding sequences of images from the plurality of cameras, which mask the identified subjects to generate masked images, process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images; and

third image processors, including foreground image recognition engines, receiving corresponding sequences of images from the plurality of cameras, which process images to identify and classify foreground changes represented in the images in the corresponding sequences of images.

2. The system of claim 1, wherein the foreground image recognition engines and the background image recognition engines comprise convolutional neural networks.

3. The system of claim 1, including logic to associate identified background changes and identified foreground changes with identified subjects.

4. The system of claim 1, wherein the second image processors include:

a background image store to store background images for corresponding sequences of images;

mask logic to process images in the sequences of images to replace foreground image data representing the identified subjects with background image data from the background images for the corresponding sequences of images to provide the masked images.

5. The system of claim 4, wherein the mask logic combines sets of N masked images in the sequences of images to generate sequences of factored images for each camera, and the second image processors identify and classify background changes by processing the sequence of factored images.

6. The system of claim 1, wherein the second image processors include

logic to produce change data structures for the corresponding sequences of images, the change data structures including coordinates in the masked images of identified background changes, identifiers of an inventory item subject of the identified background changes and classifications of the identified background changes;

and

coordination logic to process change data structures from sets of cameras having overlapping fields of view to locate the identified background changes in real space.

7. The system of claim 6, wherein the classifications of identified background changes in the change data structures indicate whether the identified inventory item has been added or removed relative to the background image.

8. The system of claim 6, wherein the classifications of identified background changes in the change data structures indicate whether the identified inventory item has been added or removed relative to the background image, and including logic to associate background changes with identified subjects, and to make detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects.

9. The system of claim 1, including:

logic to associate background changes and identified foreground changes with identified subjects, and to make detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects.

10. The system of claim 1, wherein the first image processors identify locations of hands of identified subjects;

and including;

logic to associate background changes with identified subjects by comparing the locations of the changes with the locations of hands of identified subjects, and to make detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects.

11. The system of claim 1, including

logic to associate background changes with identified subjects, and to make a first set of detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects;

logic to associate foreground changes with identified subjects, and to make a second set of detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects; and

selection logic to process the first and second sets of detections to generate log data structures including lists of inventory items for identified subjects.

12. The system of claim 1, wherein the sequences of images from cameras in the plurality of cameras are synchronized.

13. A method for tracking puts and takes of inventory items by subjects in an area of real space, comprising:

producing a plurality of sequences of images of inventory display structures in corresponding fields of view in the real space, the field of view of each sequence of images overlapping with the field of view of at least one other sequence of images in the plurality of sequences;

using first image processors, including subject image recognition engines, to process images to identify subjects represented in the images in the corresponding sequences of images;

detecting puts and takes of inventory items by identifying gestures of subjects and inventory items associated with the gestures by processing foreground data in the sequences of images, and by identifying semantically significant changes in inventory items on inventory display structures by processing background data in the sequences of images to construct sequences of background images and detect changes in the sequences of background images;

wherein said identifying semantically significant changes in inventory items includes using second image processors, including background image recognition engines, to mask identified subjects in images in the sequences of images, to generate masked images, to process the masked images to identify and to classify background changes represented in the images in the corresponding sequences of images; and

wherein said identifying gestures of subjects and inventory items associated with the gestures includes using third image processors, including foreground image recognition engines, receiving corresponding sequences of images from the plurality of cameras, to process images to identify and classify foreground changes represented in the images in the corresponding sequences of images.

14. The method of claim 13, wherein the background image recognition engines and the foreground image recognition engines comprise convolutional neural networks.

15. The method of claim 13, including associating identified background changes and foreground changes with identified subjects.

16. The method of claim 13, wherein using the second image processors includes storing background images for corresponding sequences of images;

processing images in the sequences of images to replace foreground image data representing the identified subjects with background image data from the background images for the corresponding sequences of images to provide the masked images.

17. The method of claim 16, wherein processing images in the sequences of images includes combining sets of N masked images in the sequences of images to generate sequences of factored images for each sequence of images, and the second image processors identify and classify background changes by processing the sequence of factored images.

18. The method of claim 16, wherein using the second image processors includes

producing change data structures for the corresponding sequences of images, the change data structures including coordinates in the masked images of identified background changes, identifiers of an inventory item subject of the identified background changes and classifications of the identified background changes;

and

processing change data structures from sets of sequences of images having overlapping fields of view to locate the identified background changes in real space.

19. The method of claim 18, wherein the classifications of identified background changes in the change data structures indicate whether the identified inventory item has been added or removed relative to the background image.

20. The method of claim 18, wherein the classifications of identified background changes in the change data structures indicate whether the identified inventory item has been added or removed relative to the background image, and including associating background changes with identified subjects, and making detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects.

21. The method of claim 13, wherein using the first image processors includes identifying locations of hands of identified subjects;

and including

associating background changes with identified subjects by comparing the locations of the changes with the locations of hands of identified subjects, and making detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects.

22. The method of claim 13, including

associating background changes with identified subjects, and making a first set of detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects;

associating foreground changes with identified subjects, and making a second set of detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects; and

processing the first and second sets of detections to generate log data structures including lists of inventory items for identified subjects.

23. The method of claim 13, including producing the plurality of sequences of images using synchronized cameras.

24. A computer program product, comprising:

a computer readable memory comprising a non-transitory data storage medium;

computer instructions stored in the memory executable by a computer to execute a process including;

using sequences of images of corresponding fields of view in the real space from a plurality of cameras, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;

using first image processors, including subject image recognition engines, to process images to identify subjects represented in the images in the corresponding sequences of images;

using second image processors, including background image recognition engines, to mask identified subjects in images in the sequences of images, to generate masked images, to process the masked images to identify and to classify background changes represented in the images in the corresponding sequences of images; and

using third image processors, including foreground image recognition engines, receiving corresponding sequences of images from the plurality of cameras, to process images to identify and classify foreground changes represented in the images in the corresponding sequences of images.

25. The computer program product of claim 24, wherein using the first image processors includes identifying locations of hands of identified subjects;

and including

associating background changes with identified subjects by comparing the locations of the changes with the locations of hands of identified subjects, and making detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects.

26. The computer program product of claim 24, including

associating background changes with identified subjects, and making a first set of detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects;

associating foreground changes with identified subjects, and making a second set of detections of takes of inventory items by the identified subjects and of puts of inventory items on inventory display structures by the identified subjects; and

processing the first and second sets of detections to generate log data structures including lists of inventory items for identified subjects.