Machine learning-based subject tracking

• US 10,474,992 B2
• Filed: 01/24/2019
• Issued: 11/12/2019
• Est. Priority Date: 08/07/2017
• Status: Active Grant

• Current Assignee: Standard Cognition Corp.
• Original Assignee: Standard Cognition Corp.
• Inventors: Fisher, Jordan E., Fischetti, Daniel L., Ogle, Brandon L., Novak, John F., Dorman, Kyle E., Kihara, Kenneth S., Lasheras, Juan C.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US16/256,358
• Publication Number:
  US 20190156274A1
• Time in Patent Office: 292 Days
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Systems and techniques are provided for tracking puts and takes of inventory items by subjects in an area of real space. A plurality of cameras with overlapping fields of view produce respective sequences of images of corresponding fields of view in the real space. In one embodiment, the system includes first image processors, including subject image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The first image processors process images to identify subjects represented in the images in the corresponding sequences of images. The system includes second image processors, including background image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images. Following this, the second image processors process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

Claims

1. A system for tracking subjects, in an area of real space, comprising:

a processing system receiving a plurality of synchronized sequences of images of respective fields of view in the real space at image capturing cycles, the processing including;

logic to process images in a first set of sequences including more than one member in the plurality of synchronized sequences at a first image capturing cycle at a first time to identify locations of a plurality of tracked subjects by identifying features of the tracked subjects, the features having first locations represented by positions in three dimensions of the area of real space;

logic to process images in a second set of sequences including more than one member in the plurality of synchronized sequences at a second image capturing cycle at a second time to identify candidate features for matching with features of the plurality of tracked subjects, the candidate features having second locations represented by positions in three dimensions of the area of real space; and

logic to track movement of the plurality of tracked subjects by matching the features of particular subjects from the first image capturing cycle with the candidate features in the second image capturing cycle and to update the identified locations of subjects in the plurality of tracked subjects with locations of the candidate features, wherein the features comprise joints of a plurality of different types, and subjects in the plurality of subjects are represented by constellations of said joints, and including logic to update the constellations of joints representing tracked subjects using the candidate features.

2. The system of claim 1, wherein the first set of sequences includes at least a first sequence and a second sequence in the plurality of synchronized sequences and the second set of sequences includes at least the second sequence and a third sequence not in the first set in the plurality of synchronized sequences.

3. The system of claim 2, wherein the field of view of the images of the first sequence does not overlap with the field of view of the images of the third sequence.

4. The system of claim 1, wherein the first set of sequences includes at least a first sequence and a second sequence in the plurality of synchronized sequences and the second set of sequences includes at least a third sequence and a fourth sequence in the plurality of synchronized sequences not in the first set.

5. The system of claim 4, wherein the images in the first sequence and the second sequence have overlapping fields of view and the images in the third sequence and the fourth sequence have overlapping fields of view.

6. The system of claim 4, wherein the fields of view of the images in the first set do not overlap with the fields of view of the images in the second set.

7. The system of claim 1, wherein the logic that tracks movement of the plurality of tracked subjects includes logic that determines distances between the features of particular tracked subjects from the first image capturing cycle and the candidate features in the second image capturing cycle.

8. The system of claim 1, wherein the logic to process images in the first set of sequences includes image recognition engines configured to receive sequences of images to identify the features of the tracked subjects.

9. The system of claim 8, wherein the image recognition engines comprise convolutional neural networks.

21. The system of claim 1 further including logic to identify new subjects using the candidate features not matched to particular subjects.

24. The system of claim 1, wherein the logic to update the constellations comprises entity cohesion logic.

27. The system of claim 1, wherein said candidate features comprise foot joints.

10. A method for tracking subjects, in an area of real space, the method including:

identifying locations of a plurality of tracked subjects by identifying features of the tracked subjects, the features having first locations represented by positions in three dimensions of the area of real space using data in a first set of sequences of images of respective fields of view including more than one member at a first image capturing cycle;

identifying candidate features for matching with features of the plurality of tracked subjects, the candidate features having second locations represented by positions in three dimensions of the area of real space using data in a second set of sequences of images of respective fields of view including more than one member at a second image capturing cycle;

tracking movement of the plurality of tracked subjects by matching the features of particular subjects from the first image capturing cycle with the candidate features in the second image capturing cycle; and

updating the identified locations of subjects in the plurality of tracked subjects with locations of the candidate features, wherein the features comprise joints of a plurality of different types, and subjects in the plurality of subjects are represented by constellations of said joints, and including logic to update the constellations of joints representing tracked subjects using the candidate features.

11. The method of claim 10, wherein the first set of sequences of images includes at least a first sequence and a second sequence and the second set of sequences of images includes at least the second sequence and a third sequence not in the first set.

12. The method of claim 11, wherein the field of view of the first sequence in the first set of sequences of images does not overlap with the field of view of the third sequence in the second set of sequences of images.

13. The method of claim 10, wherein the first set of sequences of includes at least a first sequence and a second sequence and the second set of sequences of images includes at least a third sequence and a fourth sequence not in the first set.

14. The method of claim 13, wherein the first sequence and the second sequence in the first set of sequences of images have overlapping fields of view and the third sequence and the fourth sequence in the second set of sequences of images have overlapping fields of view.

15. The method of claim 13, wherein the fields of view of images in the first set of sequences of images do not overlap with the fields of view of sequences in the second set of sequences of images.

22. The method of claim 10, further including identifying new subjects using the candidate features not matched to particular subjects.

25. The method of claim 10, wherein said updating the constellations comprises using entity cohesion logic.

28. The method of claim 10, wherein said candidate features comprise foot joints.

16. A non-transitory computer readable storage medium impressed with computer program instructions to track subjects, in an area of real space, the instructions, when executed on a processor, implement a method comprising:

identifying locations of a plurality of tracked subjects by identifying features of the tracked subjects, the features having first locations represented by positions in three dimensions of the area of real space using images in a first set of sequences of images of respective fields of view including more than one member at a first data capturing cycle;

identifying candidate features for matching with features of the plurality of tracked subjects, the candidate features having second locations represented by positions in three dimensions of the area of real space using images in a second set of sequences of images of respective fields of view including more than one member at a second data capturing cycle;

tracking movement of the plurality of tracked subjects by matching the features of particular subjects from the first data capturing cycle with the candidate features in the second data capturing cycle; and

updating the identified locations of subjects in the plurality of tracked subjects with locations of the candidate features, wherein the features comprise joints of a plurality of different types, and subjects in the plurality of subjects are represented by constellations of said joints, and including updating the constellations of joints representing tracked subjects using the candidate features.

17. The non-transitory computer readable storage medium of claim 16, wherein the first set of sequences of images includes at least a first sequence and a second sequence, and the second set of sequences of images includes at least a third sequence and a fourth sequence not in the first set.

18. The non-transitory computer readable storage medium of claim 17, wherein the fields of view of sequences in the first set of sequences of images do not overlap with the fields of view of sequences in the second set of sequences of images.

19. The non-transitory computer readable storage medium of claim 16, wherein the tracking movement of the plurality of tracked subjects by matching the features of particular subjects from the first data capturing cycle with the candidate features in the second data capturing cycle, includes determining distances between the features of particular tracked subjects feature from the first data capturing cycle and the candidate features in the second data capturing cycle.

20. The non-transitory computer readable storage medium of claim 16, wherein the using images in the first set of sequences of images includes using image recognition engines to identify the features of the tracked subjects.

23. The non-transitory computer readable storage medium of claim 16, the implemented method further including identifying new subjects using the candidate features not matched to particular subjects.

26. The non-transitory computer readable storage medium of claim 16, the implemented method further including updating the constellations using entity cohesion logic.

29. The non-transitory computer readable storage medium of claim 16, wherein said candidate features comprise foot joints.

30. A system for tracking subjects, in an area of real space, comprising:

a processing system receiving a plurality of synchronized sequences of images of respective fields of view in the real space at image capturing cycles, the processing including;

logic to process images in a first set of sequences including more than one member in the plurality of synchronized sequences at a first image capturing cycle to generate an array of joint data structures, the joint data structures in the array correspond to respective tracked subjects in the area of real space, the joint data structures map first locations of a plurality of joints of the respective subjects to positions in three dimensions of the area of real space and a timestamp;

logic to process images in a second set of sequences including more than one member in the plurality of synchronized sequences at a second image capturing cycle to identify candidate joints having second locations represented by positions in three dimensions of the area of real space and a timestamp; and

logic to track movement of the tracked subjects by performing entity cohesion to match the candidate joints with joints in the joint data structures in the array of joint data structures, and update the joint data structures matched to the candidate joints with locations of the candidate joints.