Subject identification and tracking using image recognition

• US 10,055,853 B1
• Filed: 12/19/2017
• Issued: 08/21/2018
• Est. Priority Date: 08/07/2017
• Status: Active Grant

• Current Assignee: Standard Cognition Corp.
• Original Assignee: Standard Cognition Corp.
• Inventors: Fisher, Jordan, Valdman, David, Novak, John, Ogle, Brandon
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US15/847,796
• Time in Patent Office: 245 Days
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

Systems and techniques are provided for tracking multi joint subjects in real space having a plurality of cameras. The field of view of each camera overlaps with at least one other camera. The plurality of cameras produce respective sequences of images of corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. In one embodiment, the processing system comprises image recognition engines receiving sequence of images from the plurality of cameras and generating corresponding arrays of joint data structures. A tracking engine is configured to receive the arrays of joint data structures and generate candidate joints having coordinates in the real space. The processing system includes the logic to identify sets of candidate joints having coordinates in the real space as multi-joint subjects in the real space.

Claims

1. A system for tracking multi joint subjects in an area of real space, comprising:

a plurality of cameras, cameras in the plurality of cameras producing respective sequences of images of corresponding fields of view in the real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;

a processing system coupled to the plurality of cameras, the processing system including;

image recognition engines, receiving the sequences of images from the plurality of cameras, which process images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;

a tracking engine configured to receive the arrays of joint data structures corresponding to images in sequences of images from cameras having overlapping fields of view, and translate the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in real space; and

logic to identify sets of candidate joints having coordinates in real space as multi-joint subjects in the real space.

2. The system of claim 1, wherein the image recognition engines comprise convolutional neural networks.

3. The system of claim 1, wherein image recognition engines process images to generate confidence arrays for elements of the image, where a confidence array for a particular element of an image includes confidence values for a plurality of joint types for the particular element, and to select a joint type for the joint data structure of the particular element based on the confidence array.

4. The system of claim 1, wherein the logic to identify sets of candidate joints comprises heuristic functions based on physical relationships among joints of subjects in real space to identify sets of candidate joints as multi-joint subjects.

5. The system of claim 4, including logic to store the sets of joints identified as multi-joint subjects, and wherein the logic to identify sets of candidate joints includes logic to determine whether a candidate joint identified in images taken at a particular time corresponds with a member of one of the sets of candidate joints identified as multi-joint subjects in preceding images.

6. The system of claim 1, wherein cameras in the plurality of cameras are configured to generate synchronized sequences of images.

7. The system of claim 1, wherein the plurality of cameras comprise cameras disposed over and having fields of view encompassing respective parts of the area in real space, and the coordinates in real space of members of a set of candidate joints identified as a multi-joint subject identify locations in the area of the multi-joint subject.

8. The system of claim 1, including logic to track locations of a plurality of multi-joint subjects in the area of real space.

9. The system of claim 8, including logic to determine when multi joint subjects in the plurality of multi-joint subjects leave the area of real space.

10. The system of claim 1, including logic to track locations in the area of real space of multiple candidate joints that are members of a set of candidate joints identified as a particular multi joint subject.

11. A method for tracking multi-joint subjects in an area of real space, comprising:

using a plurality of cameras to produce respective sequences of images of corresponding fields of view in the real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;

processing images in the sequences of images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;

translating the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in the real space; and

identifying sets of candidate joints having coordinates in real space as multi-joint subjects in the real space.

12. The method of claim 11, wherein said processing images includes using convolutional neural networks.

13. The method of claim 11, wherein said processing images includes generating confidence arrays for elements of the image, where a confidence array for a particular element of an image includes confidence values for a plurality of joint types for the particular element, and selecting a joint type for the joint data structure of the particular element based on the confidence array.

14. The method of claim 11, wherein identifying sets of candidate joints comprises applying heuristic functions based on physical relationships among joints of subjects in real space to identify sets of candidate joints as multi-joint subjects.

15. The method of claim 14, including storing the sets of joints identified as multi-joint subjects, and wherein the identifying sets of candidate joints includes determining whether a candidate joint identified in images taken at a particular time corresponds with a member of one of the sets of candidate joints identified as a multi-joint subject in a preceding image.

16. The method of claim 11, wherein the sequences of images are synchronized.

17. The method of claim 11, wherein the plurality of cameras comprise cameras disposed over and having fields of view encompassing respective parts of the area in real space, and the coordinates in real space of members of a set of candidate joints identified as a multi-joint subject identify locations in the area of the multi-joint subject.

18. The method of claim 11, including tracking locations of a plurality of multi-joint subjects in the area of real space.

19. The method of claim 18, including determining when a multi-joint subject in the plurality of multi-joint subjects leaves the area of real space.

20. The method of claim 11, including tracking locations in the area of real space of multiple candidate joints that are members of a set of candidate joints identified as a particular multi joint subject.

21. A computer program product, comprising:

a computer readable memory comprising a non-transitory data storage medium;

computer instructions stored in the memory executable by a computer to track multi-joint subjects in an area of real space by a process including;

using sequences of images from a plurality of cameras having corresponding fields of view in real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;

processing images in the sequences of images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;

translating the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in the real space; and

identifying sets of candidate joints having coordinates in real space as multi-joint subjects in the real space.

22. The product of claim 21, wherein said processing images includes using convolutional neural networks.

23. The product of claim 21, wherein said processing image includes generating confidence arrays for elements of the image, where a confidence array for a particular element of an image includes confidence values for a plurality of joint types for the particular element, and selecting a joint type for the joint data structure of the particular element based on the confidence array.

24. The product of claim 21, wherein identifying sets of candidate joints comprises applying heuristic functions based on physical relationships among joints of subjects in real space to identify sets of candidate joints as multi-joint subjects.

25. The product of claim 24, including storing the sets of joints identified as multi-joint subjects, and wherein the identifying sets of candidate joints includes determining whether a candidate joint identified in images taken at a particular time corresponds with a member of one of the sets of candidate joints identified as a multi-joint subject in a preceding image.

26. The product of claim 21, wherein the sequences of images are synchronized.

27. The product of claim 21, wherein the plurality of cameras comprise cameras disposed over and having fields of view encompassing respective parts of the area in real space, and the coordinates in real space of members of a set of candidate joints identified as a multi-joint subject identify locations in the area of the multi-joint subject.

28. The product of claim 21, including tracking locations of a plurality of multi-joint subjects in the area of real space.

29. The product of claim 28, including determining when multi-joint subjects in the plurality of multi-joint subjects leave the area of real space.

30. The product of claim 21, including tracking locations in the area of real space of multiple candidate joints that are members of a set of candidate joints identified as a particular multi joint subject.