Systems and methods for detecting, identifying and tracking objects and events over time

• US 9,898,647 B2
• Filed: 12/19/2016
• Issued: 02/20/2018
• Est. Priority Date: 06/05/2014
• Status: Active Grant

• Current Assignee: Promethean Limited (Best Assistant Education Group Co.)
• Original Assignee: Promethean Limited (Best Assistant Education Group Co.)
• Inventors: Velozo, Steven C., Decker, Alex H., Hillier, Jason C.
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US15/383,612
• Publication Number:
  US 20170103256A1
• Time in Patent Office: 428 Days
• CPC Class Codes: G06T 2207/10141 Special mode during image acquisition
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

A system for detecting, identifying and tracking objects of interest over time is configured to derive object identification data from images captured from one or more image capture devices. In some embodiments of the system, the one or more image capture devices perform a first object detection and identification analysis on images captured by the one or more image capture devices. The system may then transmit the captured images to a server that performs a second object detection and identification analysis on the captures images. In various embodiments, the second analysis is more detailed than the first analysis. The system may also be configured to compile data from the one or more image capture devices and server into a timeline of object of interest detection and identification data over time.

Claims

1. A system for detecting and tracking one or more events in an area of interest, the system comprising an image capture device comprising one or more cameras, at least one processor operatively coupled to the one or more cameras and memory operatively coupled to the at least one processor, wherein the at least one processor is configured to:

a. capture a first plurality of images at the area of interest between a first start time and a first stop time;

b. after capturing each one of the first plurality of images, analyze each one of the first plurality of images using a first detection method to detect a presence of one or more objects;

c. after capturing each one of the first plurality of images, analyze each one of the first plurality of images using a second detection method to detect a presence of one or more faces near the one or more objects;

d. in response to detecting the presence of the one or more objects, analyze each one of the first plurality of images using a third detection method to recognize each one of the one or more objects;

e. in response to detecting the presence of the one or more faces near the one or more objects, analyze each one of the first plurality of images using a fourth detection method to recognize each one of the one or more faces; and

f. store in the memory information about each one of the one or more recognized faces and objects and a particular time when the respective one or more recognized faces and objects were captured in the image.

2. The system of claim 1, wherein the system is further configured to:

a. analyze each one of the first plurality of images to associate one or more of the recognized objects with one of the one or more of the recognized faces; and

b. store in memory information about each association between the one or more of the recognized objects with the one of the one or more of the recognized faces.

3. The system of claim 2, wherein associating one or more of the recognized objects with one of the one or more of the recognized faces is associated based on proximity between the one or more recognized objects and the one of the one or more recognized faces.

4. The system of claim 1, wherein the system is further configured to:

a. analyze each one of the first plurality of images to determine a presence of one or more machine readable indicia that identify one or more of the recognized objects;

b. analyze each one of the one or more machine readable indicia to associate one or more of the recognized objects with one of the one or more of the recognized faces; and

c. store in memory information about each association between the one or more of the recognized objects with the one of the one or more of the recognized faces.

5. The system of claim 1, wherein the system is further configured to:

a. capture a second plurality of images at the area of interest between a second start time and a second stop time;

b. after capturing each one of the second plurality of images, analyze each one of the second plurality of images using the first detection method to detect a presence of one or more objects;

c. after capturing each one of the second plurality of images, analyze each one of the second plurality of images using the second detection method to detect a presence of one or more faces near the one or more objects;

d. in response to detecting the presence of the one or more objects, analyze each one of the second plurality of images using the third detection method to recognize each one of the one or more objects;

e. in response to detecting the presence of the one or more faces near the one or more objects, analyze each one of the second plurality of images using the fourth detection method to recognize each one of the one or more faces; and

f. store in the memory information about each one of the one or more recognized faces and objects and a particular time when the respective one or more recognized faces and objects were captured in the second plurality of images.

6. The system of claim 5, wherein the system is further configured to store metadata associated with the first plurality of images and the second plurality of images.

7. The system of claim 5, wherein the system is further configured to:

a. for each particular time when the respective one or more recognized objects was captured in the memory, determine a position within the area of interest of the one or more recognized objects; and

b. store information identifying a position pattern in the area of interest of the one or more recognized objects based on each particular time when the respective one or more recognized objects was captured in the memory.

8. The system of claim 7, wherein the position within the area of interest is determined based on location coordinates implemented in the image capturing device.

9. The system of claim 7, wherein the system is further configured to determine that one of the one or more of the recognized objects has been removed from the area of interest based on the position pattern for each of the one or more recognized objects.

10. The system of claim 5, wherein the system is further configured to:

a. analyze each one of the second plurality of images to associate one or more of the recognized objects with one of the one or more of the recognized faces; and

b. store in memory information about each association between the one or more of the recognized objects with the one of the one or more of the recognized faces.

11. The system of claim 5, wherein the system is further configured to:

a. capture a third plurality of images at a second area of interest between a third start time and a third stop time;

b. after capturing each one of the third plurality of images, analyze each one of the third plurality of images using the first detection method to detect the presence of the one or more objects;

c. after capturing each one of the third plurality of images, analyze each one of the third plurality of images using the second detection method to detect a presence of the one or more faces near the one or more objects;

d. in response to detecting the presence of the one or more objects, analyze each one of the third plurality of images using the third detection method to recognize each one of the one or more objects;

e. in response to detecting the presence of the one or more faces near the one or more objects, analyze each one of the third plurality of images using the fourth detection method to recognize each one of the one or more faces; and

f. store in the memory information about each one of the one or more recognized faces and objects and a particular time when the respective one or more recognized faces and recognized objects were captured in the third plurality of image.

12. The system of claim 11, wherein the system is further configured to:

a. analyze each one of the third plurality of images to determine that one or more of the recognized objects is no longer associated with the one of the one or more of the recognized faces; and

b. store in memory information that indicates that the one or more of the recognized objects is no longer associated with the one of the one or more of the recognized faces.

13. The system of claim 5, wherein the first detection method, the second detection method, the third detection method and the fourth detection method are the same detection methods.

14. The system of claim 4, wherein the machine readable indicia includes at least one of a barcode, a QR code, and a specific marking.

15. The system of claim 1, wherein each of the one or more objects further comprise a low power identifier to help identify each of the one or more objects, wherein the low power identifier is chosen from a group consisting of a RFID tag and a Bluetooth transmitter.

16. A system for detecting an event in a classroom, the system comprising an image capture device comprising one or more cameras, at least one processor operatively coupled to the one or more cameras and memory operatively coupled to the at least one processor, wherein the at least one processor is configured to:

a. capture a first plurality of images at in the classroom between a first start time and a first stop time;

b. after capturing each one of the first plurality of images, analyze each one of the first plurality of images to detect a presence of one or more objects;

c. after capturing each one of the first plurality of images, analyze each one of the first plurality of images to detect a presence of one or more faces near the one or more objects;

d. in response to detecting the presence of the one or more objects, analyze each one of the first plurality of images to recognize each one of the one or more objects;

e. in response to detecting the presence of the one or more faces near the one or more objects, analyze each one of the first plurality of images to recognize each one of the one or more faces; and

f. store in the memory information about each one of the one or more recognized faces and objects and a particular time when the respective one or more recognized faces and objects were captured.

17. The system of claim 16, wherein the at least one processor is further configured to flag a face for later review if the face is not recognizable.

18. The system of claim 16, wherein the at least one processor is further configured to flag an object for later review if the object is not recognizable.

19. A system for detecting and tracking one or more objects, the system comprising:

a. an image capture device comprising one or more cameras, at least one processor operatively coupled to the one or more cameras and memory operatively coupled to the at least one processor; and

b. a server having at least one processor and memory, wherein the server is operatively coupled to the image capture device,

i. wherein the image capture device is configured to;

a) capture a first image at a particular time;

b) at least partially in response to capturing the first image, perform identification analysis of the first image to detect and identify at least one object of interest;

c) in response to detecting and identifying the at least one object of interest, storing information associated with the first image and the particular time in the image capture device memory; and

d) transmit at least a portion of the first image to the server; and

ii. wherein the server is configured to;

a) receive the at least one portion of the first image;

b) perform identification analysis of the first image to detect and confirm the identity of the at least one object of interest; and

c) in response to detecting and identifying the at least one object of interest, storing information associated with the first image and the particular time in the server memory.

20. The system of claim 19, wherein the identification analysis further comprises:

a. detecting the presence of a low power identifier associated with at least one of the one or more objects;

b. reading, by a reader, the low power identifier associated with the at least one of the one or more objects;

c. using information received from the low power identifier to confirm the identity of the one or more objects based on the reading of the low power identifier associated with the at least one of the one or more objects.