Systems and methods for detecting, identifying and tracking objects and events over time

• US 9,524,418 B2
• Filed: 06/05/2014
• Issued: 12/20/2016
• Est. Priority Date: 06/05/2014
• Status: Active Grant

• Current Assignee: Promethean Limited (Best Assistant Education Group Co.)
• Original Assignee: Promethean Limited (Best Assistant Education Group Co.)
• Inventors: Velozo, Steven C., Decker, Alex H., Hillier, Jason C.
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US14/297,494
• Publication Number:
  US 20150356345A1
• Time in Patent Office: 929 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10141 Special mode during image acquisition
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

A system for detecting, identifying and tracking objects of interest over time is configured to derive object identification data from images captured from one or more image capture devices. In some embodiments of the system, the one or more image capture devices perform a first object detection and identification analysis on images captured by the one or more image capture devices. The system may then transmit the captured images to a server that performs a second object detection and identification analysis on the captures images. In various embodiments, the second analysis is more detailed than the first analysis. The system may also be configured to compile data from the one or more image capture devices and server into a timeline of object of interest detection and identification data over time.

Claims

1. A system for detecting one or more objects, the system comprising:

a. an image capture device comprising a camera, at least one processor operatively coupled to the camera and memory operatively coupled to the at least one processor; and

b. a server having at least one processor and memory, wherein the server is operatively coupled to the image capture device,

i. wherein the image capture device is configured to;

a) capture a first image at a particular time;

b) in response to capturing the first image, perform low resolution object detection and identification analysis of the first image to detect and identify at least one object of interest;

c) in response to detecting and identifying the at least one object of interest, storing first information associated with the first image and the particular time in the image capture device memory; and

d) transmit at least one portion of the first image to the server; and

ii. wherein the server is configured to;

a) receive the at least one portion of the first image;

b) perform high resolution object detection and identification analysis of the at least one portion of the first image to detect and identify the at least one object of interest; and

c) in response to detecting and identifying the at least one object of interest, storing second information associated with the at least one portion of the first image and the particular time in the server memory.

2. The system of claim 1, wherein the system is further configured to synchronize the first information in the image capture device memory with the second information in the server memory.

3. The system of claim 1, wherein the image capture device at least one processor is further configured to:

a. capture a second image at a second particular time;

b. detect changes in the second image as compared to the first image;

c. in response to detecting changes, perform low resolution object detection and identification analysis of the first image to detect and identify the at least one object of interest; and

d. in response to identifying the at least one object of interest, storing information associated with the at least one object of interest and the second particular time in the image capture device memory.

4. The system of claim 3, wherein the image capture device at least one processor is further configured to store metadata associated with the first and second images in the image capture device memory.

5. The system of claim 1, wherein the image capture device at least one processor is further configured to transmit the first captured image to the server.

6. The system of claim 5, wherein transmission of the first captured image to the server occurs after transmission of the at least a portion of the first captured image to the server.

7. The system of claim 1, wherein a first object of interest is a face and a second object of interest is a document associated with the face.

8. The system of claim 1, wherein a first object is a face, a second object is a second face and wherein the system is further configured to:

a. determine a first location in which the first and second faces are located;

b. capture a first temporal component associated with the first and second faces; and

c. establish a time and location relationship for the first and second faces.

9. The system of claim 8, wherein the first temporal component further comprises a first start time and a first stop time in which the first and second faces are located in the first location.

15. The system of claim 1, further comprising a plurality of image capture devices operatively coupled to the server, wherein each of the plurality of image capture devices are configured to respectively:

a. capture the first image at the particular time;

b. in response to capturing the first image, perform low resolution object detection and identification analysis of the first image to detect and identify at least one object of interest;

c. in response to detecting and identifying the at least one object of interest in the first image, storing information associated with the first image and the particular time in memory associated with the image capture device that captured the respective first image; and

d. transmit at least a portion of each first image to the server.

10. A system for detecting one or more objects, the system comprising:

a. an image capture device comprising a camera, at least one processor operatively coupled to the camera and memory operatively coupled to the at least one processor; and

b. a server having at least one processor and memory, wherein the server is operatively coupled to the image capture device,

i. wherein the image capture device is configured to;

a) capture a first image at a particular time;

b) in response to capturing the first image, perform low resolution object detection and identification analysis of the first image to detect and identify at least one object of interest;

c) in response to detecting and identifying the at least one object of interest, storing first information associated with the first image and the particular time in the image capture device memory; and

d) transmit at least one portion of the first image to the server; and

ii. wherein the server is configured to;

a. receive the at least one portion of the first image;

b. perform high resolution object detection and identification analysis of the first image to detect and identify the at least one object of interest; and

c. in response to detecting and identifying the at least one object of interest, storing second information associated with the first image and the particular time in the server memory,

iii. wherein a first object is a first face, a second object is a second face and wherein the system is further configured to;

a. determine a first location in which the first and second faces are located;

b. capture a first temporal component associated with the first and second faces;

c. establish a time and location relationship for the first and second faces;

d. determine a second location in which the first face is located and a third location in which the second face is located;

e. capture a second temporal component associated with the first and second faces; and

f. establish a second time and second location relationship for the first face and the second time and third location relationship for the second face, wherein the first temporal component further comprises a first start time and a first stop time in which the first and second faces are located in the first location.

11. The system of claim 10, wherein the system is further configured to detect when the first face is incorrectly located at the third location during the second time.

12. The system of claim 10, wherein the second time occurs on a second day and the first time occurs on a first day prior to the second day.

13. The system of claim 10, wherein the first start and second stop times occur on the same day.

14. The system of claim 11, wherein the system is further configured to notify the user when the first face is incorrectly located at the third location.

16. A system for detecting an event in a classroom, the system comprising an image capture device comprising a camera, at least one processor operatively coupled to the camera and memory operatively coupled to the at least one processor, wherein the at least one processor is configured to:

a. capture a first plurality of images at a first location between a first start time and a first stop time;

b. after capturing each one of the plurality of images, analyze each one of the first plurality of images using a first filter to detect the presence of one or more faces;

c. in response to detecting the presence of the one or more faces, analyze each one of the first plurality of images using a second filter to recognize each one of the one or more detected faces; and

d. store information about each one of the one or more recognized faces and a particular time when the respective one of the first plurality of images was captured in the memory;

e. capture a second plurality of images at a second location between a second start time and a second stop time;

f. after capturing each one of the second plurality of images, analyze each one of the second plurality of images using the first filter to detect the presence of one or more faces;

g. in response to detecting the presence of the one or more faces in the second plurality of images, analyze each one of the second plurality of images using the second filter to recognize each one of the one or more detected faces;

h. store information about the each one of the one or more faces recognized from the second plurality of images and a particular time when the respective one of the second plurality of images were captured in memory;

i. capture a third plurality of images at a third location between the second start time and the second stop time;

j. after capturing each one of the third plurality of images, analyze each one of the third plurality of images using the first filter to detect the presence of one or more faces;

k. in response to detecting the presence of the one or more faces in the third plurality of images, analyze each one of the third plurality of images using the second filter to recognize each one of the one or more detected faces; and

l. store information about the each one of the one or more faces recognized from the third plurality of images and a particular time when the respective one of the third plurality of images were captured in memory.

17. The system of claim 16, wherein the at least one processor is further configured to:

a. generate time and location relationships between the recognized one or more faces in the first plurality of images, the first location and the first start time and the second start times;

b. generate time and location relationships between the recognized one or more faces in the second plurality of images, the second location and the second start and the second stop times; and

c. generate time a location relationships between the recognized one or more faces in the third plurality of images, the third location and the second start and the second stop times.

18. The system of claim 17, wherein

a. the first start time and the first stop time represent the beginning and end of a first period of a multi-period school day;

b. the second start time and the third start time represent the beginning and end of a second period of a multi-period school day, where the second period occurs after the first period;

c. the first location is a first classroom, second location is a second classroom and the third location is a third classroom; and

d. the time and location relationships between the recognized one or more faces from the first, second and third plurality of images, the first, second and third classrooms and the first and second periods represent the attendance of students in each of the first, second and third classrooms during the respective first and second periods.

19. The system of claim 18, wherein the at least one processor is further configured to:

a. capture a fourth plurality of images at the first location between the third start time and a third stop time;

b. after capturing each one of the fourth plurality of images, analyze each one of the fourth plurality of images using the first filter to detect the presence of one or more faces;

c. in response to detecting the presence of the one or more faces, analyze each one of the fourth plurality of images using the second filter to recognize each one of the one or more detected faces;

d. store information about each one of the one or more recognized faces from the fourth plurality of images and a particular time when the respective one of the fourth plurality of images was captured in the memory;

e. capture a fifth plurality of images at the second location between a fourth start time and a fourth stop time;

f. after capturing each one of the fifth plurality of images, analyze each one of the fifth plurality of images using the first filter to detect the presence of one or more faces;

g. in response to detecting the presence of the one or more faces in the fifth plurality of images, analyze each one of the fifth plurality of images using the second filter to recognize each one of the one or more detected faces;

h. store information about the each one of the one or more faces recognized from the fifth plurality of images and a particular time when the respective one of the fifth plurality of images were captured in memory;

i. capture a sixth plurality of images at the third location between the fourth start time and the fourth stop time;

j. after capturing each one of the sixth plurality of images, analyze each one of the sixth plurality of images using the first filter to detect the presence of one or more faces;

k. in response to detecting the presence of the one or more faces in the sixth plurality of images, analyze each one of the sixth plurality of images using the second filter to recognize each one of the one or more detected faces; and

l. store information about the each one of the one or more faces recognized from the sixth plurality of images and a particular time when the respective one of the sixth plurality of images were captured in memory.

20. The system of claim 19, wherein the third start and third stop time and the fourth start and fourth stop time occur on a different day than the first start and first stop time and the second start and second stop time.