Assisted video surveillance of persons-of-interest

• US 9,111,147 B2
• Filed: 11/14/2012
• Issued: 08/18/2015
• Est. Priority Date: 11/14/2011
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Thornton, Jason R., Butler, Daniel J., Baran-Gale, Jeanette T.
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): Rahman, Mustafizur
• Application Number:
  US13/677,183
• Publication Number:
  US 20130155229A1
• Time in Patent Office: 1,007 Days
• Field of Search: 348/143, 348/142, 348/153, 348/87, 348/77, 348/61, 348/47, 348/211.14, 348/222.1, 348/376, 348/412.1, 348/415.1, 348/553, 348/564, 348/569, 348/575, 348/714, 348/719, 348/725, 348/739, 715/700, 715/727, 715/728, 715/762, 715/763, 725/37, 725/53, 725/55, 725/93, 725/153, 382/103, 382/118, 382/209, 345/475, 345/594, 345/598, 345/599, 345/650
• US Class Current: 1/1
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Methods, systems and media are described for computer-assisted video surveillance. Methods may support detection of moving persons in video frames, extraction of features of the detected moving persons and identification of which detected moving persons are likely matches to a person of interest. Identification of the likely matches may be determined using an attribute-based search, and/or using a specific person-based search. The method may include using likely matches confirmed as images of the person of interest to reconstruct a path of the person of interest.

Claims

1. A computer-implemented method for analyzing surveillance video data, the method comprising:

receiving information regarding a person of interest;

detecting a first moving person in a first frame of the video data;

creating a first record including image data from a subsection of the first frame associated with the first moving person;

detecting a second moving person in a second frame of the video data;

creating a second record including image data from a subsection of the second frame associated with the second moving person;

identifying the first record and the second record as candidate matches to the person of interest based on features of the first record, features of the second record and information received regarding the person of interest; and

reconstructing the path of the person of interest based on spatial and temporal information regarding the first record and the second record.

2. The method of claim 1, further comprising displaying the first record and the second record to a user as candidate matches to the person of interest.

3. The method of claim 2, further comprising receiving a confirmation from the user that the first record and the second record include image data for the person of interest.

4. The method of claim 1, further comprising displaying the reconstructed path to a user.

5. The method of claim 1, wherein the information received regarding the person of interest is an attribute profile, and wherein identifying the first record and the second record as candidate matches to the person of interest comprises:

calculating values for a plurality of attributes characterizing each detected moving person by employing a probabilistic model that matches attributes to image data in the record associated with the detected moving person to obtain values for the plurality of attributes associated with each record; and

calculating a score for each record based on a comparison of the received attribute profile with the calculated values for the plurality of attributes for the record.

6. The method of claim 5, wherein the attribute profile includes attributes regarding non-head portions of a person.

7. The method of claim 1, further comprising creating a composite video including a plurality of video segments with images of the person of interest.

8. The method of claim 7, wherein the plurality of video segments are obtained from a plurality of cameras.

9. The method of claim 7, wherein the composite video is based on the reconstructed path.

10. The method of claim 1, wherein the information received regarding the person of interest is an identification of the person of interest in an image, and wherein identifying the first record and the second record as candidate matches to the person of interest comprises:

detecting a moving person in a frame of comparison video data for each frame in a plurality of frames of comparison video data;

creating a record including image data from a subsection of the frame associated with the detected moving person for each of the plurality of frames of comparison video data;

creating a person-specific detector for the person of interest by training a support vector machine classifier using image data of the person of interest as positive training data and the records of the detected moving persons in the comparison video data as negative training data; and

applying the person-specific detector to records from video data of interest to identify the first record and the second record as candidate matches to the person of interest.

11. The computer-implemented method of claim 1, wherein the video data is obtained from a plurality of imaging devices.

12. A non-transitory computer-readable medium encoded with computer-executable instructions for performing a method comprising:

receiving information regarding a person of interest;

detecting a first moving person in a first frame of video data;

creating a first record including image data from a subsection of the first frame associated with the first moving person;

detecting a second moving person in a second frame of the video data;

creating a second record including image data from a subsection of the second frame associated with the second moving person;

identifying the first record and the second record as candidate matches to the person of interest based on features of the first record, features of the second record and information received regarding the person of interest; and

reconstructing the path of the person of interest based on spatial and temporal information regarding the first record and the second record.

13. The non-transitory computer-readable medium of claim 12, wherein the method further comprises displaying the first record and the second record to a user as candidate matches to the person of interest.

14. The non-transitory computer-readable medium of claim 13, wherein the method further comprises receiving a confirmation from the user that the first record and the second record include image data for the person of interest.

15. The non-transitory computer-readable medium of claim 12, wherein the method further comprises displaying the reconstructed path to a user.

16. The non-transitory computer-readable medium of claim 12, wherein the method further comprises creating a composite video including a plurality of video segments with images of the person of interest.

17. The non-transitory computer-readable medium of claim 16, wherein the plurality of video segments are obtained from a plurality of cameras.

18. The non-transitory computer-readable medium of claim 16, wherein the composite video is based on the reconstructed path.

19. The non-transitory computer-readable medium of claim 12, wherein the information received regarding the person of interest is an attribute profile, and wherein identifying the first record and the second record as candidate matches to the person of interest comprises:

calculating values for a plurality of attributes characterizing each detected moving person by employing a probabilistic model that matches attributes to image data in the record associated with the detected moving person to obtain values for the plurality of attributes associated with each record; and

calculating a score for each record based on a comparison of the received attribute profile with the calculated values for the plurality of attributes for the record.

20. The non-transitory computer-readable medium of claim 19, wherein the attribute profile includes attributes regarding non-head portions of a person.

21. The non-transitory computer-readable medium of claim 19, wherein the attribute profile includes a full-body attribute of a person.

22. The non-transitory computer-readable medium of claim 12, wherein the information received regarding the person of interest is an identification of the person of interest in an image, and wherein identifying the first record and the second record as candidate matches to the person of interest comprises:

detecting a moving person in a frame of comparison video data for each frame in a plurality of frames of comparison video data;

creating a record including image data from a subsection of the frame associated with the detected moving person for each of the plurality of frames of comparison video data;

creating a person-specific detector for the person of interest by training a support vector machine classifier using image data of the person of interest as positive training data and the records of the detected moving persons in the comparison video data as negative training data; and

applying the person-specific detector to records from video data of interest to identify the first record and the second record as candidate matches to the person of interest.

23. The non-transitory computer-readable medium of claim 12, wherein the video data is obtained from a plurality of imaging devices.

24. A system for video surveillance, the system comprising:

a graphical user interface for receiving search criteria from a user and displaying search results to a user;

a processor for executing computer-executable instructions; and

a storage for storing records including images of detected moving persons in video data, for storing features extracted from the records and for storing computer-executable code, the computer-executable code comprising computer executable instructions for;

a module that detects one or moving persons in a plurality of frames of input video data and saves a record including image information for each detected moving person in each frame;

a module that extracts features for each record, wherein the features include features of a non-head portion of a person; and

a module that receives a user input regarding a person of interest and calculates a match score for a record based on a comparison of the user input regarding the person of interest and the extracted features for the record.

25. The system of claim 24, wherein the computer-executable instructions further comprise instructions for a module for video retrieval and path reconstruction that reconstructs a path of the person of interest based on a user'"'"'s selection of two or more records as records of the person of interest.