Assisted video surveillance of persons-of-interest

• US 9,251,424 B2
• Filed: 01/18/2013
• Issued: 02/02/2016
• Est. Priority Date: 11/14/2011
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Thornton, Jason R., Butler, Daniel J., Baran-Gale, Jeanette T.
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): Rahman, Mustafizur
• Application Number:
  US13/745,401
• Publication Number:
  US 20130201329A1
• Time in Patent Office: 1,110 Days
• Field of Search: 348/143, 348/142, 348/153, 348/87, 348/77, 348/61, 348/47, 348/211.14, 348/222.1, 348/376, 348/412.1, 348/415.1, 348/553, 348/564, 348/569, 348/575, 348/714, 348/719, 348/725, 348/739, 348/231.99, 348/237, 715/700, 715/727, 715/728, 715/762, 715/763, 725/37, 725/53, 725/55, 725/93, 725/153, 382/103, 382/118, 382/209, 345/475, 345/594, 345/598, 345/599, 345/650
• US Class Current: 1/1
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Methods, systems and media are described for computer-assisted video surveillance. Methods may support detection of moving persons in video frames, extraction of features of the detected moving persons and identification of which detected moving persons are likely matches to a person of interest. Identification of the likely matches may be determined using an attribute-based search, and/or using a specific person-based search. The method may include using likely matches confirmed as images of the person of interest to reconstruct a path of the person of interest.

Claims

1. A computer-implemented method for analyzing surveillance video data, the method comprising:

detecting one or more moving persons in a frame for each frame in a plurality of frames of the video data;

creating a record including image data from a subsection of the frame associated with the detected moving person for each detected moving person and for each frame;

calculating values for a plurality of attributes characterizing each detected moving person by employing a probabilistic model that matches attributes to image data in the record associated with the detected moving person;

receiving an attribute profile including a value for at least one attribute of a person, wherein the plurality of attributes comprises the at least one attribute;

calculating a score for each record based on a comparison of the received attribute profile with the calculated values for the plurality of attributes for the record; and

identifying one or more records as candidate matches to the person of interest based on the calculated scores.

2. The computer-implemented method of claim 1, wherein the plurality of attributes includes attributes regarding non-head portions of a person.

3. The computer-implemented method of claim 1, wherein the plurality of attributes includes a-full body attribute of a person.

4. The computer-implemented method of claim 1, further comprising:

displaying the one or more candidate match records to a user; and

receiving a selection of at least one of the one or more candidate match records as a match to the person of interest.

5. The computer implemented method of claim 4, further comprising estimating a spatial path between the selected match records based on time information and on location information associated with each of the selected match records.

6. The computer-implemented method of claim 5, further comprising storing the estimated spatial path and the selected match records.

7. The computer-implemented method of claim 5, further comprising creating a composite video including a plurality of video segments with images of the person of interest.

8. The computer-implemented method of claim 7, wherein the plurality of video segments are obtained from a plurality of cameras.

9. The computer-implemented method of claim 7, wherein the composite video is based on the estimated spatial path.

10. The computer-implemented method of claim 1, further comprising:

detecting one or more moving persons in a frame of comparison video data for a plurality of frames of comparison video data that do not include an image of the person of interest; and

creating a record including image data from a subsection of the frame associated with the detected moving person for each of the detected moving persons; and

creating a person-specific detector for the person of interest by training a support vector machine classifier using the selected at least one match record as positive training data and the records of the detected moving persons in the comparison video data as negative training data.

11. The computer-implemented method of claim 1, wherein the video data is obtained from a plurality of imaging devices.

12. A non-transitory computer-readable medium encoded with computer-executable instructions for performing a method comprising:

detecting one or more moving persons in a frame for each frame in a plurality of frames of the video data;

creating a record including image data from a subsection of the frame associated with the detected moving person for each detected moving person and for each frame;

calculating values for a plurality of attributes characterizing each detected moving person by employing a probabilistic model that matches attributes to image data in the record associated with the detected moving person;

receiving an attribute profile including a value for at least one attribute of a person, wherein the plurality of attributes comprises the at least one attribute;

calculating a score for each record based on a comparison of the received attribute profile with the calculated values for the plurality of attributes for the record; and

identifying one or more records as candidate matches to the person of interest based on the calculated scores.

13. The non-transitory computer-readable medium of claim 12, wherein the plurality of attributes includes attributes regarding non-head portions of a person.

14. The non-transitory computer-readable medium of claim 12, wherein the plurality of attributes includes a-full body attribute of a person.

15. The non-transitory computer-readable medium of claim 12, wherein the method further comprises:

displaying the one or more candidate match records to a user; and

receiving a selection of at least one of the one or more candidate match records as a match to the person of interest.

16. The non-transitory computer-readable medium of claim 12, wherein the method further comprises estimating a spatial path between the selected match records based on time information and on location information associated with each of the selected match records.

17. The non-transitory computer-readable medium of claim 16, wherein the method further comprises storing the estimated spatial path and the selected match records.

18. The non-transitory computer-readable medium of claim 16, wherein the method further comprises creating a composite video including a plurality of video segments with images of the person of interest.

19. The non-transitory computer-readable medium of claim 18, wherein the plurality of video segments are obtained from a plurality of cameras.

20. The non-transitory computer-readable medium of claim 18, wherein the composite video is based on the reconstructed path.

21. The non-transitory computer-readable medium of claim 12, wherein the method further comprises:

detecting one or more moving persons in a frame of comparison video data for a plurality of frames of comparison video data that do not include an image of the person of interest; and

creating a record including image data from a subsection of the frame associated with the detected moving person for each of the detected moving persons; and

creating a person-specific detector for the person of interest by training a support vector machine classifier using the selected at least one match record as positive training data and the records of the detected moving persons in the comparison video data as negative training data.

22. The non-transitory computer-readable medium of claim 12, wherein the video data is obtained from a plurality of imaging devices.

23. A system for video surveillance, the system comprising:

a graphical user interface for receiving search criteria from a user and displaying search results to a user;

a processor for executing computer-executable instructions; and

a storage for storing records including images of detected moving persons in video data, for storing features extracted from the records and for storing computer-executable code, the computer-executable code comprising computer executable instructions for;

a module that detects one or moving persons in a plurality of frames of input video data and saves a record including image information for each detected moving person in each frame;

a module that extracts features for each record, wherein the features include features of a non-head portion of a person; and

a module that receives a user input regarding a person of interest and calculates a match score for record based on a comparison of the user input regarding the person of interest and the extracted features for the record, the module that calculates a match score including an attribute-based search module that;

calculates values for a plurality of attributes characterizing each detected moving person by employing a probabilistic model that matches attributes to image data in the record associated with the detected moving person to obtain values for the plurality of attributes associated with each record; and

calculates an attribute-based match score for each record based on a comparison of the received attribute profile with the calculated values for the plurality of attributes for the record.