Assisted video surveillance of persons-of-interest

• US 9,189,687 B2
• Filed: 01/18/2013
• Issued: 11/17/2015
• Est. Priority Date: 11/14/2011
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Thornton, Jason R., Butler, Daniel J., Baran-Gale, Jeanette T.
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): Rahman, Mustafizur
• Application Number:
  US13/745,502
• Publication Number:
  US 20130201330A1
• Time in Patent Office: 1,033 Days
• Field of Search: 348/143, 348/142, 348/153, 348/87, 348/77, 348/61, 348/47, 348/211.14, 348/222.1, 348/376, 348/412.1, 348/415.1, 348/553, 348/564, 348/569, 348/575, 348/714, 348/719, 348/725, 348/739, 715/700, 715/727, 715/728, 715/762, 715/763, 725/37, 725/53, 725/55, 725/93, 725/153, 382/103, 382/118, 382/209, 345/475, 345/594, 345/598, 345/599, 345/650
• US Class Current: 1/1
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Methods, systems and media are described for computer-assisted video surveillance. Methods may support detection of moving persons in video frames, extraction of features of the detected moving persons and identification of which detected moving persons are likely matches to a person of interest. Identification of the likely matches may be determined using an attribute-based search, and/or using a specific person-based search. The method may include using likely matches confirmed as images of the person of interest to reconstruct a path of the person of interest.

Claims

1. A computer implemented method for analyzing video data including a plurality of people, the method comprising:

receiving an identification of a person of interest in an image;

detecting one or more moving persons in a frame for each frame in a plurality of frames of comparison video data, wherein the plurality of frames of comparison video data does not include an image of the person of interest;

creating a record including image data from a subsection of the frame associated with the detected moving person for each of the detected moving persons and for each of the plurality of frames of comparison video data;

creating a person-specific detector for the person of interest by training a support vector machine classifier using at least one image of the person of interest as positive training data and the records of the detected moving persons in plurality of frames of comparison video data as negative training data; and

applying the person-specific detector to a plurality of records from video data of interest to identify one or more records as candidate matches to the person of interest.

2. The computer-implemented method of claim 1, further comprising:

detecting one or more moving persons in a frame for each in a plurality of frames of the video data of interest; and

creating a record including image data from a subsection of the frame associated with the detected moving person for each detected moving person in the plurality of frames of the video of interest.

3. The computer-implemented method of claim 1, wherein receiving an identification of a person of interest in an image comprises receiving an identification of one or more records in one or more frames of the video of interest.

4. The computer-implemented method of claim 1, wherein the support vector machine classifier compares information regarding full-body or nearly full-body images to obtain the person-specific detector.

5. The computer-implemented method of claim 1, wherein the person-specific detector incorporates information about non-facial features of a person.

6. The computer-implemented method of claim 1, further comprising:

displaying the one or more one or more candidate match records to a user; and

receiving a selection of at least one of the one or more candidate match records from the user.

7. The computer implemented method of claim 6, further comprising estimating a spatial path between the selected candidate match records based on time information and on location information associated with each of the selected candidate match records.

8. The computer implemented method of claim 7, further comprising storing the estimated spatial path and the selected candidate match records.

9. The computer-implemented method of claim 6, further comprising creating a composite video including a plurality of video segments with images of the person of interest.

10. The computer-implemented method of claim 9, wherein the plurality of video segments are obtained from a plurality of cameras.

11. The computer-implemented method of claim 9, wherein the composite video is based on the reconstructed path.

12. A non-transitory computer-readable medium encoded with computer-executable instruction for performing a method comprising:

receiving an identification of a person of interest in an image;

detecting one or more moving persons in a frame for each frame in a plurality of frames of comparison video data, wherein the plurality of frames of comparison video data does not include an image of the person of interest;

creating a record including image data from a subsection of the frame associated with the detected moving person for each of the detected moving persons and for each of the plurality of frames of comparison video data;

creating a person-specific detector for the person of interest by training a support vector machine classifier using at least one image of the person of interest as positive training data and the records of the detected moving persons in plurality of frames of comparison video data as negative training data; and

applying the person-specific detector to a plurality of records from video data of interest to identify one or more records as candidate matches to the person of interest.

13. The non-transitory computer-readable medium of claim 12, wherein the method further comprises:

detecting one or more moving persons in a frame for each in a plurality of frames of the video data of interest; and

creating a record including image data from a subsection of the frame associated with the detected moving person for each detected moving person in the plurality of frames of the video of interest.

14. The non-transitory computer-readable medium of claim 12, wherein receiving an identification of a person of interest in an image comprises receiving an identification of one or more records in one or more frames of the video of interest.

15. The non-transitory computer-readable medium of claim 12, wherein the support vector machine classifier compares full-body or nearly full-body images to obtain the person-specific detector.

16. The non-transitory computer-readable medium of claim 12, wherein the person-specific detector incorporates information about non-facial features of a person.

17. The non-transitory computer-readable medium of claim 12, wherein the method further comprises:

displaying the one or more one or more candidate match records to a user; and

receiving a selection of at least one of the one or more candidate match records from the user.

18. The non-transitory computer-readable medium of claim 17, wherein the method further comprises estimating a spatial path between the selected candidate match records based on time information and on location information associated with each of the selected candidate match records.

19. The non-transitory computer-readable medium of claim 18, wherein the method further comprises storing the estimated spatial path and the selected candidate match records.

20. The non-transitory computer-readable medium of claim 18, wherein the method further comprises creating a composite video including a plurality of video segments with images of the person of interest.

21. The non-transitory computer-readable medium of claim 20, wherein the plurality of video segments are obtained from a plurality of cameras.

22. The computer-implemented method of claim 20, wherein the composite video is based on the reconstructed path.

23. A system for video surveillance, the system comprising:

a graphical user interface for receiving search criteria from a user and displaying search results to a user;

a processor for executing computer-executable instructions; and

a storage for storing records including images of detected moving persons in video data, for storing features extracted from the records and for storing computer-executable code, the computer-executable code comprising computer executable instructions for;

a module that detects one or moving persons in a plurality of frames of input video data and saves a record including image information for each detected moving person in each frame;

a module that extracts features for each record, wherein the features include features of a non-head portion of a person; and

a module that receives a user input regarding a person of interest and calculates a match score for a record based on a comparison of the user input regarding the person of interest and the extracted features for the record, the module that calculates a match score including an instance-based search module that;

creates a person-specific detector for the person of interest by training a support vector machine classifier using image data of the person of interest as positive training data and the records of detected moving persons in comparison video data that does not include the person of interest as negative training data; and

applies the person-specific detector to records from video data of interest to calculates an instance-based match score