Facilitating people search in video surveillance

• US 10,424,342 B2
• Filed: 07/28/2010
• Issued: 09/24/2019
• Est. Priority Date: 07/28/2010
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Brown, Lisa M., Feris, Rogerio S., Hampapur, Arun, Vaquero, Daniel A.
• Primary Examiner(s): Adams, Charles D
• Application Number:
  US12/845,116
• Publication Number:
  US 20120030208A1
• Time in Patent Office: 3,345 Days
• Field of Search: 707673, 707679, 707681, 707688, 707691, 707696, 707706, 707711, 707713, 707715, 707722, 707723, 707732, 707739, 707741-746, 707748-756, 707781, 707805, 707 80
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Techniques for facilitating a video surveillance search of a person are provided. The techniques include maintaining a database of one or more attributes of one or more people captured on one or more video cameras, indexing the one or more attributes in the database extracted from the one or more video cameras, and pruning one or more images captured from the one or more video cameras using the one or more attributes and one or more items of qualifying information to facilitate a video surveillance search of a person.

Claims

1. A method for facilitating a video surveillance search of a person, wherein the method comprises:

maintaining a database of attributes of one or more people captured on one or more video cameras, wherein the attributes comprise clothing type, shirt color, pants color presence of sunglasses, presence of eyeglasses, presence of a beard, presence of a mustache, presence of a hat, nose type, sideburns, clothes type, skin color, eye color, presence of a carried bag, and absence or presence of a companion;

indexing the attributes in the database extracted from the one or more video cameras;

pruning one or more images captured from the one or more video cameras using the attributes and one or more items of qualifying information to facilitate a video surveillance search of a specific person, wherein the qualifying information includes (i) information, provided by a user requesting the search, about a particular attribute of the specific person sought in the video surveillance search, (ii) a utility level associated with the particular attribute for identifying a person in a video surveillance search, and (iii) an estimated error value for attribute detection accuracy of the particular attribute in relation to other attributes, and wherein said pruning comprises;

identifying a number of people with the particular attribute captured by the one or more video cameras over a pre-determined period of time to determine an empirical likelihood for identifying the sought person based on distribution of the particular attribute;

pruning the identified number of people with the particular attribute captured by the one or more cameras over the pre-determined period of time to determine an updated empirical likelihood for identifying the sought person, wherein said pruning the identified number of people comprises pruning the images based on the time the sought person was last seen, the direction in which the sought person was travelling, and the speed at which the sought person was travelling;

determining an historical likelihood of the particular attribute being found in the general population; and

determining an accuracy with which the particular attribute is recognized given the utility level associated with the particular attribute and the determined updated empirical likelihood and the determined historical likelihood; and

outputting, to at least one user, a result for the video surveillance search of the person based on said pruning of the one or more images captured from the one or more video cameras.

2. The method of claim 1, wherein indexing comprises ranking and ordering performed by an optimization module, wherein the ranking is based on optimization results that return a likelihood assessment for each person in the database.

3. The method of claim 1, wherein pruning comprises culling the one or more images captured from the one or more video cameras to a subset.

4. The method of claim 1, wherein maintaining a database comprises recording each person seen in a camera in the database with each of one or more detected attributes of the person.

5. The method of claim 1, wherein one or more items of qualifying information comprise at least one of empirical information and historical information.

6. The method of claim 1, wherein one or more items of qualifying information comprise a correlation between two or more attributes.

7. The method of claim 1, further comprising using a user interface to enable a user to supply information about the person.

8. The method of claim 1, further comprising sending a user a set of one or more candidates in the search for the person, wherein the set of one or more candidates are sent to the user in order of their likelihood based on joint probabilities for each candidate.

9. The method of claim 1, further comprising iterating the process of pruning one or more images captured from the one or more video cameras using the attributes and one or more items of qualifying information to facilitate a video surveillance search of a person if the person is not found until a satisfactory match is found or the database is exhausted.

10. The method of claim 1, further comprising modifying attribute description and detection, and finding a most likely feature set based on correlation between one or more attributes and corresponding error distributions if the person is not found.

11. The method of claim 1, further comprising repeating the process of pruning one or more images captured from the one or more video cameras using the attributes and one or more items of qualifying information to facilitate a video surveillance search of a person in a second image captured by the one or more video cameras if the person is found in one image captured by the one or more video cameras.

12. The method of claim 11, further comprising using an updated description of the person based on results from the one image captured by the one or more video cameras.

13. A computer program product comprising a tangible non-transitory computer readable recordable storage medium including computer useable program code for facilitating a video surveillance search of a person, the computer program product including:

computer useable program code for maintaining a database of attributes of one or more people captured on one or more video cameras, wherein the attributes comprise clothing type, shirt color, pants color, presence of sunglasses, presence of eyeglasses, presence of a beard, presence of a mustache, presence of a hat, nose type, sideburns, clothes type, skin color, eye color, presence of a carried bag, and absence or presence of a companion;

computer useable program code for indexing the attributes in the database extracted from the one or more video cameras;

computer useable program code for pruning one or more images captured from the one or more video cameras using the attributes and one or more items of qualifying information to facilitate a video surveillance search of a specific person, wherein the qualifying information includes (i) information, provided by a user requesting the search, about a particular attribute of the specific person sought in the video surveillance search, (ii) a utility level associated with the particular attribute for identifying a person in a video surveillance search, and (iii) an estimated error value for attribute detection accuracy of the particular attribute in relation to other attributes, and wherein said pruning comprises;

identifying a number of people with the particular attribute captured by the one or more video cameras over a pre-determined period of time to determine an empirical likelihood for identifying the sought person based on distribution of the particular attribute;

pruning the identified number of people with the particular attribute captured by the one or more cameras over the pre-determined period of time to determine an updated empirical likelihood for identifying the sought person, wherein said pruning the identified number of people comprises pruning the images based on the time the sought person was last seen, the direction in which the sought person was travelling, and the speed at which the sought person was travelling;

determining an historical likelihood of the particular attribute being found in the general population; and

determining an accuracy with which the particular attribute is recognized given the utility level associated with the particular attribute and the determined updated empirical likelihood and the determined historical likelihood; and

computer useable program code for outputting, to at least one user, a result for the video surveillance search of the person based on said pruning of the one or more images captured from the one or more video cameras.

14. The computer program product of claim 13, wherein one or more items of qualifying information comprise at least one of empirical information, historical information, and a correlation between two or more attributes.

15. The computer program product of claim 13, further comprising:

computer useable program code for using a user interface to enable a user to supply information about the person; and

computer useable program code for sending a user a set of one or more candidates in the search for the person, wherein the set of one or more candidates are sent to the user in order of their likelihood based on joint probabilities for each candidate.

16. The computer program product of claim 13, further comprising computer useable program code for modifying attribute description and detection, and finding a most likely feature set based on correlation between one or more attributes and corresponding error distributions if the person is not found.

17. A system for facilitating a video surveillance search of a person, comprising:

a memory; and

at least one processor coupled to the memory and operative to;

maintain a database of attributes of one or more people captured on one or more video cameras, wherein the attributes comprise clothing type, shirt color, pants color, presence of sunglasses, presence of eyeglasses, presence of a beard, presence of a mustache, presence of a hat, nose type, sideburns, clothes type, skin color, eye color, presence of a carried bag, and absence or presence of a companion;

index the attributes in the database extracted from the one or more video cameras;

prune one or more images captured from the one or more video cameras using the attributes and one or more items of qualifying information to facilitate a video surveillance search of a specific person, wherein the qualifying information includes (i) information, provided by a user requesting the search, about a particular attribute of the specific person sought in the video surveillance search, (ii) a utility level associated with the particular attribute for identifying a person in a video surveillance search, and (iii) an estimated error value for attribute detection accuracy of the particular attribute in relation to other attributes, and wherein said pruning comprises;

identifying a number of people with the particular attribute captured by the one or more video cameras over a pre-determined period of time to determine an empirical likelihood for identifying the sought person based on distribution of the particular attribute;

pruning the identified number of people with the particular attribute captured by the one or more cameras over the pre-determined period of time to determine an updated empirical likelihood for identifying the sought person, wherein said pruning the identified number of people comprises pruning the images based on the time the sought person was last seen, the direction in which the sought person was travelling, and the speed at which the sought person was travelling;

determining an historical likelihood of the particular attribute being found in the general population; and

determining an accuracy with which the particular attribute is recognized given the utility level associated with the particular attribute and the determined updated empirical likelihood and the determined historical likelihood; and

output, to at least one user, a result for the video surveillance search of the person based on said pruning of the one or more images captured from the one or more video cameras.

18. The system of claim 17, wherein one or more items of qualifying information comprise at least one of empirical information, historical information, and a correlation between two or more attributes.

19. The system of claim 17, wherein the at least one processor coupled to the memory is further operative to:

use a user interface to enable a user to supply information about the person; and

send a user a set of one or more candidates in the search for the person, wherein the set of one or more candidates are sent to the user in order of their likelihood based on joint probabilities for each candidate.

20. The system of claim 17, wherein the at least one processor coupled to the memory is further operative to modifying attribute description and detection, and finding a most likely feature set based on correlation between one or more attributes and corresponding error distributions if the person is not found.