Interactive system for recognition analysis of multiple streams of video

• US 20050207622A1
• Filed: 03/15/2005
• Published: 09/22/2005
• Est. Priority Date: 03/16/2004
• Status: Active Grant

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Frederick, Timothy B., Freeman, J. Andrew
• Granted Patent: US 7,529,411 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method of identifying an object captured in a video image in a multi-camera video surveillance system is disclosed. Sets of identifying information are stored in profiles, each profile being associated with one object. The disclosed method of identifying an object includes comparing identifying information extracted from images captured by the video surveillance system to one or more stored profiles. A confidence score is calculated for each comparison and used to determine a best match between the extracted set of identifying information and an object. In one embodiment, the method is used as part of a facial recognition system incorporated into a video surveillance system.

Claims

1. A method of determining a best match between a target profile and a set of stored profiles, wherein a profile contains a set of identifying information extracted from an image set associated with the profile, the method comprising computer-implemented steps of:

generating a plurality of confidence scores based on comparisons between the target profile and the set of stored profiles;

weighting the generated confidence scores using information external to the confidence scores; and

selecting a stored profile as the best match for the target profile based on the plurality of weighted confidence scores.

2. The method of claim 1 wherein each stored profile has an age, and the information used to weight the confidence scores comprises profile age information.

3. The method of claim 2 wherein a first stored profile is older than a second stored profile, and wherein a weighting factor for said second stored profile is greater than a weighting factor for said first stored profile.

4. The method of claim 1 wherein the image set associated with the target profile is captured by a first camera at a first time, and wherein the information used to weight the confidence scores comprises information associated with whether a best matching profile has been determined for another profile associated with another image set captured by the first camera within a predetermined time frame of the first time.

5. The method of claim 1 wherein the image set associated with the target profile is captured by a first camera at a first time, and the information used to weight the confidence scores comprises information associated with whether a best matching profile has been determined for another profile associated with another image set captured by a second camera within a predetermined time frame of the first time.

6. The method of claim 1 wherein the information used to weight the confidence scores comprises a rate of confirmed correct identifications associated with at least one profile in the set of stored profiles.

7. The method of claim 6 wherein a first stored profile has a confirmed correct identification rate that is greater than a confirmed correct identification rate of a second stored profile, and wherein a weighting factor for said first stored profile is greater than a weighting factor for said second stored profile.

8. The method of claim 1 wherein an image set is acquired from a multi-camera video surveillance system.

35. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 1.

9. A method for maintaining associations between profiles and objects in an object recognition system, wherein a profile comprises a set of identifying information extracted from an image set associated with the profile, comprising the computer-implemented steps of:

automatically creating a first association between a first stored profile and a first object;

automatically creating a second association between a second stored profile and the first object;

providing a view of the image sets associated with the first and second stored profiles to a user;

receiving feedback from the user about the second association; and

modifying the second association in accordance with the received feedback.

10. The method of claim 9, wherein an object comprises a person, and a set of identifying information comprises a facial feature set.

11. The method of claim 9, wherein the step of providing a view of the image sets associated with the first and second stored profiles additionally comprises providing a view of the first and second associations to the user.

12. The method of claim 11, wherein the step of receiving feedback includes receiving feedback that the second association is correct, and wherein the step of modifying includes storing confirmation information with the second association.

13. The method of claim 11, wherein the step of receiving feedback includes receiving feedback that the second association is incorrect, and the step of modifying includes removing the second stored profile from the object recognition system.

14. The method of claim 11, wherein the step of receiving feedback includes receiving feedback indicating that the second stored profile should be associated with a second object;

and the step of modifying includes creating an association between the second stored profile and the second object.

15. The method of claim 9, further comprising the computer-implemented steps of:

receiving a new image set;

extracting a set of identifying information from the new image set;

creating a new profile that includes the extracted set of identifying information;

providing a view of the new image set to a user;

receiving feedback from the user indicating an object that should be associated with the new profile; and

creating a new association between the new profile and the object indicated by the user.

16. The method of claim 15, wherein the step of receiving feedback from the user indicating an object comprises receiving feedback from the user indicating that the new profile should be associated with the first object;

and wherein the step of creating an association comprises creating an association between the new profile and the first object, such that the first stored profile, the second stored profile and the new profile are all associated with the first object.

17. The method of claim 15, wherein the step of providing a view of the new image set additionally comprises providing a list of potential object identifications for the new profile, and the step of receiving feedback from the user indicating an object comprises receiving a choice of one object from the list.

18. The method of claim 15, wherein the step of receiving feedback from the user indicating an object comprises receiving feedback that a new object should be created, and the step of creating a new association comprises creating a new object and creating an association between the new profile and the new object.

36. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 9.

19. A method of determining a best matching object for a target profile in an object recognition system in which each object recognized by the system is associated with a plurality of stored profiles, wherein a profile contains a set of identifying information extracted from an image set associated with the profile, the method comprising the computer-implemented steps of:

generating a plurality of confidence scores based on comparisons between the target profile and a set of stored profiles;

determining an object associated with each generated confidence score;

analyzing the generated confidence scores and determined associated objects; and

selecting a best matching object in accordance with the analyzed confidence scores and determined associated objects.

20. The method of claim 19, wherein the step of analyzing comprises determining, for each object, a number of occurrences of that object being associated with a confidence score, and the step of selecting a best matching object comprises selecting the object with a greatest number of occurrences.

21. The method of claim 20, wherein the step of analyzing comprises determining, for each object, a number of occurrences of that object being associated with a confidence score that exceeds a predetermined minimum threshold.

22. The method of claim 19, wherein the step of analyzing comprises calculating, for each object, a weighted average of confidence scores associated with that object;

and the step of selecting a best matching object comprises selecting the object with a greatest weighted average.

23. The method of claim 22, wherein each stored profile has an age, wherein the step of calculating a weighted average of confidence scores comprises:

for each object, weighting each confidence score associated with the object in accordance with the age of the profile associated with the confidence score, and

averaging all weighted confidence scores associated with that object.

24. The method of claim 22, wherein the step of calculating a weighted average of confidence scores comprises:

for each object, determining when the object was last identified by the system, and

weighting the average confidence scores associated with that object in accordance with when the object was last identified.

25. The method of claim 22, additionally comprising the step of:

ranking the generated confidence scores;

wherein the step of calculating a weighted average of confidence scores comprises using the rankings to weight confidence scores.

26. The method of claim 22, additionally comprising the step of:

receiving access information from an access control system;

wherein the step of calculating a weighted average of confidence scores comprises using the received access information to weight the confidence scores.

27. The method of claim 22, wherein an object comprises a person, and a set of identifying information comprises a facial feature set.

28. The method of claim 27, additionally comprising the step of:

for each person associated with a profile used to generate a confidence score, determining organizational information about the person,

wherein the step of calculating a weighted average of confidence scores comprises using the organizational information to weight the confidence scores.

29. The method of claim 28, wherein the target profile has a creation time, and the organizational information includes a work shift associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether the work shift of the person corresponds to the creation time of the target profile, and using said determination to weight the confidence scores.

30. The method of claim 28, wherein the target profile has a location, and the organizational information includes at least one authorized location associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether any authorized location of the person corresponds to the location of the target profile, and using said determination to weight the confidence scores.

31. The method of claim 28, wherein the target profile has a location, and the organizational information includes at least one organizational department associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether any organizational department of the person corresponds to the location of the target profile, and using said determination to weight the confidence scores.

32. The method of claim 28, wherein the organizational information includes at least one organizational department associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether another person associated with at least one organization department associated with the person has been identified by the system in time and location proximity with the person.

33. The method of claim 22, wherein each stored profile has a rate of correct identifications, wherein the step of calculating a weighted average of confidence scores comprises:

for each object, weighting each confidence score associated with the object in accordance with the correct identification rate of the profile associated with the confidence score, and averaging all weighted confidence scores associated with that object.

34. The method of claim 19, additionally comprising the steps of:

determining if any of the plurality of confidence scores exceed a predetermined minimum threshold, and if no scores exceed the minimum threshold, then the method further comprising the steps of;

providing a view of the image set associated with the target profile to a user to determine if the object in the image set matches an object recognized by the object recognition system, and if the user determines that the object in the image does not match a recognized object, enrolling the object as a new object and saving the target profile associated with the new object.

37. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 19.