Interactive system for recognition analysis of multiple streams of video

• US 7,529,411 B2
• Filed: 03/15/2005
• Issued: 05/05/2009
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: Identiv, Inc.
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Frederick, Timothy B., Freeman, J. Andrew
• Primary Examiner(s): Mariam; Daniel G
• Application Number:
  US11/081,753
• Publication Number:
  US 20050207622A1
• Time in Patent Office: 1,512 Days
• Field of Search: 382/209, 382/218, 382/118, 382/217, 382/103, 382/224, 382/305, 382/220, 348/143, 707 1- 10
• US Class Current: 382/220
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method of identifying an object captured in a video image in a multi-camera video surveillance system is disclosed. Sets of identifying information are stored in profiles, each profile being associated with one object. The disclosed method of identifying an object includes comparing identifying information extracted from images captured by the video surveillance system to one or more stored profiles. A confidence score is calculated for each comparison and used to determine a best match between the extracted set of identifying information and an object. In one embodiment, the method is used as part of a facial recognition system incorporated into a video surveillance system.

Claims

1. A method of determining a best match between a target profile and a set of stored profiles, wherein a profile contains a set of identifying information extracted from an image set associated with the profile, wherein the set of stored profiles corresponds to a plurality of different objects, the method comprising computer-implemented steps of:

generating a plurality of confidence scores, wherein each confidence score of the plurality of confidence scores is based on a comparison between a feature set of the target profile and a feature set of a profile in the set of stored profiles, wherein each confidence score numerically represents the similarity between the feature set of the target profile and a feature set of a profile in the set of stored profiles;

generating, from the plurality of confidence scores, a plurality of weighted confidence scores based on information external to the feature sets; and

selecting, from the set of stored profiles, a particular profile as the best match for the target profile based on the plurality of weighted confidence scores;

wherein the method is performed by one or more computing devices programmed to be a special purpose machine pursuant to instructions from program software.

2. The method of claim 1 wherein each stored profile has an age, and the information used to weight the confidence scores comprises profile age information.

3. The method of claim 2 wherein a first stored profile is older than a second stored profile, and wherein a weighting factor for said second stored profile is greater than a weighting factor for said first stored profile.

4. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 3.

5. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 2.

6. The method of claim 1 wherein the image set associated with the target profile is captured by a first camera at a first time, and wherein the information used to weight the confidence scores comprises information associated with whether a best matching profile has been determined for another profile associated with another image set captured by the first camera within a predetermined time frame of the first time.

7. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 6.

8. The method of claim 1 wherein the image set associated with the target profile is captured by a first camera at a first time, and the information used to weight the confidence scores comprises information associated with whether a best matching profile has been determined for another profile associated with another image set captured by a second camera within a predetermined time frame of the first time.

9. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 8.

10. The method of claim 1 wherein the information used to weight the confidence scores comprises a rate of confirmed correct identifications associated with at least one profile in the set of stored profiles.

11. The method of claim 10 wherein a first stored profile has a confirmed correct identification rate that is greater than a confirmed correct identification rate of a second stored profile, and wherein a weighting factor for said first stored profile is greater than a weighting factor for said second stored profile.

12. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 11.

13. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 10.

14. The method of claim 1 wherein an image set is acquired from a multi-camera video surveillance system.

15. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 14.

16. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 1.

17. The method of claim 1, wherein the information external to the feature sets includes information about people that correspond to profiles in the set of stored profiles.

18. The method of claim 17, wherein the information external to the feature sets includes information about categories assigned to people that correspond to profiles in the set of stored profiles.

19. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 18.

20. The method of claim 17, wherein the information external to the feature sets includes information about schedules of people that correspond to profiles in the set of stored profiles.

21. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 20.

22. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 17.

23. A method of determining a best matching object for a target profile in an object recognition system in which each object recognized by the system is associated with one or more profiles in a set of stored profiles, wherein each profile in the set of stored profiles contains a set of identifying information extracted from an image set associated with the profile, the method comprising the computer-implemented steps of:

generating a plurality of confidence scores, wherein each confidence score of the plurality of confidence scores is based on a comparison between the target profile and a different profile in the set of stored profiles;

for each profile in the set of stored profiles, identifying an object associated with said each profile;

generating a value for each object of a plurality of different objects wherein the method is performed by one or more computing devices programmed to be a special purpose machine pursuant to instructions from program software, wherein the value generated for each object is based on one or more confidence scores, of the plurality of confidence scores, associated with one or more profiles of said each object;

wherein the step of generating a value for each object includes, for at least one of the plurality of different objects, generating a value for the object based on confidence scores associated with a plurality of profiles of the object; and

selecting, from the plurality of different objects, a best matching object based, at least in part, on the values generated for each object of the plurality of different objects.

24. The method of claim 23, wherein:

the step of generating a value comprises determining, for each object, a number of occurrences of that object being associated with a confidence score, and

the step of selecting a best matching object comprises selecting the object with a greatest number of occurrences.

25. The method of claim 24, wherein the step of generating a value comprises determining, for each object, a number of occurrences of that object being associated with a confidence score that exceeds a predetermined minimum threshold.

26. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 25.

27. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 24.

28. The method of claim 23, wherein:

the step of generating a value comprises calculating, for each object, a weighted average of confidence scores associated with that object; and

the step of selecting a best matching object comprises selecting the object with a greatest weighted average.

29. The method of claim 28, wherein each stored profile has an age, the step of calculating a weighted average of confidence scores comprises:

for each object, weighting each confidence score associated with the object in accordance with the age of the profile associated with the confidence score, and averaging all weighted confidence scores associated with that object.

30. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 29.

31. The method of claim 28, wherein the step of calculating a weighted average of confidence scores comprises:

for each object, determining when the object was last identified by the object recognition system, and weighting the average confidence scores associated with that object in accordance with when the object was last identified.

32. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 31.

33. The method of claim 28, additionally comprising the step of:

ranking the generated confidence scores;

wherein the step of calculating a weighted average of confidence scores comprises using the rankings to weight confidence scores.

34. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 33.

35. The method of claim 28, additionally comprising the step of:

receiving access information from an access control system;

wherein the step of calculating a weighted average of confidence scores comprises using the received access information to weight the confidence scores.

36. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 35.

37. The method of claim 28, wherein an object comprises a person, and a set of identifying information comprises a facial feature set.

38. The method of claim 37, additionally comprising the step of:

for each person associated with a profile used to generate a confidence score, determining organizational information about the person,

wherein the step of calculating a weighted average of confidence scores comprises using the organizational information to weight the confidence scores.

39. The method of claim 38, wherein the target profile has a creation time, and the organizational information includes a work shift associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether the work shift of the person corresponds to the creation time of the target profile, and using said determination to weight the confidence scores.

40. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 39.

41. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 37.

42. The method of claim 38, wherein the target profile has a location, and the organizational information includes at least one authorized location associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether any authorized location of the person corresponds to the location of the target profile, and using said determination to weight the confidence scores.

43. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 42.

44. The method of claim 38, wherein the target profile has a location, and the organizational information includes at least one organizational department associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether any organizational department of the person corresponds to the location of the target profile, and using said determination to weight the confidence scores.

45. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 44.

46. The method of claim 38, wherein the organizational information includes at least one organizational department associated with the person, wherein the step of calculating a weighted average of confidence scores comprises determining whether another person associated with at least one organization department associated with the person has been identified by the system in time and location proximity with the person.

47. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 46.

48. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 38.

49. The method of claim 28, wherein each stored profile has a rate of correct identifications, wherein the step of calculating a weighted average of confidence scores comprises:

for each object, weighting each confidence score associated with the object in accordance with the correct identification rate of the profile associated with the confidence score, and averaging all weighted confidence scores associated with that object.

50. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 49.

51. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 28.

52. The method of claim 23, additionally comprising the steps of:

determining if any of the plurality of confidence scores exceed a predetermined minimum threshold, and if no scores exceed the minimum threshold, then the method further comprising the steps of;

providing a view of the image set associated with the target profile to a user to determine if the object in the image set matches an object recognized by the object recognition system, and if the user determines that the object in the image does not match a recognized object, enrolling the object as a new object and saving the target profile associated with the new object.

53. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 52.

54. A computer-readable storage medium storing instructions, wherein the computer-readable storage medium includes at least one of volatile storage or non-volatile storage, wherein the instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the method recited in claim 23.