Interactive system for recognition analysis of multiple streams of video

• US 8,803,975 B2
• Filed: 10/30/2008
• Issued: 08/12/2014
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Haupt, Gordon T., Freeman, J. Andrew, Fleischer, Stephen D., Vallone, Robert P., Russell, Stephen G., Frederick, Timothy B.
• Primary Examiner(s): Dharia, Rupal
• Assistant Examiner(s): Ma, Wing
• Application Number:
  US12/261,554
• Publication Number:
  US 20090074261A1
• Time in Patent Office: 2,112 Days
• Field of Search: 348/156, 348/161, 348169-170, 382/118, 382/218, 382/220, 382226-228
• US Class Current: 348/161
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method of identifying an object captured in a video image in a multi-camera video surveillance system is disclosed. Sets of identifying information are stored in profiles, each profile being associated with one object. The disclosed method of identifying an object includes comparing identifying information extracted from images captured by the video surveillance system to one or more stored profiles. A confidence score is calculated for each comparison and used to determine a best match between the extracted set of identifying information and an object. In one embodiment, the method is used as part of a facial recognition system incorporated into a video surveillance system.

Claims

1. A method for maintaining associations between profiles and objects in an object recognition system, wherein a profile comprises a set of identifying information extracted from an image set associated with the profile, comprising the computer-implemented steps of:

receiving a first image set;

generating a stored profile based on the first image set;

causing to be displayed, to a user, first data that identifies one or more objects that are automatically determined by the object recognition system to be possible matches to the stored profile;

wherein the one or more objects includes a particular object that is associated with a particular stored profile;

after the first data is displayed to the user, receiving, from the user, feedback that indicates that the particular object does not match the stored profile;

in response to receiving the feedback from the user, updating a running score that is (a) maintained for the particular stored profile and (b) based on how often the object recognition system identified the particular object as a possible match but a user indicated the particular object was not a match;

receiving a second image set;

generating a second stored profile based on the second image set;

determining a confidence score in a subsequent possible match that involves the particular stored profile and the second stored profile;

weighting the confidence score based, at least in part, on the running score;

causing to be displayed, to a user, second data that identifies one or more second objects that are automatically determined by the object recognition system to be possible matches to the second stored profile, wherein the object recognition system determines whether to include the particular object in the one or more second objects based on the weighted confidence score;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the objects comprise people and a set of identifying information comprises a facial feature set.

4. The method of claim 1, wherein:

the one or more objects are a plurality of objects that includes the particular object and a second object that is different than the particular object;

the feedback further indicates that the stored profile is of the second object.

5. The method of claim 1, further comprising the computer-implemented steps of:

receiving a new image set;

extracting a first set of identifying information from the new image set;

creating a new profile that includes the first set of identifying information;

providing the new image set to the user;

receiving, from the user, second feedback that indicates a first object that should be associated with the new profile; and

creating a new association between the new profile and the first object.

6. The method of claim 5, wherein:

the step of providing the new image set additionally comprises providing a list of potential object identifications for the new profile; and

the step of receiving the second feedback comprises receiving a choice of one object from the list.

7. The method of claim 5, wherein:

the second feedback indicates that another object should be created; and

the step of creating a new association comprises creating a new object and creating the new association between the new profile and the new object.

8. The method of claim 1, further comprising:

comparing the stored profile with the particular stored profile to generate a second confidence score;

determining whether the second confidence score satisfies one or more criteria;

wherein causing the first data to be displayed is performed in response to determining that the second confidence score satisfies the one or more criteria.

9. The method of claim 8, wherein the one or more criteria includes at least that the second confidence score does not exceed a particular threshold.

10. The method of claim 8, wherein the one or more criteria includes at least that the second confidence score does not exceed a first threshold and does exceed a second threshold that is different than the first threshold.

11. The method of claim 1, wherein:

causing the second data to be displayed comprises causing the second image set that is associated with the second stored profile to be displayed.

12. The method of claim 1, further comprising:

receiving a third image set;

generating a third stored profile based on the third image set;

comparing the third stored profile with the particular stored profile to generate a second confidence score;

determining whether the second confidence score satisfies one or more criteria;

in response to determining that the second confidence score satisfies the one or more criteria, providing the third image set that is associated with the third stored profile and a particular image set that is associated with the particular stored profile to an object identification process that is more computationally expensive than the one or more processes that perform the steps of comparing and determining.

13. The method of claim 8, wherein the one or more criteria includes at least that the second confidence score exceeds a particular threshold.

14. The method of claim 1, further comprising:

after generating the stored profile and prior to receiving the feedback, causing the image set to be displayed.

15. The method of claim 14, further comprising, causing a particular image set upon which the particular stored profile is based to be displayed concurrently with the image set.

16. The method of claim 1, further comprising:

receiving a third image set;

generating a third stored profile based on the third image set;

causing to be displayed, to a second user, a second list of identities of one or more third objects that are automatically determined to be possible matches to the third stored profile;

receiving, from the second user, second feedback that indicates that a first object of the one or more third objects matches an object of the third stored profile;

in response to receiving the second feedback, updating a first possible match score that is (a) associated with a first stored profile of the first object and (b) based on how often the object recognition system identified the first stored profile as a possible match and a user indicated the first stored profile as a match;

determining a second confidence score in a subsequent possible match that involves the first stored profile;

weighting the second confidence score based, at least in part, on the first possible match score.

17. The method of claim 1, further comprising:

determining, based on the running score, whether to remove the particular stored profile from a set of stored profiles.

3. A method for maintaining associations between profiles and objects in an object recognition system, wherein a profile comprises a set of identifying information extracted from an image set associated with the profile, comprising the computer-implemented steps of:

receiving a particular image set;

generating a stored profile based on the particular image set;

causing to be displayed, to a user, a list of identities of one or more objects that are automatically determined to be possible matches to the stored profile;

receiving, from the user, feedback that indicates that a particular object of the one or more objects is not a possible match;

in response to receiving the feedback, updating a possible match score that is (a) associated with a particular stored profile of the particular object and (b) based on how often the object recognition system identified the particular stored profile as a possible match but a user indicated the particular stored profile as not a possible match;

determining a confidence score in a subsequent possible match that involves the particular stored profile;

weighting the confidence score based, at least in part, on the possible match score;

receiving a first image set;

generating a first stored profile based on the first image set;

receiving a second image set;

generating a second stored profile based on the second image set;

comparing the second stored profile with the first stored profile to generate a second confidence score;

determining whether the second confidence score satisfies one or more criteria;

in response to determining that the second confidence score satisfies the one or more criteria, causing to be displayed, to a second user, a second list of identities of one or more second objects that includes a first object and the second image set that is associated with the second stored profile;

receiving, from the second user, feedback that indicates that the first stored profile and the second stored profile are profiles of the same object; and

storing confirmation information that indicates that the second stored profile is of the first object;

wherein the method is performed by one or more computing devices.

18. One or more non-transitory computer-readable media storing instructions for maintaining associations between profiles and objects in an object recognition system, wherein a profile comprises a set of identifying information extracted from an image set associated with the profile, wherein the instructions, when executed by one or more processors, cause:

receiving a first image set;

generating a stored profile based on the first image set;

wherein the one or more objects includes a particular object that is associated with a particular stored profile;

causing to be displayed, to a user, first data that identifies one or more objects that are automatically determined by the object recognition system to be possible matches to the stored profile;

after the first data is displayed to the user, receiving, from the user, feedback that indicates that the particular object does not match the stored profile;

in response to receiving the feedback from the user, updating a running score that is (a) maintained for the particular stored profile and (b) based on how often the object recognition system identified the particular object as a possible match but a user indicated the particular object was not a match;

receiving a second image set;

generating a second stored profile based on the second image set;

determining a confidence score in a subsequent possible match that involves the particular stored profile;

weighting the confidence score based, at least in part, on the possible match score;

causing to be displayed, to a user, second data that identifies one or more second objects that are automatically determined by the object recognition system to be possible matches to the second stored profile, wherein the object recognition system determines whether to include the particular object in the one or more second objects based on the weighted confidence score.

19. The one or more non-transitory computer-readable media of claim 18, wherein the objects comprises people and a set of identifying information comprises a facial feature set.

21. The one or more non-transitory computer-readable media of claim 18, wherein:

the one or more objects are a plurality of objects that includes the particular object and a second object that is different than the particular object;

the feedback further indicates that the stored profile is of the second object.

22. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more processors, further cause:

receiving a new image set;

extracting a first set of identifying information from the new image set;

creating a new profile that includes the first set of identifying information;

providing the new image set to the user;

receiving, from the user, second feedback that indicates a first object that should be associated with the new profile; and

creating a new association between the new profile and the first object.

23. The one or more non-transitory computer-readable media of claim 22, wherein:

the step of providing the new image set additionally comprises providing a list of potential object identifications for the new profile; and

the step of receiving the second feedback comprises receiving a choice of one object from the list.

24. The one or more non-transitory computer-readable media of claim 22, wherein:

the second feedback indicates that another object should be created; and

the step of creating a new association comprises creating a new object and creating the new association between the new profile and the new object.

25. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more processors, further cause:

comparing the stored profile with the particular stored profile to generate a second confidence score;

determining whether the second confidence score satisfies one or more criteria;

wherein causing the first data to be displayed is performed in response to determining that the second confidence score satisfies the one or more criteria.

26. The one or more non-transitory computer-readable media of claim 25, wherein the one or more criteria includes at least that the second confidence score exceeds a particular threshold.

27. The one or more non-transitory computer-readable media of claim 25, wherein the one or more criteria includes at least that the second confidence score does not exceed a particular threshold.

28. The one or more non-transitory computer-readable media of claim 25, wherein the one or more criteria includes at least that the second confidence score does not exceed a first threshold and does exceed a second threshold that is different than the first threshold.

29. The one or more non-transitory computer-readable media of claim 18, wherein:

causing the second data to be displayed comprises causing the second image set that is associated with the second stored profile to be displayed.

30. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more processors, further cause:

receiving a third image set;

generating a third stored profile based on the third image set;

comparing the third stored profile with the particular stored profile to generate a second confidence score;

determining whether the second confidence score satisfies one or more criteria;

in response to determining that the second confidence score satisfies the one or more criteria, providing the third image set that is associated with the third stored profile and a particular image set that is associated with the particular stored profile to an object identification process that is more computationally expensive than the one or more processes that perform the steps of comparing and determining;

second confidence score satisfies the one or more criteria.

31. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more processors, further cause:

after generating the stored profile and prior to receiving the feedback, causing the image set to be displayed.

32. The one or more non-transitory computer-readable media of claim 31, wherein the instructions, when executed by the one or more processors, further cause, causing a particular image set upon which the particular stored profile is based to be displayed concurrently with the image set.

33. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more processors, further cause:

receiving a third image set;

generating a third stored profile based on the third image set;

causing to be displayed, to a second user, a second list of identities of one or more third objects that are automatically determined to be possible matches to the third stored profile;

receiving, from the second user, second feedback that indicates that a first object of the one or more third objects matches an object of the third stored profile;

in response to receiving the second feedback, updating a first possible match score that is (a) associated with a first stored profile of the first object and (b) based on how often the object recognition system identified the first stored profile as a possible match and a user indicated the first stored profile as a match;

determining a second confidence score in a subsequent possible match that involves the first stored profile;

weighting the second confidence score based, at least in part, on the first possible match score.

34. The one or more non-transitory computer-readable media of claim 18, wherein the instructions, when executed by the one or more processors, further cause:

determining, based on the running score, whether to remove the particular stored profile from a set of stored profiles.

20. One or more non-transitory computer-readable media storing instructions for maintaining associations between profiles and objects in an object recognition system, wherein a profile comprises a set of identifying information extracted from an image set associated with the profile, wherein the instructions, when executed by one or more processors, cause:

receiving a particular image set;

generating a stored profile based on the particular image set;

causing to be displayed, to a user, a list of identities of one or more objects that are automatically determined to be possible matches to the stored profile;

receiving, from the user, feedback that indicates that a particular object of the one or more objects is not a possible match;

in response to receiving the feedback, updating a possible match score that is (a) associated with a particular stored profile of the particular object and (b) based on how often the object recognition system identified the particular stored profile as a possible match but a user indicated the particular stored profile as not a possible match;

determining a confidence score in a subsequent possible match that involves the particular stored profile;

weighting the confidence score based, at least in part, on the possible match score;

receiving a first image set;

generating a first stored profile based on the first image set;

receiving a second image set;

generating a second stored profile based on the second image set;

comparing the second stored profile with the first stored profile to generate a second confidence score;

determining whether the second confidence score satisfies one or more criteria;

in response to determining that the second confidence score satisfies the one or more criteria, causing to be displayed, to a second user, a second list of identities of one or more second objects that includes a first object and the second image set that is associated with the second stored profile;

receiving, from the second user, feedback that indicates that the first stored profile and the second stored profile are profiles of the same object; and

storing confirmation information that indicates that the second stored profile is of the first object.