Intelligent event determination and notification in a surveillance system

• US 7,847,820 B2
• Filed: 03/15/2005
• Issued: 12/07/2010
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Kirkman, Thomas W., Marinelli, Marcus S., Russell, Stephen G., Vallone, Robert P., Freeman, J. Andrew
• Primary Examiner(s): Ustaris; Joseph G
• Assistant Examiner(s): Pe; Geepy
• Application Number:
  US11/082,026
• Publication Number:
  US 20050271250A1
• Time in Patent Office: 2,093 Days
• Field of Search: 348/143, 348150-156, 348/159
• US Class Current: 348/143
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method that determines whether a detected event is a significant event requiring action in a video surveillance system. Determined event attributes and object attributes are analyzed to determine if the potential event should trigger an action by the surveillance system. If the potential event should trigger an action, at least one action is triggered. Actions may include relevant event attribute and object attribute information.

Potential event may include events detected by a sensor, or external events communicated by an external system to the surveillance system. Event attributes may include location and type of the potential event. Object attributes may include an identification of an object, and attributes associated with the identified object. If an object cannot be positively identified, object attributes may include potential object identifications or determined group associations of the object.

Claims

1. A method of determining an occurrence of a significant event requiring action in a surveillance system, the method comprising computer-implemented steps of:

for each entity of a plurality of entities, storing one or more event attribute criteria associated with said each entity, wherein the one or more event attribute criteria associated with said each entity indicates at least one of (a) time criteria, (b) location criteria, or (c) direction of motion criteria;

wherein the plurality of entities includes a first entity and a second entity;

wherein the one or more event attribute criteria associated with the first entity are different than the one or more event attribute criteria associated with the second entity;

after storing the one or more event attribute criteria for each entity of the plurality of entities, detecting a potential event;

wherein the potential event is associated with a video surveillance image;

determining one or more event attributes about the potential event, wherein the one or more event attributes indicates at least one of (1) a time, (2) a location, or (3) a direction of motion;

selecting, from among the plurality of entities, an entity that matches an object in the video surveillance image;

if the first entity is selected, then comparing the one or more event attributes about the potential event with the one or more event attribute criteria associated with the first entity to determine whether an action should be triggered by the video surveillance system; and

if the second entity is selected, then comparing the one or more event attributes about the potential event with the one or more event attribute criteria associated with the second entity to determine whether an action should be triggered by the video surveillance system; and

in response to determining that an action should be triggered by the video surveillance system;

determining at least one action to be triggered by the video surveillance system, and

triggering the at least one action;

wherein each entity of said plurality of entities is an object or a group of objects;

wherein the steps are performed by one or more computing devices.

2. The method of claim 1 wherein the step of detecting a potential event comprises:

receiving input from a sensor.

3. The method of claim 1 wherein the step of detecting a potential event comprises:

detecting motion in a video feed from a camera in the surveillance system.

4. The method of claim 1 wherein the step of detecting a potential event comprises:

receiving a potential event indicator, from an external system, by the surveillance system.

5. The method of claim 1, wherein the potential event is a motion detection event, and the one or more event attributes include location and time of motion detection.

6. The method of claim 1, wherein the step of selecting the entity that matches the object in the video surveillance image comprises:

determining a group associated with the object in the video surveillance image; and

determining one or more event attribute criteria associated with the group associated with the identified object in the video surveillance image.

7. The method of claim 1, wherein the step of selecting the entity that matches the object in the video surveillance image comprises:

determining a group not associated with the object in the video surveillance image;

determining one or more event attribute criteria associated with the determined group; and

determining that the one or more event attribute criteria associated with the object in the video surveillance image does not include the one or more event attribute criteria associated with the determined group.

8. The method of claim 1, wherein the one or more event attributes include a time, and the one or more event attribute criteria associated with the first entity include an authorization time range, wherein the step of comparing comprises:

determining that the first potential event should trigger an action by the video surveillance system when the time is not within the authorization time range.

9. The method of claim 1, wherein the step of determining at least one action comprises analyzing the one or more event attributes and the one or more event attribute criteria associated with the first entity or the second entity to determine:

at least one person to notify about the potential event, and

a method of notification associated with the at least one person.

10. The method of claim 9, wherein the determined method of notification includes sending a multimedia e-mail that includes the surveillance video image associated with the potential event.

11. The method of claim 1, wherein the potential event is a first potential event, additionally comprising the steps of:

determining a second potential event;

determining one or more second event attributes about the second potential event;

analyzing the one or more second event attributes and one or more second event attribute criteria associated with a third entity in the plurality of entities to determine whether the second potential event should trigger an action by the video surveillance system; and

in response to determining that the second potential event should trigger an action by the video surveillance system, determining a second action to be triggered by the video surveillance system, wherein the second action is the same as an action determined for the first potential event.

12. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 1.

13. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 2.

14. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 3.

15. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 4.

16. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 5.

17. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 6.

18. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 7.

19. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 8.

20. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 9.

21. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 10.

22. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 11.

33. The method of claim 1, wherein each entity of the plurality of entities is an object.

34. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 33.

35. The method of claim 33, wherein the step of selecting the entity that matches the object in the video surveillance image comprises:

comparing a set of identifying information extracted from the video surveillance image to a plurality of stored sets of identifying information;

calculating a confidence score for each comparison;

for each stored set of identifying information whose comparison confidence score exceeds a predetermined minimum threshold, determining an object associated with the stored set of identifying information; and

identifying a best matching object for the object in the video surveillance image by determining which object has a greatest number of comparison confidence scores that exceed the predetermined minimum threshold.

36. The method of claim 33, wherein the step of selecting the entity that matches the object in the video surveillance image comprises:

identifying a list of known objects that may match the object in the video surveillance image; and

determining, based on one or more event attribute criteria associated with each object in the list of known objects, one or more common event attribute criteria associated with a plurality of objects in the list of known objects.

37. The method of claim 36, wherein the step of identifying a list of known objects that may match the object in the video surveillance image comprises:

comparing a set of identifying information extracted from the video surveillance image to a plurality of stored sets of identifying information;

calculating a confidence score for each comparison;

for each stored set of identifying information whose comparison confidence score exceeds a predetermined minimum threshold, determining an object associated with the stored set of identifying information and including the determined associated object in the list of known objects.

38. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 35.

39. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 36.

40. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 37.

23. A method of determining an occurrence of a significant event requiring action in a surveillance system, the method comprising computer-implemented steps of:

capturing a first video surveillance image from a first camera of the surveillance system;

capturing a second video surveillance image from a second camera of the surveillance system that is different than the first camera;

detecting a first potential event in the first video surveillance image;

determining one or more first event attributes about the first potential event, wherein the one or more first event attributes relate to the first potential event but do not relate to the identity of any objects in the first video surveillance image;

detecting a second potential event in the second video surveillance image;

determining one or more second event attributes about the second potential event, wherein the one or more second event attributes relate to the second potential event but do not relate to the identity of any objects in the second video surveillance image;

comparing the one or more first event attributes with the one or more second event attributes to determine that the first video surveillance image correlates with the second video surveillance image;

based on the correlation of the first video surveillance image with the second video surveillance image, determining that an action should be triggered by the video surveillance system; and

in response to determining that an action should be triggered by the video surveillance system;

determining at least one action to be triggered by the video surveillance system, and

triggering the at least one action;

wherein the steps are performed by one or more computing devices.

28. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 23.

24. A method of determining an occurrence of a significant event requiring action in a surveillance system, the method comprising computer-implemented steps of:

detecting a motion event;

wherein the motion event is associated with a video surveillance image;

determining the direction of the motion that was detected;

determining one or more event attribute criteria associated with an object in the video surveillance image, wherein the one or more event attribute criteria do not relate to the identity of one or more objects in the video surveillance image;

analyzing one or more event attributes about the motion event, including the direction of the motion, and the one or more event attribute criteria to determine whether the motion event should trigger an action by the video surveillance system; and

in response to determining that the motion event should trigger an action by the video surveillance system;

determining at least one action to be triggered by the video surveillance system,

triggering the at least one action;

wherein the steps are performed by one or more computing devices.

29. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 24.

25. A method of determining an occurrence of a significant event requiring action in a surveillance system, the method comprising computer-implemented steps of:

detecting a potential event;

wherein the potential event is associated with a video surveillance image;

determining one or more event attributes about the potential event, wherein the one or more event attributes relate to the potential event but do not relate to the identity of any objects in the video surveillance image;

identifying a list of known objects that may match an object in the video surveillance image; and

determining one or more event attribute criteria that are common among objects in the list of known objects, wherein the one or more event attribute criteria do not relate to the identity of one or more objects in the video surveillance image;

analyzing the one or more event attributes and the one or more event attribute criteria to determine whether the potential event should trigger an action by the video surveillance system; and

in response to determining that the potential event should trigger an action by the video surveillance system;

determining at least one action to be triggered by the video surveillance system, and

triggering the at least one action;

wherein the steps are performed by one or more computing devices.

30. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 25.

26. A method of determining an occurrence of a significant event requiring action in a surveillance system, the method comprising computer-implemented steps of:

for each object of a plurality of objects, storing an authorization time range for said each object;

after storing the authorization time range for said each object of the plurality of objects, detecting a potential event;

wherein the potential event is associated with a video surveillance image;

determining a particular time at which the potential event occurred;

determining an identity of an object in the video surveillance image;

based on the identity of the object and the stored authorization time ranges, determining an authorization time range for the object;

comparing the particular time to the authorization time range for the object to determine whether the potential event should trigger an action by the video surveillance system; and

in response to determining that the potential event should trigger an action by the video surveillance system;

determining at least one action to be triggered by the video surveillance system, and

triggering the at least one action;

wherein the steps are performed by one or more computing devices.

31. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 26.

27. A method of determining an occurrence of a significant event requiring action in a surveillance system, the method comprising computer-implemented steps of:

detecting a potential event;

wherein the potential event is associated with a video surveillance image;

determining one or more event attributes about the potential event, wherein the one or more event attributes relate to the potential event but do not relate to the identity of any objects in the video surveillance image;

determining one or more event attribute criteria associated with an object in the video surveillance image associated with the potential event, wherein the one or more event attribute criteria do not relate to the identity of one or more objects in the video surveillance image;

analyzing the one or more event attributes and the one or more event attribute criteria to determine whether the potential event should trigger an action by the video surveillance system; and

in response to determining that the potential event should trigger an action by the video surveillance system;

selecting, from a plurality of possible persons, at least one person to notify about the potential event based on the one or more event attributes and the one or more event attribute criteria, and

notifying the at least one person about the potential event;

wherein the steps are performed by one or more computing devices.

32. A computer-readable memory medium storing instructions which, when executed by one or more processors, cause performance of the method recited in claim 27.