Video surveillance, storage, and alerting system having network management, hierarchical data storage, video tip processing, and vehicle plate analysis

• US 7,382,244 B1
• Filed: 10/04/2007
• Issued: 06/03/2008
• Est. Priority Date: 10/04/2007
• Status: Active Grant

• Current Assignee: SecureNet Solutions Group, LLC
• Original Assignee: KD Secure LLC
• Inventors: Donovan, John J., Hussain, Daniar
• Primary Examiner(s): La, Anh V
• Application Number:
  US11/867,575
• Time in Patent Office: 243 Days
• Field of Search: 340/506, 340/531, 340/539.25, 340/517, 340/521, 340/532, 340/541, 340/3.1, 340/519, 340/540, 340/545.2, 340/933, 340/936, 340/937, 340/942, 340/934, 340/286.07, 340/573.4, 348/143, 348/148, 348/149, 348/14.01, 382/103, 382/104, 702/159, 701/119
• US Class Current: 340/506
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: G08B 13/19697 Arrangements wherein non-video detectors generate an alarm themselves
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

The present invention is a video surveillance, storage, and alerting system having surveillance cameras, video analytics devices, audio sensory devices, other sensory devices, and a plurality of data storage devices. A network management module monitors network status of all subsystems including cameras, servers, storage devices, etc. and shows actively monitored areas on a physical map. A vehicle information module retrieves information from a law enforcement database about vehicles detected in the video data based on the vehicle'"'"'s license plate, including information about stolen vehicles, as well as warrant, wanted person, and mug shot information for registered drivers of the vehicles. Video tips are received and processed from anonymous and non-anonymous sources. A correlation engine correlates primitive events and compound events from each of the subsystems, weighted by attributes of the events, across both space and time, and an alerting engine generates alerts and performs actions based on the correlation. A hierarchical storage manager manages storage of the vast amounts of data, including video data, based on importance of the data calculated from attributes of the data. A privacy filter ensures no private data is detected, correlated, or stored.

Claims

1. A video surveillance, storage, and alerting system, comprising:

one or more surveillance cameras for capturing video data having attribute data, the attribute data representing importance of the surveillance cameras;

one or more video analytics devices, adapted to process the video data from one or more of the surveillance cameras and to detect primitive video events in the video data;

one or more audio sensory devices for capturing audio data having attribute data, the attribute data representing importance of the audio sensory devices;

one or more audio analytics devices adapted to process the audio data from one or more of the audio sensory devices and to detect audio events in the audio data;

a video tip module for receiving video tips from one or more external sources, adapted to extract meta-data and attribute data from the video tips and to generate tip events based on the extracted meta-data and attribute data, the attribute data representing the importance of the video tips;

a hierarchy of two or more data storage devices for storing the video data from the surveillance cameras, the audio data from the audio sensory devices, and the video tips from the video tip module, the hierarchy of data storage devices connected to the surveillance cameras, the audio sensory devices, and the video tip module via a network;

a hierarchical storage manager for managing storage and cascade of the video data, the audio data, and the video tips in the hierarchy of data storage devices based on the attribute data corresponding to the source of the video data, the audio data, and the video tips;

a network management module for monitoring network status of the surveillance cameras, the audio sensory devices, and the data storage devices, the network management module adapted to generate network events reflective of the network status of all subsystems;

a vehicle information module for retrieving information about a vehicle detected in the video data based on the detected vehicle'"'"'s license plate, and adapted to generate vehicle events based on the information retrieved about the vehicle;

a correlation engine for correlating two or more primitive events, the primitive events being one or more primitive video events from the video analytics devices weighted by the attribute data of the surveillance cameras used to capture the video data, audio events from the audio analytics devices weighted by the attribute data of the audio devices used to capture the audio data, tip events from the video tip module weighted by the extracted attribute data of the video tips, network events from the network management module weighted by attribute data of device corresponding to the network event, and vehicle events from the vehicle information module weighted by the information retrieved about the vehicle; and

an alerting engine for generating one or more alerts and performing one or more actions based on the correlation performed by the correlation engine.

2. The video surveillance, storage, and alerting system of claim 1, further comprising:

a normalization engine for normalizing the primitive events from the video analytics devices, the audio sensory devices, the video tip module, the network management module, and the vehicle information module.

3. The video surveillance, storage, and alerting system of claim 2, wherein the correlation engine further comprises:

a privacy filter for filtering out primitive events normalized by the normalization engine based on a set of privacy rules; and

a business filter for filtering out primitive events normalized by the normalization engine based on a set of business rules.

4. The video surveillance, storage, and alerting system of claim 3, wherein the correlation engine further comprises:

a compound event detection module for detecting compound events composed of two or more primitive events.

5. The video surveillance, storage, and alerting system of claim 4, wherein the correlation engine further comprises:

a first event correlation module for correlating the primitive events and the compound events across time;

a second event correlation module for correlating the primitive events and the compound events across space; and

a rules engine for evaluating one or more rules based on the correlation performed by the first event correlation module and the second event correlation module.

6. The video surveillance, storage, and alerting system of claim 5, further comprising:

a learning engine for generating one or more new rules based on the primitive events correlated by the correlation engine and the alerts generated by the alert engine.

7. The video surveillance, storage, and alerting system of claim 1, wherein the network management module further comprises:

a topological map module for constructing a topological map of the network,

wherein the topological map includes icons for the surveillance cameras, the audio sensory devices, and the data storage devices, and

wherein the icons are connected by lines representing a backbone of the network.

8. The video surveillance, storage, and alerting system of claim 7, wherein the network management module further comprises:

a physical map module for constructing a physical map of the network,

wherein the physical map includes icons corresponding to physical locations of the surveillance cameras, the audio sensory devices, and the data storage devices, and

wherein the physical map includes at least a street map view and a satellite map view.

9. The video surveillance, storage, and alerting system of claim 8,

wherein the icons corresponding to the physical locations of the surveillance cameras have plumes indicating line-of-sight of the surveillance cameras.

10. The video surveillance, storage, and alerting system of claim 9,

wherein the icons and their associated plumes indicate a network state as well as a change of network state of the surveillance cameras as determined by the network management module, and

wherein the physical map shows areas of coverage as well as dark areas indicative of the network state of the surveillance cameras.

11. The video surveillance, storage, and alerting system of claim 8,

wherein the icons corresponding to the physical locations of the audio sensory devices have concentric circles indicating an area of coverage of the audio sensory devices.

12. The video surveillance, storage, and alerting system of claim 1, wherein the hierarchical storage manager queries a sources table database to extract attribute data about sensory devices used to capture data being cascaded.

13. The video surveillance, storage, and alerting system of claim 12,

wherein the hierarchy of data storage devices includes at least a first-tier device and a second-tier device, the first-tier device having a higher data access performance and a lower storage capacity than the second-tier device, and

wherein the hierarchical storage manager cascades the video data from the first-tier device to the second-tier device based at least on importance of the video data.

14. The video surveillance, storage, and alerting system of claim 13, wherein the hierarchical storage manager further comprises:

a rules engine for determining storage locations for segments of video data based on a set of rules based on the importance of the video data; and

a rules update module for updating the set of rules for segments of video data based on historical access patterns.

15. The video surveillance, storage, and alerting system of claim 14,

wherein the importance of the video data is calculated based on attribute data of the surveillance camera used to capture the video data, the primitive events detected in the video data, time period the video data was recorded, and time since the video data was last accessed.

16. The video surveillance, storage, and alerting system of claim 14,

wherein the importance of the video data is calculated as a weighted average of attributes of the video data,

wherein the attributes include resolution of the video data, age of the surveillance camera used to capture the video data, time since the surveillance camera was last maintained, location of the surveillance camera used to capture the video data, and primitive events detected in the video data.

17. The video surveillance, storage, and alerting system of claim 14,

wherein the first-tier device is a disk array and the second-tier device is a tape array.

18. The video surveillance, storage, and alerting system of claim 1, wherein the vehicle information module further comprises:

an automatic license plate recognition engine for recognizing a license plate on the vehicle,

wherein the vehicle information module generates license plate events corresponding to the recognized license plate, and

wherein the vehicle information module retrieves information from a law enforcement database based on the recognized license plate.

19. The video surveillance, storage, and alerting system of claim 18,

wherein the vehicle information module generates warrant events corresponding to warrant information for a registered owner of the vehicle, and

wherein the correlation engine correlates warrant events from the vehicle information module with other primitive events.

20. The video surveillance, storage, and alerting system of claim 19,

wherein the vehicle information module generates wanted person events corresponding to wanted person information for a registered owner of the vehicle, and

wherein the correlation engine correlates wanted person events from the vehicle information module with other primitive events.

21. The video surveillance, storage, and alerting system of claim 20,

wherein the vehicle information module generates stolen plate events if the license plate corresponds to a stolen plate, and

wherein the correlation engine correlates stolen plate events from the vehicle information module with other primitive events.

22. The video surveillance, storage, and alerting system of claim 21,

wherein the vehicle information module returns pictures of a registered owner of the vehicle, and

wherein the alerting engine sends the picture of the registered owner of the vehicle to a designated destination if a wanted person event is triggered for the registered owner of the vehicle.

23. A video surveillance, storage, and alerting system, comprising:

means for capturing video data from one or more surveillance cameras having attribute data representing importance of the video data from the surveillance cameras;

means for capturing audio data from one or more audio sensory devices having attribute data representing importance of the audio data from the audio sensory devices;

means for detecting primitive video events in the video data by performing image processing on the video data;

means for detecting audio events in the audio data by performing audio processing on the audio data;

means for receiving video tips from one or more external sources;

means for extracting tip meta-data and attribute data from the video tips, adapted to generate tip events using the extracted meta-data and attribute data, the attribute data representing importance of the video tips;

means for storing the video data, the audio data, and the video tips in a hierarchy of two or more data storage devices;

means for cascading the video data, the audio data, and the video tips from a first-level storage device to a second-level storage device based at least on importance of data, the first-tier device having a higher data access performance and a lower storage capacity than the second-tier device, the importance of the data based on attribute data of device used to capture the data, primitive events detected in the data, time period the data was recorded, and time since the data was last accessed;

means for monitoring network status of all subsystems, adapted to generate network events indicative of the network status of all subsystems;

means for retrieving information about a vehicle detected in the video data based on the vehicle'"'"'s license plate, adapted to generate vehicle events based on the retrieved vehicle information;

means for correlating two or more primitive events, the primitive events being one or more primitive video events from the video analytics devices weighted by the attribute data of the surveillance cameras used to capture the video data, audio events from the audio analytics devices weighted by the attribute data of the audio devices used to capture the audio data, tip events from the video tip module weighted by the extracted attribute data of the video tips, network events from the network management module weighted by attribute data of device corresponding to the network event, and vehicle events from the vehicle information module weighted by the information retrieved about the vehicle; and

means for activating one or more actions based on the correlation.

24. A method of video surveillance, storage, and alerting, comprising the steps of:

capturing video data from one or more surveillance cameras having attribute data, the attribute data representing importance of the surveillance cameras;

capturing audio data from one or more audio sensory devices having attribute data, the attribute data representing importance of the audio sensory devices;

detecting primitive video events in the video data by performing image processing on the video data;

detecting audio events in the audio data by performing audio processing on the audio data;

receiving video tips from one or more external sources;

generating tip events from meta-data and attribute data extracted from the video tips, the attribute data representing importance of the video tips;

storing the video data, the audio data, and the video tips in a hierarchy of two or more data storage devices;

cascading the video data, the audio data, and the video tips from a first-level storage device to a second-level storage device based at least on importance of data, the first-tier device having a higher data access performance and a lower storage capacity than the second-tier device, the importance of the data based on attribute data corresponding to the source of the video data, the audio data, and the video tips, primitive events detected in the data, time period the data was recorded, and time since the data was last accessed;

generating network events indicative of the network status of all subsystems;

generating vehicle events based on information retrieved about a vehicle detected in the video data using the detected vehicle'"'"'s license plate;

correlating two or more primitive events, the primitive events being one or more primitive video events from the video analytics devices weighted by the attribute data of the surveillance cameras used to capture the video data, audio events from the audio analytics devices weighted by the attribute data of the audio devices used to capture the audio data, tip events from the video tip module weighted by the extracted attribute data of the video tips, network events from the network management module weighted by attribute data of device corresponding to the network event, and vehicle events from the vehicle information module weighted by the information retrieved about the vehicle; and

activating one or more actions based on the correlation performed in the correlating step.

25. The method of claim 24, further comprising:

recognizing a license plate on the detected vehicle;

generating license plate events corresponding to the recognized license plate containing the detected license plate;

retrieving information about the detected vehicle from a law enforcement database based on the recognized license plate;

generating warrant events corresponding to warrant information for a registered owner of the detected vehicle;

generating wanted person events corresponding to wanted person information for the registered owner of the detected vehicle;

generating stolen plate events if the license plate corresponds to a stolen plate;

normalizing the primitive video events, the audio events, the tip events, the network events, and the vehicle events;

filtering out primitive events based on a set of privacy rules;

filtering out primitive events based on a set of business rules;

detecting compound events composed of two or more primitive events, the primitive events being one or more primitive video events, audio events, tip events, network events, vehicle events, license plate events, warrant events, wanted person events, and stolen plate events;

correlating the primitive events and the compound events across time weighted by the associated attribute data;

correlating the primitive events and the compound events across space weighted by the associated attribute data;

evaluating one or more rules based on the correlation performed by the correlating steps; and

activating one or more actions based on the evaluated rules.