Apparatus, system and method for automated and adaptive digital image/video surveillance for events and configurations using a rich multimedia relational database

• US 20040143602A1
• Filed: 10/17/2003
• Published: 07/22/2004
• Est. Priority Date: 10/18/2002
• Status: Abandoned Application

• Current Assignee: Antonio Ruiz, John Vance Meyers
• Original Assignee: Antonio Ruiz, John Vance Meyers
• Inventors: Ruiz, Antonio, Meyers, John Vance
• Application Number:
  US10/686,578
• Publication Number:
  US 20040143602A1
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G07C 9/37 using biometric data, e.g. fingerprints, iris scans or voice recognition
• CPC Class Codes: G08B 13/122 for a perimeter fence features peculiar to electrified fences A01K3/00
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An automated and adaptive digital image/video and/or sensor surveillance system is provided in a massively and pervasively deployed sensor/image surveillance environment using virtual configuration perimeters for all the subsystems and processes which allow triggered events to be automatically captured by virtual event perimeters in environments where unattended operation and automatic support needs to be provided for real-time event analysis, automatic event tracking, or for storage and retrieval of sensory or visual event information within the scope of the large scale spatio-temporal domain of a target surveillance environment. All operations are performed in the framework of the captured data, information, and knowledge derived through fusion operations and captured in a relational surveillance database subsystem. The information collected and derived knowledge may be used to dynamically create new virtual event perimeters and new virtual configuration perimeters to enable the system to learn and adapt to events as they take place.

Claims

1. A method for operating a surveillance system in a surveillance environment, the method comprising:

providing a plurality of surveillance devices in the surveillance environment for gathering surveillance data, thereby producing gathered data;

establishing a virtual configuration perimeter for the surveillance environment, said virtual configuration perimeter comprising configurable parameters for operating said surveillance devices;

providing a relational database containing information;

establishing a virtual event perimeter comprising at least one event-driven agent that is an object of said gathered data, whereby said gathered data is related to said information in said database for generating an automated response.

2. The method of claim 1 further including the step wherein said virtual event perimeter establishes a new virtual configuration perimeter based upon the operation of said at least one event-driven agent and the relation of said gathered data to said information in said database.

3. The method of claim 2 wherein said step of establishing a virtual configuration perimeter includes the step of establishing a virtual configuration perimeter that comprises profiles comprised of data structures and agents that allow multiple layered processes to be configured and scheduled according to operational characteristics of the surveillance system.

4. The method of claim 1 wherein said automated response includes the step of generating a new virtual event perimeter, said new virtual event perimeter controlling at least one event-driven agent that is different from the original event-driven agent.

5. The method of claim 4 wherein the step of generating said new virtual event perimeter includes the step of generating said new virtual event perimeter recursively so that said new virtual event perimeter may recursively generate additional new virtual event perimeters.

6. The method of claim 1 further including the step of organizing the surveillance system into layers, wherein a physical layer includes physical components of the system, a utility layer includes utility algorithms of the system, an abstraction layer includes abstraction processes of the system, an application layer includes applications of the system, and a management/control layer includes a control means for the system.

7. The method of claim 6 further including the step of including said virtual event perimeter and said virtual configuration perimeter in said management/control layer, whereby said virtual event perimeter and said virtual configuration perimeter may be provided by the management/control layer to the utility layer and the abstraction layer.

8. The method of claim 6 further including the step of providing both off-the-shelf algorithms and system-specific algorithms in said utility layer for performing utility operations on and controlling the gathering of said gathered data by said surveillance devices.

9. The method of claim 6 further including the step of providing processes in said abstraction layer for performing spatio-temporal processing of said gathered data.

10. The method of claim 6 further including the step of providing a graphic user interface in said application layer for interfacing with a user for configuring the surveillance system.

11. The method of claim 10 further including the step of the user operating said graphic user interface to manually configure said virtual event perimeter and said virtual configuration perimeter.

12. The method of claim 6 further including the step of providing a data mining application in said application layer for extracting data from said database for obtaining extracted data and relating said extracted data with said gathered data for producing said automated response.

13. The method of claim 6 further including the step of providing an analysis application in said application layer for performing at least one analysis operation on said gathered data, said analysis operation being chosen from real-time analysis, statistical analysis, and trend analysis.

14. A method for an adaptive surveillance system for automatically responding and adapting to events in a surveillance environment, said method comprising:

disposing at least one surveillance device in the surveillance environment;

operating said at least one surveillance device in accordance with at least one pre-configured profile for gathering surveillance data;

providing operands for examining said surveillance data in comparison with a relational database to extract events; and

reconfiguring at least one of said at least one profiles to adapt said at least one surveillance device in response to said events.

15. The method of claim 14 further including the step of changing said operands in response to said events for extracting additional events.

16. The method of claim 15 wherein the step of changing said operands includes the step of changing said operands recursively so that said operands are able to continually change in response to said events.

17. The method of claim 14 further including the step of organizing the surveillance system into layers, wherein a physical layer includes physical components of the system, a utility layer includes utility algorithms of the system, an abstraction layer includes abstraction processes of the system, an application layer includes applications of the system, and a management/control layer includes a control means for the system.

18. The method of claim 17 further including the step of providing both off-the-shelf algorithms and system-specific algorithms in said utility layer for performing utility operations on and controlling the gathering of said surveillance data by said surveillance devices.

19. The method of claim 17 further including the step of providing processes in said abstraction layer for performing spatio-temporal processing of said surveillance data.

20. The method of claim 17 further including the step of providing a graphic user interface in said application layer for interfacing with a user for configuring the surveillance system.

21. The method of claim 20 further including the step of the user operating said graphic user interface to manually configure said operands and said profiles.

22. The method of claim 17 further including the step of providing a data mining application in said application layer for extracting data from said database for obtaining extracted data and relating said extracted data with said surveillance data for producing an automated response to said event.

23. The method of claim 17 further including the step of providing an analysis application in said application layer for performing at least one analysis operation on said surveillance data, said analysis operation being chosen from real-time analysis, statistical analysis, and trend analysis.

24. An automatically adaptive surveillance system for operating in a surveillance environment, said system comprising:

at least one surveillance device located within the surveillance environment, said surveillance device having controllable operation parameters, said surveillance device further being capable of producing surveillance data;

a controller in communication with said at least one surveillance device for providing pre-configured control operands for controlling said operation parameters of said surveillance device;

a relational database containing information, said database being in communication with said controller; and

said controller further including pre-configured event-detection operands for examining said surveillance data delivered from said surveillance device and comparing said at least one surveillance data with said information in said relational database for determining if an event has occurred, whereby if an event has occurred, said control operands are automatically reconfigured for adapting said at least one surveillance device in response to said event.

25. The system of claim 24, wherein said reconfiguration of said control operands takes place in real-time.

26. The system of claim 24, wherein said pre-configured event-detection operands are reconfigured in response to said event to produce reconfigured event-detection operands.

27. The system of claim 26 wherein said reconfiguration of said pre-configured event-detection operands takes place recursively, so that said reconfigured event-detection operands are capable of further self-reconfiguration.

28. The system of claim 26 wherein said reconfiguration of said event-detection operands takes place in real-time.

29. The system of claim 26 wherein said event-detection operands include a recognition function for recognizing a predetermined characteristic of interest.

30. The system of claim 29 wherein said surveillance data includes digital images, and said recognition function is a recognition application for recognizing features contained in said digital images and comparing said features with said information contained in said database for determining if said digital images contain said predetermined characteristic of interest.

31. The system of claim 30 wherein said recognition application is a facial recognition application for recognizing and identifying the faces of people in the surveillance environment.

32. The system of claim 30 wherein said recognition application is a vehicle license plate recognition application for recognizing and identifying license plates on vehicles in the surveillance environment.

33. The system of claim 30 wherein said recognition application is a human/vehicle interaction recognition application for recognizing and identifying unordinary human/vehicle interaction in the surveillance environment.

34. The system of claim 29 wherein said surveillance data includes digital surveillance sensor data, and said recognition function is a recognition application for recognizing features and patterns contained in said digital surveillance sensor data and comparing said features and patterns with said information contained in said database for determining if said digital surveillance sensor data contains said predetermined characteristic of interest.

35. The system of claim 24 wherein the surveillance system is organized to comprise a physical layer including physical components of the system, a utility layer including utility algorithms of the system, an abstraction layer including abstraction processes of the system, an application layer including applications of the system, and a management/control layer including a control means for the system.

36. The system of claim 24 further including both off-the-shelf and system-specific algorithms for performing utility operations on and controlling the operation of said at least one surveillance device for producing said surveillance data.

37. The system of claim 24 further including processes for performing spatio-temporal processing of said surveillance data.

38. The system of claim 24 further including a graphic user interface for enabling a user to configure the surveillance system.

39. The system of claim 38 wherein the user can operate said graphic user interface to manually configure said pre-configured control operands and said pre-configured event-detection operands.

40. The system of claim 24 further including a data mining application for extracting data from said database for obtaining extracted data and relating said extracted data with said surveillance data for producing an automated response.

41. The method of claim 24 further including an analysis application for performing at least one analysis operation on said surveillance data, said analysis operation being chosen from real-time analysis, statistical analysis, and trend analysis.

42. An automatically adaptive surveillance system for operating in a surveillance environment, said system comprising:

a physical layer including a plurality of surveillance devices for gathering surveillance data from the environment;

a utility layer including algorithms for performing utility operations on and controlling the operation of said surveillance devices;

an abstraction layer including processes for processing the surveillance data gathered by the surveillance devices and determining whether an event has occurred;

an application layer including a graphic user interface for enabling a user to configure the system; and

a management/control layer for automatically controlling and coordinating the operation of the system.

43. The system of claim 42 further including a relational database containing information, said database being in communication with said management/control layer, said management/control layer further including pre-configured control operands provided to said utility layer for controlling said surveillance devices, said management/control layer further including pre-configured event-detection operands provided to said abstraction layer for examining said surveillance data and comparing said surveillance data with said information in said relational database for determining if an event has occurred, whereby if an event has occurred, said control operands are reconfigured for adapting said surveillance devices in response to said event.

44. The system of claim 43 wherein said event-detection operands are reconfigured in response to said event.