ENVIRONMENTALLY AWARE, INTELLIGENT SURVEILLANCE DEVICE

• US 20060007308A1
• Filed: 01/18/2005
• Published: 01/12/2006
• Est. Priority Date: 07/12/2004
• Status: Abandoned Application

• Current Assignee: Siemens Schweiz AG (Siemens AG)
• Original Assignee: Siemens Schweiz AG (Siemens AG)
• Inventors: Jackson, John, Ide, Curtis Evan, McGonnigle, Glenn Matthew
• Application Number:
  US10/905,719
• Publication Number:
  US 20060007308A1
• Time in Patent Office: Days
• US Class Current: 348/143
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19663 Surveillance related processing done local to the camera
• CPC Class Codes: G08B 13/19697 Arrangements wherein non-video detectors generate an alarm themselves
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The present invention relates to methods and systems for the remote monitoring and sensing of a remote deployable, stand-alone, environmentally aware surveillance sensor device or unit that is capable of self-determining its location and orientation relative to a real world, 3D environment, detect conditions or events within the sensor'"'"'s range of detection within that environment, and provide event information indicative of detected conditions or events including their location relative to the 3D real world environment as well as the raw sensor data feed to an external utilization system such as a security monitoring system. The exemplary sensor unit obtains position information from a position information source, orientation information from an orientation information source and time information, thereafter the unit processes the sensor feed to detect objects and types of objects, and provides event information or information output to an external utilization system such as a security monitoring system.

Claims

1. An environmentally aware surveillance device, comprising;

An environmentally aware surveillance device, comprising;

a surveillance sensor for detecting objects within an area under surveillance;

environmental awareness means operative to determine the position and orientation of the surveillance sensor relative to a real world spatial coordinate system, for associating a position of an object detected by the surveillance sensor with respect to the real world spatial coordinate system, and for generating an event information output corresponding to the detected object and associated position; and

an output port for providing said event information output to an external utilization system.

2. The surveillance device of claim 1, further comprising time means for associating time information of the object detected by the surveillance sensor, and wherein the event information output includes the time information.

3. The surveillance device of claim 2, wherein said time means comprises a clock receiver for receiving time information from an external source.

4. The surveillance device of claim 3, wherein said clock receiver is operative to obtain time information from an atomic time clock.

5. The surveillance device of claim 4, wherein said clock receiver is operative to obtain time information from a global positioning system.

6. The surveillance device of claim 1, wherein the self-determination of position and orientation is derived from information provided by a global positioning system satellite and from known physical attributes of the sensor.

7. The surveillance device of claim 6, wherein the known physical attributes of the sensor include position, elevation, angle of view, lens focal length, and inertial measurement.

8. The surveillance device of claim 1, wherein the event information comprises a data feed from the sensor and a mapping of the location of the detected object to a three-dimensional spatial model of the area under surveillance.

9. The surveillance device of claim 1, wherein the sensor device is a video camera.

10. The surveillance device of claim 9, further comprising image stabilization means for stabilizing the raw output of the video camera prior to provision as event information.

11. The surveillance device of claim 9, further comprising image enhancement means for enhancing the output of the video camera prior to provision as event information.

12. The surveillance device of claim 1, wherein the sensor device is operative to convert two-dimensional coordinates of a data array of the sensor into a three-dimensional real world coordinate, and wherein the event information includes the three-dimensional real world coordinate of the detected object.

13. The surveillance device of claim 1, wherein the detected object within the area under surveillance comprises an object that is moving within the field-of-view of the sensor.

14. The surveillance device of claim 13, wherein the object is of a predetermined type of a plurality of object types, each type having an object type identifier, and further comprising the steps of identifying the type of object, assigning an object identifier to the object, and providing the object identifier and the object type identifier as a part of the event information.

15. The surveillance device of claim 1, wherein the surveillance sensor is selected from the group of:

video camera, audio listening device, sonar, radar, seismic, laser, infrared, thermal, and electrical field.

16. A self-contained stand alone environmentally aware surveillance device operative to provide event information as an output, comprising:

a surveillance sensor for provided surveillance signals detected in an area under surveillance;

a time circuit for determining the time of surveillance signals provided by the surveillance sensor;

a position information source for self-detecting the physical location of the surveillance device in a geographic coordinate system;

a sensor orientation information source operative for detecting the relative position of the surveillance sensor with respect to the geographic coordinate system;

a processor responsive to signals from the surveillance sensor, the clock receiver, the position sensor, and the sensor orientation unit for processing such signals and generating event information corresponding to a detected event detected by the sensor, said event information comprising attributes of objects identified by said surveillance signals, time information, and position and orientation information with respect to a detected event in the area under surveillance; and

an output port for providing said event information for external utilization.

17. The surveillance device of claim 16, wherein the processor is operative to determine a three-dimensional location of a detected event in the AUS derived by processing signals from the sensor and mapping such event data to a three-dimensional data model of the area under surveillance and providing the three-dimensional location information as a part of the event information in a three-dimensional coordinate system.

18. The surveillance device of claim 16, wherein the time circuit comprises an atomic clock.

19. The surveillance device of claim 16, wherein the position information source comprises a global positioning system satellite receiver.

20. The surveillance device of claim 16, wherein the processor is further responsive to predetermined information about the known physical attributes of the sensor including position, elevation, angle of view, lens focal length, and inertial measurement, in connection with generating the event information.

21. The surveillance device of claim 16, wherein the event information comprises a data feed from the surveillance sensor and a mapping of the location of the detected event to a three-dimensional spatial model of the area under surveillance.

22. The surveillance device of claim 16, wherein the surveillance sensor is a video camera.

23. The surveillance device of claim 22, further comprising image stabilization means for stabilizing the raw output of the video camera prior to provision as event information.

24. The surveillance device of claim 22, further comprising image enhancement means for enhancing the output of the video camera prior to provision as event information.

25. The surveillance device of claim 16, wherein the surveillance sensor is operative to convert two-dimensional coordinates of a data array representing the output of the surveillance sensor into three-dimensional real world coordinates, and wherein the event information includes the three-dimensional real world coordinate of the detected event.

26. The surveillance device of claim 16, wherein the detected event of the area under surveillance comprises an object that is detected as moving within the range of detection of the sensor.

27. The surveillance device of claim 26, wherein the object is of a predetermined type of a plurality of object types, each type having an object type identifier, and wherein the processor is further operative for executing program steps for identifying the type of object, assigning an object identifier to the object, and providing the object identifier and the object type identifier as a part of the event information.

28. The surveillance device of claim 16, wherein the surveillance sensor is selected from the group of:

video camera, audio listening device, sonar, radar, seismic, laser, infrared, thermal, and electrical field.

29. An environmentally aware video camera, comprising;

a video camera for monitoring an area under surveillance and for providing video output signals;

environmental awareness means operative to self determine the position, orientation, and time of the video camera relative to a real world spatial and temporal coordinate system, and for generating event information corresponding to the position and time of the AUS by the video camera; and

an output port for providing said event information and said video signals to an external utilization system.

30. The environmentally aware video camera of claim 29, further comprising time means for obtaining time information and for associating time information with an object detected by the video camera, and wherein the event information includes the time information.

31. The environmentally aware video camera of claim 30, wherein the time means comprises an atomic clock.

32. The environmentally aware video camera of claim 29, wherein the self-determination of position and orientation is derived from information provided by a global positioning system receiver and from known physical attributes of the video camera.

33. The environmentally aware video camera of claim 29, wherein the known physical attributes of the video camera include position, elevation, angle of view, lens focal length, and orientation measurements.

34. The environmentally aware video camera of claim 29, wherein the event information comprises a data feed from the video camera and a mapping of the location of an object within the field-of-view of the video camera to a three-dimensional spatial model of the area under surveillance.

35. The environmentally aware video camera of claim 34, further comprising image stabilization means for stabilizing the raw output of the video camera prior to provision as event information.

36. The environmentally aware video camera of claim 34, further comprising image enhancement means for enhancing the output of the video camera prior to provision as event information.

37. The environmentally aware video camera of claim 29, further comprising a processor operative for executing program steps for converting two-dimensional coordinates of event data from the video camera into three-dimensional real world coordinates, and wherein the event information include the three-dimensional real world coordinates of a detected object in the AUS within the field of view of the video camera in the AUS.

38. The environmentally aware video camera of claim 37 wherein the detected object comprises an object that is moving within the field-of-view of the video camera.

39. The environmentally aware video camera of claim 38, wherein the object is of a predetermined type of a plurality of object types, each of the object types having an object type identifier, and wherein the process is further operative for executing program steps for identifying the type of object, assigning an object identifier to the object, and providing the object identifier and the object type identifier as a part of the event information.

40. A method for determining the characteristics of an object detected by a surveillance sensor in an area under surveillance, comprising the steps of:

self determining the position, orientation, and time index of signals provided by the surveillance sensor, based on position, orientation, and time inputs or signals, relative to a predetermined real world spatial and temporal coordinate system;

detecting an object within the AUS by the surveillance sensor; and

providing event information to an external utilization system, the event information comprising signals from the surveillance sensor, information corresponding to attributes of the detected object, and position information associated with the detected object relative to the predetermined real world spatial and temporal coordinate system.

41. The method of claim 40, wherein the time index is time information provided by a time input, and further comprising the step of associating the time information with the object detected by the surveillance sensor, and wherein the event information includes the time information.

42. The method of claim 41, wherein the time input is provided by an atomic clock.

43. The method of claim 40, wherein the self-determination of position and orientation is derived from information provided by a global positioning system satellite and from known physical attributes of the sensor.

44. The method of claim 43, wherein the known physical attributes of the sensor include position, elevation, angle of view, lens focal length, and orientation.

45. The method of claim 40, wherein the event information comprises a data feed from the sensor and a mapping of the location of the detected object to a three-dimensional spatial model of the area under surveillance.

46. The method of claim 40, wherein the surveillance sensor is a video camera.

47. The method of claim 46, further comprising image stabilization means for stabilizing the raw output of the video camera prior to provision as event information.

48. The method of claim 46, further comprising image enhancement means for enhancing the output of the video camera prior to provision as event information.

49. The method of claim 40, further comprising the steps of converting two-dimensional coordinates of a data array of the sensor into three-dimensional real world coordinates, and wherein the event information includes the three-dimensional real world coordinates of the detected object.

50. The method of claim 40, wherein the detected object in the AUS comprises an object that is moving within the field-of-view of the sensor.

51. The method of claim 50, wherein the object is of a predetermined type of a plurality of object types, each type having an object type identifier, and further comprising the steps of identifying the type of object, assigning an object identifier to the object, and providing the object identifier and the object type identifier as a part of the event information.

52. The method of claim 40, wherein the surveillance sensor is selected from the group of:

video camera, audio listening device, sonar, radar, seismic, laser, infrared, thermal, and electrical field.

53. The method of claim 40, wherein attributes of the detected object include information selected from the group of object position, object size, object type, object speed, object direction of travel.

54. An environmentally aware sensor for a surveillance system, comprising:

a video sensor for providing video signals detected in the field-of-view of an area under surveillance;

a global positioning system receiver for obtaining position information from global positioning satellites relative to a geographic coordinate system;

an inertial measurement unit for detecting the relative orientation of the video sensor;

a camera lens on the video sensor having predetermined optical characteristics;

a clock receiver for providing time signals;

a computer processor responsive to signals from the video sensor, position signals from the global positioning system receiver, position signals from the inertial measurement unit, time signals from the clock receiver, and predetermined other characteristics of the sensor, for executing predetermined program modules;

a memory for storing predetermined program modules for execution on the processor utilizing said video sensor signals, said global positioning system position signals, said inertial measurement unit position signals, and said time signals;

said processor is operative to compute stored program modules for detecting motion within the field-of-view of the video sensor, detecting an object based on the video signals, tracking the motion of a detected object, classifying the object according to a predetermined classification scheme, and for providing event record data comprising object identification data, object tracking data, object position information, time information, and video signals associated with a detected objected; and

a data communications network interface for providing said event record data to an external utilization system.

55. The environmentally aware sensor of claim 54, wherein said event record data comprises at least information pertaining to the time, date and location of an event.

56. An environmentally aware sensor system for surveillance, comprising:

a video sensor for providing video signals corresponding a field-of-view of an area under surveillance;

a computer processor;

a position information input for receiving position signals indicative of the location of the system with respect to a real world coordinate system;

an orientation information input for receiving orientation signals indicative of the orientation of the video sensor relative to the real world coordinate system;

program modules operative to execute on said computer processor including;

a video processing module for detecting motion of a region within the field-of-view of the sensor;

a tracking module for determining a path of motion of said region within the field-of-view of the sensor;

a behavioral awareness module for identifying predetermined behaviors of said region within the field-of-view of the sensor;

an environmental awareness module responsive to predetermined information relating to characteristics of the video sensor, said position signals, and said orientation signals, and outputs from said video processing module, said tracking module, and said behavioral awareness module, for computing geometric equations and mapping algorithms, and for providing video frame output and event record output indicative of predetermined detected conditions to an external utilization system.

57. The sensor system of claim 56, further comprising a time signal input for receiving time signals, and wherein said program modules are responsive to said time signals for associating a time with said event record output.

58. The sensor system of claim 56, wherein the program modules include an object detection module responsive to said video signals for detecting an object having predetermined characteristics and for providing an object identifier corresponding to a detected object for utilization by other program modules within the sensor system.

59. The sensor system of claim 58, wherein the tracking module is operative to determine a path of motion of a detected object detected by said object detection module.

60. The sensor system of claim 58, wherein the program modules include an object classification module responsive to said video signals or said object identifier for identifying an object having predetermined characteristics and for providing an object type identifier corresponding to a detected type of object for utilization by other program modules within the sensor system.

61. The sensor system of claim 58, wherein the behavioral awareness module is operative for identifying predetermined behaviors of a detected and tracked object based on information provided by the object detection module and the tracking module.

62. The sensor system of claim 56, wherein the program modules include an image stabilization module responsive to said video signals for stabilizing an image provided as an output by the sensor system.

63. The sensor system of claim 56, wherein the characteristics of the video sensor include characteristics of the lens associated with the video sensor.

64. The sensor system of claim 56, wherein the event record output provided by the environmental awareness module includes information for mapping (associating) the two-dimensional location of said detected region to a three dimensional location within a real world coordinate system.

65. A method for providing object information from a sensor in a security monitoring environment for utilization by a security monitoring system, comprising these steps of:

placing an environmentally aware sensor in an area under surveillance, the sensor having a range of detection and predetermined sensor characteristics, and inputs for receipt of position information from a position information source and orientation information from an orientation information source;

at the environmentally aware sensor, self determining the location of the sensor relative to a three-dimensional real world coordinate system based on the position information and the orientation information;

determining the three-dimensional coordinates of points within the area within the range of detection of the EA sensor based on the predetermined sensor characteristics and the determined location of the sensor;

detecting an object within the range of detection of the environmentally aware sensor;

determining the three-dimensional location of the detected object within the range of detection of the environmentally aware sensor; and

providing the location of the detected object, identifying information relating to the detected object, and a data feed from the environmentally aware sensor to the external security monitoring system.

66. The method of claim 65, wherein the EA sensor is a video camera.

67. The method of claim 66, wherein the range of detection is the field-of-view of the video camera.

68. The method of claim 66, wherein the data feed from the environmentally aware sensor is a video feed.

69. The method of claim 66, wherein the predetermined sensor characteristics include optical characteristics of a lens on the video camera, and information about the mounting height of the video camera.

70. The method of claim 66, further comprising the step of applying image processing algorithm to the data feed from the video camera so as to identify an object within the range of detection of the video camera as being of a predetermined type of object, assign an object identifier to a detected object, and provide object type information as a part of the identifying information about the detected object.

71. The method of claim 65, further comprising the step of providing geographic information system database information to the environmentally aware sensor for utilization in determining the three-dimensional location of the environmentally aware sensor.

72. The method of claim 65, wherein the orientation information source is an inertial measurement unit coupled to the environmentally aware sensor that detects the movements of the sensor so as to determine the limits of the range of detection of the sensor relative to the determined position of the sensor.