Video analysis, archiving and alerting methods and apparatus for a distributed, modular and extensible video surveillance system

• US 20060078047A1
• Filed: 10/14/2004
• Published: 04/13/2006
• Est. Priority Date: 10/12/2004
• Status: Active Grant

• Current Assignee: Nortek Security & Control LLC (Melrose Industries Plc)
• Original Assignee: International Business Machines Corporation
• Inventors: Hampapur, Arun, Lu, Max, Shu, Chiao-Fe
• Granted Patent: US 7,746,378 B2
• Time in Patent Office: Days
• US Class Current: 375/240.10
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Systems and methods are disclosed for analyzing, managing and cataloguing video tracks generated by a video surveillance system. Systems and methods operating in accordance with the present invention receive video tracks from a video surveillance system; analyze them for particular categories of, for example, objects or events, assign unique identifying tags to the video tracks; and save them in highly-cross-referenced databases so that video tracks containing, for example, particular objects or events can be easily searched for and recalled. In addition, the present invention provides browsing facilities for viewing events that have just elapsed, events that are occurring in near real-time.

Claims

1. A video surveillance system comprising:

at least one video surveillance system module comprising;

a video surveillance camera, wherein the video surveillance camera is remotely programmable;

a video encoder to encode video gathered by the video surveillance camera; and

a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera and to create data derived from the video;

at least one video surveillance system network coupled to the at least one video surveillance system module;

at least one database and application system coupled to the at least one video surveillance system network to store video gathered by the video surveillance camera and data created by the video analysis engine, and to receive and process queries for the video and data;

a video surveillance system control interface coupled to the at least one video surveillance system network to control the operations of the video surveillance system; and

a video surveillance application and control interface coupled to the at least one video surveillance system network for controlling video surveillance activities, the video surveillance application and control interface comprising;

a program for controlling video surveillance activities;

a display for displaying video gathered by the video surveillance camera and data created by the video analysis engine; and

at least one input device for creating queries of the database and application system.

2. The video surveillance system of claim 1 wherein:

the video analysis engine further comprises;

a track creation system to generate at least one track in dependence on at least one user-specified video analysis criterion, wherein the at least one track comprises at least a pointer to a portion of the video gathered by the video surveillance system satisfying the specified video analysis criterion; and

a track identification system for assigning a unique identifier to the track.

3. The video surveillance system of claim 2 wherein the track further comprises track summary information.

4. The video surveillance system of claim 3 where track identification system applies the same track identifier assigned to the at least one track to the track summary information associated with the at least one track.

5. The video surveillance system of claim 3 wherein the track summary information further comprises timestamp information memorializing the creation time of the track.

6. The video surveillance system of claim 3 wherein the track summary information further comprises object class information specifying a class of an object.

7. The video surveillance system of claim 6 where the object class corresponds to a vehicle class.

8. The video surveillance system of claim 6 where the object class corresponds to a human class.

9. The video surveillance system of claim 3 where the track summary information further comprises object area information.

10. The video surveillance system of claim 3 where the track summary information further comprises object velocity information.

11. The video surveillance system of claim 3 where the track summary information further comprises activity class information.

12. The video surveillance system of claim 11 where the activity class information corresponds to loitering activity.

13. The video surveillance system of claim 4 where the track summary information further comprises a pointer to a video segment.

14. The video surveillance system of claim 4 where the track summary information further comprises a pointer to a keyframe.

15. The video surveillance system of claim 3 where the track summary information is formatted as an XML document.

16. The video surveillance system of claim 2 wherein the track further comprises track data information.

17. The video surveillance system of claim 16 where track identification system applies the same track identifier assigned to the at least one track to the track data information associated with the at least one track.

18. The video surveillance system of claim 16 where the track data information further comprises trajectory information.

19. The video surveillance system of claim 16 where the track data information is formatted as an XML document.

20. The video surveillance system of claim 2 wherein the track further comprises background image information.

21. The video surveillance system of claim 20 where track identification system applies the same track identifier assigned to the at least one track to the background image information associated with the at least one track.

22. The video surveillance system of claim 20 where the background image information is formatted as an XML document.

23. The video surveillance system of claim 2 wherein:

the video surveillance application and control interface further comprises;

a track creation criterion specification system for permitting a user to specify the track creation criterion which controls when a track is created.

24. The video surveillance system of claim 2 wherein the video analysis engine performs the following operation:

creating tracks in dependence on the at least one specified criterion, wherein each track is assigned a unique identifier which is used to identify all information related to that specific track.

25. The video surveillance system of claim 24 wherein the at least one database and application system performs the following operation:

saving tracks created when the specified criterion is satisfied using the unique identifiers assigned to each track.

26. The video surveillance system of claim 2 wherein the at least one database and application system further comprises:

a database server to save the at least one track using the unique identifier.

27. The video surveillance system of claim 26 where the video surveillance application and control interface further comprises:

a track query formulation application to formulate queries to locate tracks saved in the database server satisfying a user-specified criterion.

28. The video surveillance system of claim 27 where the database and application system further comprises:

an event retrieval system to accept user-specified track location queries and to locate tracks satisfying the criterion contained in the user-specified queries.

29. The video surveillance system of claim 28 where the database and application system further comprises:

a track identifier forwarding system to forward track identifiers satisfying the user-specified query to the user requesting the track identifiers.

30. The video surveillance system of claim 29 where the video surveillance application and control interface further comprises:

a track information request application to request track information associated with forwarded track identifiers.

31. The video surveillance system of claim 30 where the at least one database and application system further comprises:

a track information retrieval application to retrieve track information associated with track identifiers.

32. The video surveillance system of claim 31 where the at least one database and application system further comprises:

a transformation application for transforming the track information into a desired output format.

33. The video surveillance system of claim 32 where the desired output format is an XML document.

34. The video surveillance system of claim 32 where the desired output format is an HTML document

35. The video surveillance system of claim 31 where the at least one database and application system further comprises:

a track information forwarding system for forwarding track information retrieved from the at least one database and application system to the video surveillance application and control interface.

36. The video surveillance system of claim 30 wherein the track information further comprises track summary information.

37. The video surveillance system of claim 30 wherein the track information further comprises timestamp information memorializing the creation time of the track.

38. The video surveillance system of claim 30 wherein the track information further comprises object class information specifying a class of an object.

39. The video surveillance system of claim 38 where the object class corresponds to a vehicle class.

40. The video surveillance system of claim 38 where the object class corresponds to a human class.

41. The video surveillance system of claim 30 where the track information further comprises object area information.

42. The video surveillance system of claim 30 where the track information further comprises object velocity information.

43. The video surveillance system of claim 30 where the track information further comprises activity class information.

44. The video surveillance system of claim 43 where the activity class information corresponds to loitering activity.

45. The video surveillance system of claim 30 where the track information further comprises a pointer to a video segment.

46. The video surveillance system of claim 30 where the track information further comprises a pointer to a keyframe.

47. The video surveillance system of claim 30 wherein the track information further comprises track data information.

48. The video surveillance system of claim 30 where the track information further comprises trajectory information.

49. The video surveillance system of claim 30 wherein the track information further comprises background image information.

50. The video surveillance system of claim 2 where the video surveillance system further comprises:

a local storage database associated with the at least one video surveillance system module and coupled to the at least one video surveillance system network for storing data created by the video analysis engine.

51. The video surveillance system of claim 50 further comprising:

an off-peak forwarding application to forward data stored in the local storage database to the at least one database and application system during off-peak periods.

52. The video surveillance system of claim 1 where the at least one database and application system further comprises:

an alert system, wherein the alert system further comprises;

an alert criterion specification system to specify an alert based on data created by the video analysis engine; and

an alert action specification system to specify alert actions to be taken when the alert criterion is satisfied.

53. The video surveillance system of claim 52 where the alert criterion concerns vehicle speed.

54. The video surveillance system of claim 52 where the alert criterion concerns loitering activity.

55. The video surveillance system of claim 52 where the alert action comprises an e-mail issued to a video surveillance analyst.

56. The video surveillance system of claim 52 where the alert action comprises a browser page comprising alert information.

57. The video surveillance system of claim 56 where the alert information comprises a keyframe.

58. The video surveillance system of claim 56 where the alert information comprises track identifiers.

59. The video surveillance system of claim 56 where the alert information comprises a video segment capturing activity that satisfied the alert criterion.

60. The video surveillance system of claim 1 further comprising:

a summary system to create statistical data based on video analysis data generated by the video analysis engine.

61. The video surveillance system of claim 1 where:

the video surveillance camera has a multiple programmable view feature, the video surveillance camera further comprising;

a view programming system for programming views of the video surveillance camera; and

the video surveillance system control interface further comprises;

a video surveillance camera view programming system to program views of the video surveillance camera by interacting with the view programming system of the video surveillance camera.

62. The video surveillance system of claim 61 where the video analysis engine further comprises:

a video surveillance camera view selection system to select a particular view of the video surveillance camera having the multiple programmable view feature and to analyze video associated with the view.

63. The video surveillance system of claim 62 wherein:

the video analysis engine further comprises;

a track creation system to generate at least one track in dependence on at least one user-specified video analysis criterion, wherein the at least one track comprises at least a pointer to a portion of the video gathered by the selected view of the video surveillance camera which satisfies the specified video analysis criterion; and

a track identification system for assigning a unique identifier to the track created by the video analysis engine where the unique identifier references at least the selected view of the video surveillance camera.

64. The video surveillance system of claim 63 where the at least one database and application system performs the following operation:

saving the track to a database server of the at least one database and application system.

65. The video surveillance system of claim 64 where the at least one database and application system performs the following operation:

performing a query of the database server for tracks, wherein the query seeks tracks associated with the selected view of the video surveillance camera.

66. The video surveillance system of claim 1 wherein:

the video analysis engine further comprises;

a track creation system to generate tracks in dependence on at least one specified video analysis criterion, wherein the tracks comprise at least pointers to portions of the video gathered by the video surveillance system satisfying the specified criterion.

67. The video surveillance system of claim 66 wherein the video analysis engine performs the following operation:

creating tracks in dependence on the at least one specified criterion.

68. The video surveillance system of claim 67 wherein the at least one database and application system performs the following operation:

saving the tracks created when the specified criterion is satisfied.

69. The video surveillance system of claim 68 wherein the video surveillance system application and control interface performs the following operations:

receiving a query for tracks corresponding to the specified criterion; and

transmitting the query for the tracks to the at least one database and application system.

70. The video surveillance system of claim 69 wherein the at least one database and application system performs the following operations:

receiving the query for the tracks corresponding to the specified criterion; and

locating tracks corresponding to the specified criterion.

71. The video surveillance system of claim 70 wherein the at least one database and application system performs the following operation:

transmitting the tracks satisfying the specified criterion to the video surveillance system application and control interface.

72. The video surveillance system of claim 71 wherein the video surveillance system application and control interface performs the following operation:

displaying the tracks received from the at least one database on the display.

73. The video surveillance system of claim 70 wherein the at least one database and application system performs the following operations:

locating the video corresponding to the tracks; and

transmitting the video satisfying the specified criterion to the video surveillance system application and control interface.

74. The video surveillance system of claim 73 wherein the video surveillance application and control interface performs the following operations:

displaying the video satisfying the specified criterion.

75. The video surveillance system of claim 2 wherein the video surveillance system module further comprises:

a keyframe generation system to create a keyframe of a specific field of view viewed by the video surveillance camera satisfying the specified video analysis criterion.

76. The video surveillance system of claim 75 where the keyframe generation system further comprises:

a keyframe zoom system for creating a keyframe comprising an enlarged view of an object in the field of view of the video surveillance camera.

77. The video surveillance system of claim 75 where the track further comprise a pointer to the keyframe.

78. The video surveillance system of claim 77 wherein

the keyframe generation system performs the following operation:

generating a keyframe satisfying the at least one specified video analysis criterion; and

the video analysis engine performs the following operation;

creating tracks in dependence on the at least one specified video analysis criterion.

79. The video surveillance system of claim 78 wherein the at least one database and application system performs the following operation:

saving the keyframe and track created when the specified criterion is satisfied.

80. The video surveillance system of claim 79 wherein the video surveillance application and control interface performs the following operations:

receiving a query for tracks corresponding to the specified criterion; and

transmitting the query for the tracks to the at least one database and application system.

81. The video surveillance system of claim 80 wherein the at least one database and application system performs the following operations:

receiving the query for tracks corresponding to the specified criterion; and

locating tracks corresponding to the specified criterion.

82. The video surveillance system of claim 81 wherein the at least one database and application system performs the following operations:

locating the keyframe corresponding to the specified criterion by using the tracks; and

transmitting the keyframe satisfying the specified criterion to the video surveillance application and control interface.

83. The video surveillance system of claim 82 wherein the video surveillance application and control interface performs the following operations:

displaying the keyframe satisfying the specified criterion.

84. The video surveillance system of claim 83 wherein the video surveillance application and control interface further comprises:

a keyframe-initiated video request system for requesting video corresponding to the specified criterion when the keyframe is of interest to a user.

85. The video surveillance system of claim 84 wherein the video surveillance application and control interface performs the following operations:

receiving a request for video corresponding to the specified criterion from the keyframe-initiated video request system; and

transmitting the request to the at least one database.

86. The video surveillance system of claim 85 wherein the at least one database and application system performs the following operations:

locating the video corresponding to the specified criterion; and

transmitting the video to the video surveillance application and control interface.

87. The video surveillance system of claim 86 wherein the video surveillance application and control interface performs the following operation:

displaying the video corresponding to the specified criterion on the display.

88. The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an object appearing in the video gathered by the video surveillance camera.

89. The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an event appearing in video gathered by the video surveillance camera.

90. The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an activity appearing in video gathered by the video surveillance camera.

91. The video surveillance system of claim 88 wherein the video analysis engine performs the following operation:

creating class data by identifying and categorizing an object appearing in the video gathered by the video surveillance camera.

92. The video surveillance system of claim 88 wherein the video analysis engine performs the following operation:

creating a track for the object appearing in the video gathered by the video surveillance camera.

93. The video surveillance system of claim 92 wherein the track further comprises time stamp data identifying the time the object appeared in the video gathered by the video surveillance camera.

94. The video surveillance system of claim 92 wherein the track further comprises mask data for specifying locations of the object in frames of the video gathered by the video surveillance camera.

95. The video surveillance system of claim 1 wherein:

the video analysis engine further comprises;

a data formatting system to accept data created by the video analysis engine and to format the data in machine readable data documents.

96. The video surveillance system of claim 95 wherein:

the at least one database and application system further comprises;

a data document transcription system to recover data stored in the machine readable data documents generated by the data formatting system and to convert the data to at least one format used by the database and application system.

97. The video surveillance system of claim 1 wherein the video surveillance system module further comprises:

a video identification system to assign unique identification information to video gathered by the video surveillance camera; and

wherein

the video analysis engine further comprises;

a data association system to associate data created by analyzing the video gathered by the video surveillance camera with the video using the unique identification information.

122. The video surveillance system of claim 52 where the alert criterion is specified based on statistics derived from data generated by the video analysis engine.

123. The video surveillance system of claim 61 where the video surveillance system comprises a plurality of video surveillance system modules, wherein each of the video surveillance system modules further comprises a video surveillance camera having the multiple view programming feature, the video surveillance system further comprising:

a cluster specification system to specify a cluster of views, the cluster of views corresponding to at least two views selected from at least two of the video surveillance cameras comprising the video surveillance system modules.

124. The video surveillance system of claim 123 further comprising:

a cluster video analysis criterion specification system to specify a cluster video analysis criterion to be applied to a cluster of views specified with the cluster specification system, wherein the video analysis will be performed by the video analysis engines of the video surveillance system modules.

125. The video surveillance system of claim 124 further comprising:

a cluster alert criterion specification system to specify an alert criterion related to a video analysis criterion applied to video gathered by the video surveillance cameras comprising the cluster of views, wherein the video analysis criterion is specified with the cluster video analysis criterion specification system.

126. The video surveillance system of claim 124 where the video analysis engines perform video analysis based on a cluster video analysis criterion, and generate video analysis data for the cluster of views in dependence on the video analysis criterion.

127. The video surveillance system of claim 126 where the at least one database and application system saves the video analysis data generated by the video analysis engines.

128. The video surveillance system of claim 127 where the at least one database and application system receives queries for video analysis data saved in the at least one database and application system, wherein the video analysis data was generated by the video analysis engines applying the cluster video analysis criterion.

98. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus of a computer system to perform operations comprising management of video surveillance activities using an electronic video surveillance system, the operations comprising:

applying a video analysis application tool to video gathered by a video surveillance camera in real time, wherein the video analysis application tool comprises at least one user-specified video analysis criterion, and where application of the video analysis application tool to the video generates a data stream related to the at least one user-specified video analysis criterion;

transmitting the data stream to an electronic database and application system;

storing information comprising the data stream in the electronic database and application system;

selecting at least a portion of the information comprising the data stream for forwarding to a video surveillance analyst performing video surveillance activities, wherein the selection is made based on at least one user-specified information forwarding criterion; and

forwarding the selected information to the video surveillance analyst for display on an interface device of the electronic video surveillance system.

99. The signal-bearing medium of claim 98 wherein forwarding the selected information occurs in real time.

100. The signal-bearing medium of claim 98 where the user-specified information forwarding criterion comprises an alarm event related to the at least one user-specified video analysis criterion, wherein the alarm event is intended to alert the video surveillance analyst.

101. The signal-bearing medium of claim 100 where the user-specified video analysis criterion comprises vehicle speed and where the user-specified information forwarding criterion comprises an alarm triggered when a speed limit is exceeded by a vehicle observed by the video surveillance camera.

102. The signal-bearing medium of claim 100 where the user-specified video analysis criterion comprises loitering activity and where the user-specified information forwarding criterion comprises an alarm triggered by the occurrence of loitering activity in a field of view observed by the video surveillance camera.

103. The signal-bearing medium of claim 100 where the user-specified video analysis criterion comprises a vehicular accident condition and where the user-specified information forwarding criterion comprises an alarm triggered upon occurrence of a vehicular accident occurring in a field of view monitored by the video surveillance camera.

104. The signal-bearing medium of claim 100 where the user-specified video analysis criterion comprises identification of a specific human being and where the user-specified forwarding criterion comprises an alarm triggered when the specific human being is observed in a field of view monitored by the video surveillance camera.

105. The signal-bearing medium of claim 100 where the user-specified video analysis criterion comprises the detection of a specific activity and where the user-specified forwarding criterion comprises an alarm triggered when the specific activity occurs in a field of view monitored by the video surveillance camera.

106. The signal-bearing medium of claim 98 where the operations further comprise:

assigning unique identifiers to segments of video gathered by the video surveillance camera; and

transmitting the segments to the electronic database and application system for storage.

107. The signal-bearing medium of claim 106 where the data stream comprises at least pointers to video segments satisfying the at least one user-specified video analysis criterion, wherein the pointers comprise video tracks.

108. The signal-bearing medium of claim 107 where the operations further comprise:

forwarding video corresponding to video segments identified by the video track to video surveillance analysts for viewing on a display device of the video surveillance system, wherein the forwarding occurs in dependence on at least one user-specified video forwarding criterion.

109. The signal-bearing medium of claim 108 where forwarding the video corresponding to video segments identified by video track occurs in real time.

110. The signal-bearing medium of claim 108 where the user-specified video forwarding criterion comprises an alarm event related to the at least one user-specified video analysis criterion, wherein the alarm event is intended to alert the video surveillance analyst.

111. The signal-bearing medium of claim 108 where the operations further comprise:

generating at least one keyframe each for at least a portion of the video segments identified by video tracks;

associating each keyframe with its respective video segment through a unique identifier; and

saving the keyframe to the electronic database and application system.

112. The signal-bearing medium of claim 111 where the operations further comprise:

selecting at least one keyframe for forwarding to a video surveillance analyst performing video surveillance activities, wherein the selection is made based on at least one user-specified keyframe forwarding criterion; and

forwarding the selected keyframe to the video surveillance analyst for display on a display device of the electronic video surveillance system.

113. The signal-bearing medium of claim 112 where forwarding the at least one keyframe occurs in real time.

114. The signal-bearing medium of claim 112 where the user-specified keyframe forwarding criterion comprises an alarm event related to the at least one user-specified video analysis criterion, wherein the alarm event is intended to alert the video surveillance analyst.

115. The signal-bearing medium of claim 112 where the operations further comprise:

receiving a request for the video segment corresponding to the keyframe from the video surveillance analyst;

retrieving the video segment corresponding to the keyframe from the electronic database and application system using the track; and

transmitting the video segment to a display device of the video surveillance system for viewing by the video surveillance analyst.

116. The signal-bearing medium of claim 111 where the operations further comprise:

selecting a plurality of keyframes for forwarding to a display device of the video surveillance system for viewing by a video surveillance analyst;

forwarding the plurality of keyframes to the display device; and

simultaneously displaying the plurality of keyframes on the display device.

117. The signal-bearing medium of claim 116 where the operations further comprise:

receiving a request for the video segment corresponding to at least one of the keyframes from the video surveillance analyst;

retrieving the video segment corresponding to the at least one keyframe from the electronic database and application system using the track; and

transmitting the video segment for display on the display device of the video surveillance system for viewing by the video surveillance analyst.

118. The signal-bearing medium of claim 98 where the operations further comprise:

receiving a query for information stored in the electronic database and application system;

retrieving information responsive to the query from the electronic database and application system; and

forwarding the information responsive to the query to a display device of the video surveillance system for viewing by a video surveillance analyst.

119. The signal-bearing medium of claim 107 where the operations further comprise:

receiving a query for information stored in the electronic database and application system, wherein the query comprises at least a request for video satisfying a user-specified selection criterion;

retrieving information responsive to the query from the electronic database and application system, wherein the information comprises video track;

retrieving the video segments corresponding to the video track from the electronic database and application system; and

transmitting the video segments to a display device of the video surveillance system for viewing by a video surveillance analyst.

120. The signal-bearing medium of claim 111 where the operations further comprise:

receiving a query for information stored in the electronic database and application system, wherein the query comprises at least a request for keyframes satisfying a user-specified selection criterion;

retrieving information responsive to the query from the electronic database and application system, wherein the information comprises keyframe identification data;

retrieving the keyframes corresponding to the keyframe identification data from the electronic database and application system; and

transmitting the keyframes to a display device of the video surveillance system for viewing by a video surveillance analyst.

121. The signal-bearing medium of claim 119 wherein the operations further comprise:

receiving a request for video segments corresponding to the keyframes from the video surveillance analyst;

retrieving the video segments corresponding to the keyframes from the electronic database and application system using video tracks; and

transmitting the video segments for display on the display device of the video surveillance system for viewing by the video surveillance analyst.