Video analysis, archiving and alerting methods and apparatus for a distributed, modular and extensible video surveillance system

• US 7,746,378 B2
• Filed: 10/14/2004
• Issued: 06/29/2010
• Est. Priority Date: 10/12/2004
• Status: Expired due to Fees

• Current Assignee: Nortek Security & Control LLC (Melrose Industries Plc)
• Original Assignee: International Business Machines Corporation
• Inventors: Hampapur, Arun, Lu, Max, Shu, Chiao-Fe
• Primary Examiner(s): An; Shawn
• Application Number:
  US10/967,048
• Publication Number:
  US 20060078047A1
• Time in Patent Office: 2,084 Days
• Field of Search: 348/143, 348/153, 348/159, 348/158, 348/152, 348/154, 375/240, 375/240.01, 375/240.24, 725/105, 725/118, 725/112, 725/108
• US Class Current: 348/143
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Systems and methods are disclosed for analyzing, managing and cataloguing video tracks generated by a video surveillance system. Systems and methods operating in accordance with the present invention receive video tracks from a video surveillance system; analyze them for particular categories of, for example, objects or events, assign unique identifying tags to the video tracks; and save them in highly-cross-referenced databases so that video tracks containing, for example, particular objects or events can be easily searched for and recalled. In addition, the present invention provides browsing facilities for viewing events that have just elapsed, events that are occurring in near real-time.

Claims

1. A video surveillance system comprising:

at least one video surveillance system module comprising;

a video surveillance camera, wherein the video surveillance camera is remotely programmable;

a video encoder to encode video gathered by the video surveillance camera; and

a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera and to create data derived from the video, where the video analysis engine comprises a track creation system to generate at least one track in dependence on at least one user-specified video analysis criterion, wherein the at least one track comprises at least one pointer to a portion of the video gathered by the video surveillance system satisfying the specified video analysis criterion;

at least one video surveillance system network coupled to the at least one video surveillance system module;

at least one database and application system coupled to the at least one video surveillance system network to store video gathered by the video surveillance camera and data created by the video analysis engine, and to receive and process queries for the video and data;

a video surveillance system control interface coupled to the at least one video surveillance system network to control the operations of the video surveillance system; and

a video surveillance application and control interface coupled to the at least one video surveillance system network for controlling video surveillance activities, the video surveillance application and control interface comprising;

a program for controlling video surveillance activities;

a display for displaying video gathered by the video surveillance camera and data created by the video analysis engine; and

at least one input device for creating queries of the database and application system.

2. The video surveillance system of claim 1 wherein:

the video analysis engine further comprises;

a track identification system for assigning a unique identifier to the at least one track.

3. The video surveillance system of claim 2 wherein the track further comprises track summary information.

4. The video surveillance system of claim 3 where the track identification system applies the unique identifier assigned to the at least one track to the track summary information associated with the at least one track.

5. The video surveillance system of claim 4 where the track summary information further comprises a pointer to a video segment.

6. The video surveillance system of claim 4 where the track summary information further comprises a pointer to a keyframe.

7. The video surveillance system of claim 3 wherein the track summary information further comprises timestamp information memorializing a creation time of the track.

8. The video surveillance system of claim 3 wherein the track summary information further comprises object class information specifying an object class.

9. The video surveillance system of claim 8 where the object class corresponds to a vehicle class.

10. The video surveillance system of claim 8 where the object class corresponds to a human class.

11. The video surveillance system of claim 3 where the track summary information further comprises object area information.

12. The video surveillance system of claim 3 where the track summary information further comprises object velocity information.

13. The video surveillance system of claim 3 where the track summary information further comprises activity class information.

14. The video surveillance system of claim 13 where the activity class information corresponds to loitering activity.

15. The video surveillance system of claim 3 where the track summary information is formatted as an extensible markup language (XML) document.

16. The video surveillance system of claim 2 wherein the track further comprises track data information.

17. The video surveillance system of claim 16 where the track identification system applies the unique identifier assigned to the at least one track to the track data information associated with the at least one track.

18. The video surveillance system of claim 16 where the track data information further comprises trajectory information.

19. The video surveillance system of claim 16 where the track data information is formatted as an extensible markup language (XML) document.

20. The video surveillance system of claim 2 wherein the track further comprises background image information.

21. The video surveillance system of claim 20 where the track identification system applies the unique identifier assigned to the at least one track to the background image information associated with the at least one track.

22. The video surveillance system of claim 20 where the background image information is formatted as an extensible markup language (XML) document.

23. The video surveillance system of claim 2 wherein:

the video surveillance application and control interface further comprises;

a track creation criterion specification system for permitting a user to specify the track creation criterion which controls when a track is created.

24. The video surveillance system of claim 2 wherein the video analysis engine performs the following operation:

creating tracks in dependence on the at least one specified criterion, wherein each track is assigned a unique identifier which is used to identify all information related to that specific track.

25. The video surveillance system of claim 24 wherein the at least one database and application system performs the following operation:

saving tracks created when the specified criterion is satisfied using the unique identifiers assigned to each track.

26. The video surveillance system of claim 2 wherein the at least one database and application system further comprises:

a database server to save the at least one track using the unique identifier.

27. The video surveillance system of claim 26 where the video surveillance application and control interface further comprises:

a track query formulation application to formulate queries to locate tracks saved in the database server satisfying a user-specified criterion.

28. The video surveillance system of claim 27 where the database and application system further comprises:

an event retrieval system to accept user-specified track location queries and to locate tracks satisfying the criterion contained in the user-specified queries.

29. The video surveillance system of claim 28 where the database and application system further comprises:

a track identifier forwarding system to forward track identifiers satisfying the user-specified query to the user requesting the track identifiers.

30. The video surveillance system of claim 29 where the video surveillance application and control interface further comprises:

a track information request application to request track information associated with forwarded track identifiers.

31. The video surveillance system of claim 30 where the at least one database and application system further comprises:

a track information retrieval application to retrieve track information associated with track identifiers.

32. The video surveillance system of claim 31 where the at least one database and application system further comprises:

a transformation application for transforming the track information into a desired output format.

33. The video surveillance system of claim 32 where the desired output format is an extensible markup language (XML) document.

34. The video surveillance system of claim 32 where the desired output format is an HTML document.

35. The video surveillance system of claim 31 where the at least one database and application system further comprises:

a track information forwarding system for forwarding track information retrieved from the at least one database and application system to the video surveillance application and control interface.

36. The video surveillance system of claim 30 wherein the track information further comprises track summary information.

37. The video surveillance system of claim 30 wherein the track information further comprises timestamp information memorializing a creation time of the track.

38. The video surveillance system of claim 30 wherein the track information further comprises object class information specifying an object class.

39. The video surveillance system of claim 38 where the object class corresponds to a vehicle class.

40. The video surveillance system of claim 38 where the object class corresponds to a human class.

41. The video surveillance system of claim 30 where the track information further comprises object area information.

42. The video surveillance system of claim 30 where the track information further comprises object velocity information.

43. The video surveillance system of claim 30 where the track information further comprises activity class information.

44. The video surveillance system of claim 43 where the activity class information corresponds to loitering activity.

45. The video surveillance system of claim 30 where the track information further comprises a pointer to a video segment.

46. The video surveillance system of claim 30 where the track information further comprises a pointer to a keyframe.

47. The video surveillance system of claim 30 wherein the track information further comprises track data information.

48. The video surveillance system of claim 30 where the track information further comprises trajectory information.

49. The video surveillance system of claim 30 wherein the track information further comprises background image information.

50. The video surveillance system of claim 2 where the video surveillance system further comprises:

a local storage database associated with the at least one video surveillance system module and coupled to the at least one video surveillance system network for storing data created by the video analysis engine.

51. The video surveillance system of claim 50 further comprising:

an off-peak forwarding application to forward data stored in the local storage database to the at least one database and application system during off-peak periods.

52. The video surveillance system of claim 1 where the at least one database and application system further comprises:

an alert system, wherein the alert system further comprises;

an alert criterion specification system to specify an alert based on data created by the video analysis engine; and

an alert action specification system to specify alert actions to be taken when the alert criterion is satisfied.

53. The video surveillance system of claim 52 where the alert criterion concerns vehicle speed.

54. The video surveillance system of claim 52 where the alert criterion concerns loitering activity.

55. The video surveillance system of claim 52 where the alert action comprises an e-mail issued to a video surveillance analyst.

56. The video surveillance system of claim 52 where the alert action comprises a browser page comprising alert information.

57. The video surveillance system of claim 56 where the alert information comprises a keyframe.

58. The video surveillance system of claim 56 where the alert information comprises track identifiers.

59. The video surveillance system of claim 56 where the alert information comprises a video segment capturing activity that satisfied the alert criterion.

60. The video surveillance system of claim 1 further comprising:

a summary system to create statistical data based on video analysis data generated by the video analysis engine.

61. The video surveillance system of claim 1 where:

the video surveillance camera has a multiple programmable view feature, the video surveillance camera further comprising;

a view programming system for programming views of the video surveillance camera; and

the video surveillance system control interface further comprises;

a video surveillance camera view programming system to program views of the video surveillance camera by interacting with the view programming system of the video surveillance camera.

62. The video surveillance system of claim 61 where the video analysis engine further comprises:

a video surveillance camera view selection system to select a particular view of the video surveillance camera having the multiple programmable view feature and to analyze video associated with the view.

63. The video surveillance system of claim 62 wherein:

the video analysis engine further comprises;

a track identification system for assigning a unique identifier to the track created by the video analysis engine where the unique identifier references at least the selected view of the video surveillance camera.

64. The video surveillance system of claim 63 where the at least one database and application system performs the following operation:

saving the track to a database server of the at least one database and application system.

65. The video surveillance system of claim 64 where the at least one database and application system performs the following operation:

performing a query of the database server for tracks, wherein the query seeks tracks associated with the selected view of the video surveillance camera.

66. The video surveillance system of claim 1 wherein the at least one database and application system performs the following operation:

saving the at least one tracks generated when the at least one user-specified video analysis criterion is satisfied.

67. The video surveillance system of claim 66 wherein the video surveillance system application and control interface performs the following operations:

receiving a query for tracks corresponding to the at least one user-specified video analysis criterion; and

transmitting the query for the tracks to the at least one database and application system.

68. The video surveillance system of claim 67 wherein the at least one database and application system performs the following operations:

receiving the query for the tracks corresponding to the specified criterion; and

locating tracks corresponding to the at least one user-specified video analysis criterion.

69. The video surveillance system of claim 68 wherein the at least one database and application system performs the following operation:

transmitting the tracks satisfying the at least one user-specified video analysis criterion to the video surveillance system application and control interface.

70. The video surveillance system of claim 69 wherein the video surveillance system application and control interface performs the following operation:

displaying the tracks received from the at least one database on the display.

71. The video surveillance system of claim 68 wherein the at least one database and application system performs the following operations:

locating the video corresponding to the tracks; and

transmitting the video satisfying the at least one user-specified video analysis criterion to the video surveillance system application and control interface.

72. The video surveillance system of claim 71 wherein the video surveillance application and control interface performs the following operations:

displaying the video satisfying the at least one user-specified video analysis criterion.

73. The video surveillance system of claim 2 wherein the video surveillance system module further comprises:

a keyframe generation system to create a keyframe of a specific field of view viewed by the video surveillance camera satisfying the specified video analysis criterion.

74. The video surveillance system of claim 73 where the keyframe generation system further comprises:

a keyframe zoom system for creating a keyframe comprising an enlarged view of an object in the field of view of the video surveillance camera.

75. The video surveillance system of claim 73 where the track further comprise a pointer to the keyframe.

76. The video surveillance system of claim 75 wherein

the keyframe generation system performs the following operation:

generating a keyframe satisfying the at least one specified video analysis criterion; and

the video analysis engine performs the following operation;

creating tracks in dependence on the at least one specified video analysis criterion.

77. The video surveillance system of claim 76 wherein the at least one database and application system performs the following operation:

saving the keyframe and track created when the specified criterion is satisfied.

78. The video surveillance system of claim 77 wherein the video surveillance application and control interface performs the following operations:

receiving a query for tracks corresponding to the specified criterion; and

transmitting the query for the tracks to the at least one database and application system.

79. The video surveillance system of claim 78 wherein the at least one database and application system performs the following operations:

receiving the query for tracks corresponding to the specified criterion; and

locating tracks corresponding to the specified criterion.

80. The video surveillance system of claim 79 wherein the at least one database and application system performs the following operations:

locating the keyframe corresponding to the specified criterion by using the tracks; and

transmitting the keyframe satisfying the specified criterion to the video surveillance application and control interface.

81. The video surveillance system of claim 80 wherein the video surveillance application and control interface performs the following operations:

displaying the keyframe satisfying the specified criterion.

82. The video surveillance system of claim 81 wherein the video surveillance application and control interface further comprises:

a keyframe-initiated video request system for requesting video corresponding to the specified criterion when the keyframe is of interest to a user.

83. The video surveillance system of claim 82 wherein the video surveillance application and control interface performs the following operations:

receiving a request for video corresponding to the specified criterion from the keyframe-initiated video request system; and

transmitting the request to the at least one database.

84. The video surveillance system of claim 83 wherein the at least one database and application system performs the following operations:

locating the video corresponding to the specified criterion; and

transmitting the video to the video surveillance application and control interface.

85. The video surveillance system of claim 84 wherein the video surveillance application and control interface performs the following operation:

displaying the video corresponding to the specified criterion on the display.

86. The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an object appearing in the video gathered by the video surveillance camera.

87. The video surveillance system of claim 86 wherein the video analysis engine performs the following operation:

creating class data by identifying and categorizing an object appearing in the video gathered by the video surveillance camera.

88. The video surveillance system of claim 86 wherein the video analysis engine performs the following operation:

creating a track for the object appearing in the video gathered by the video surveillance camera.

89. The video surveillance system of claim 88 wherein the track further comprises time stamp data identifying the time the object appeared in the video gathered by the video surveillance camera.

90. The video surveillance system of claim 88 wherein the track further comprises mask data for specifying locations of the object in frames of the video gathered by the video surveillance camera.

91. The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an event appearing in video gathered by the video surveillance camera.

92. The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an activity appearing in video gathered by the video surveillance camera.

93. The video surveillance system of claim 1 wherein:

the video analysis engine further comprises;

a data formatting system to accept data created by the video analysis engine and to format the data in machine readable data documents.

94. The video surveillance system of claim 93 wherein:

the at least one database and application system further comprises;

a data document transcription system to recover data stored in the machine readable data documents generated by the data formatting system and to convert the data to at least one format used by the database and application system.

95. The video surveillance system of claim 1 wherein the video surveillance system module further comprises:

a video identification system to assign unique identification information to video gathered by the video surveillance camera; and

wherein

the video analysis engine further comprises;

a data association system to associate data created by analyzing the video gathered by the video surveillance camera with the video using the unique identification information.