Apparatus and methods for establishing and managing a distributed, modular and extensible video surveillance system

• US 8,054,330 B2
• Filed: 10/14/2004
• Issued: 11/08/2011
• Est. Priority Date: 10/12/2004
• Status: Active Grant

• Current Assignee: NICE North America LLC
• Original Assignee: International Business Machines Corporation
• Inventors: Hampapur, Arun, Shu, Chiao-Fe
• Primary Examiner(s): Patel; Jay
• Assistant Examiner(s): Pe; Geepy
• Application Number:
  US10/966,630
• Publication Number:
  US 20060077254A1
• Time in Patent Office: 2,581 Days
• Field of Search: 348/143, 348/159, 348/161, 382/103
• US Class Current: 348/143
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Systems and methods are disclosed for implementing and managing distributed video surveillance systems. Systems and methods in accordance with the present invention provide a well-organized structure for describing and implementing distributed video surveillance systems and service protocols for managing the surveillance activities of clusters of video surveillance cameras. In particular, the service protocols provide a high degree of control over the surveillance activities of video surveillance cameras. In addition, the service protocols also provide a great deal of flexibility in selecting and managing a wide range of analytical and managerial tools for available for analyzing and organizing video track information.

Claims

1. A video surveillance system module for use in a distributed and extensible video surveillance system, the video surveillance system module comprising:

a video surveillance camera, wherein the video surveillance camera is remotely programmable;

at least one memory comprising computer readable instructions, and at least one processor configured by execution of the computer readable instructions to cause the module to perform the following;

create a video encoder configured to encode video gathered by the video surveillance camera and to transmit the encoded video to an electronic database system; and

create a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera, wherein the video analysis engine is configured

to be programmable to permit a user to specify video analysis criteria;

to perform activity analysis of the video in accordance with the user-specified video analysis criteria;

to detect and classify objects and object activity observed by the video surveillance camera;

to generate data classifying objects and object activity for use by an entity performing surveillance, the data comprising track information comprising at least a pointer providing access to a segment of video satisfying the user-specified video analysis criteria; and

to transmit the track information, including the pointer providing access to the segment of video satisfying the user-specified video analysis criteria, to the electronic database system.

2. A video surveillance camera cluster comprising:

at least one video surveillance system module, the video surveillance system module comprising;

a video surveillance camera, wherein the video surveillance camera is remotely programmable;

at least one memory comprising computer readable instructions, and at least one processor configured by execution of the computer readable instructions to cause the module to perform the following;

create a video encoder configured to encode video gathered by the video surveillance camera and to transmit the encoded video to an electronic database system; and

create a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera,

wherein the video analysis engine is configured to be programmable to permit a user to specify video analysis criteria;

to perform activity analysis of the video in accordance with the user-specified video analysis criteria;

to detect and classify objects and object activity observed by the video surveillance camera;

to generate data classifying objects and object activity for use by an entity performing surveillance, the data comprising track information comprising at least a pointer providing access to a segment of video satisfying video analysis criteria; and

to transmit the track information, including the pointer providing access to the segment of video satisfying the user-specified video analysis criteria, to the electronic database system.

3. A video surveillance camera cluster for incorporation in a video surveillance camera system, the video surveillance camera cluster comprising:

a unique identifier for use by the video surveillance camera system to identify the video surveillance camera cluster;

at least one video surveillance system module, the video surveillance system module comprising;

a video surveillance camera, wherein the video surveillance camera is remotely programmable;

at least one memory comprising computer readable instructions, and at least one processor configured by execution of the computer readable instructions to cause the module to perform the following;

create a video encoder configured to encode video gathered by the video surveillance camera and to transmit the encoded video to an electronic database system;

create a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera,

wherein the video analysis engine is configured to be programmable to permit a user to specify video analysis criteria;

to perform activity analysis of the video in accordance with the user-specified video analysis criteria;

to detect and classify objects and object activity observed by the video surveillance camera;

to generate data classifying objects and object activity for use by an entity performing surveillance, the data comprising track information comprising at least a pointer providing access to a segment of video satisfying the user-specified criteria; and

to transmit the track information, including the pointer providing access to the segment of video satisfying the user-specified video analysis criteria, to the electronic database system; and

at least one of an output of a camera controller coupled to the video analysis engine and an output of the video encoder coupled to the video analysis engine is coupled to the electronic database system having at least a database server, a video manager server, an application server, and a surveillance application configured to generate an alarm, wherein the electronic database system is configured to display the data and the video in a browser format.

4. The video surveillance camera cluster of claim 3, wherein the video surveillance camera cluster comprises a plurality of video surveillance system modules, and where each camera comprising each video surveillance system module has a unique identifier for use by the video camera surveillance system in identifying the camera.

5. A video surveillance camera system front end comprising:

a video surveillance camera cluster comprising;

at least one video surveillance system module, the video surveillance system module comprising;

a video surveillance camera, wherein the video surveillance camera is remotely programmable;

at least one memory comprising computer readable instructions, and at least one processor configured by execution of the computer readable instructions to cause the module to perform the following;

create a video encoder configured to encode video gathered by the video surveillance camera and to transmit the encoded video to an electronic database system; and

create a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera,

wherein the video analysis engine is configured to be programmable to permit a user to specify video analysis criteria;

to perform activity analysis of the video in accordance with the user-specified video analysis criteria;

to detect and classify objects and object activity observed by the video surveillance camera;

to generate data classifying objects and object activity for use by an entity performing a surveillance function, the data comprising track information comprising at least a pointer providing access to a segment of video satisfying the user-specified video analysis criteria; and

to transmit the track information, including the pointer providing access to the segment of video satisfying the user-specified video analysis criteria, to the electronic database system;

at least one video surveillance system network coupled to the at least one video surveillance system module; and

a remote video surveillance system control interface coupled to the at least one video surveillance system network for controlling the operations of the at least one video surveillance system module.

6. The video surveillance camera system front end of claim 5, where the video surveillance camera has a panning function, and where the remote video surveillance system control interface further comprises a camera panning function programming module for permitting a user to program remotely the panning function of the video surveillance camera.

7. The video surveillance system front end of claim 5, where the video surveillance camera has a tilting function, and where the remote video surveillance system control interface further comprises a camera tilting function programming module for permitting a user to program remotely the tilting function of the video surveillance camera.

8. The video surveillance system front end of claim 5, where the video surveillance camera has a zoom function, and where the remote video surveillance system control interface further comprises a camera zoom function programming module for permitting a user to program remotely the zoom function of the video surveillance camera.

9. The video surveillance system front end of claim 5, where the video surveillance camera has a digital zoom function, and where the remote video surveillance system control interface further comprises a digital zoom function programming module for permitting a user to program remotely the digital zoom function of the video surveillance camera.

10. The video surveillance system front end of claim 5, where the video surveillance camera has a multiple view function, wherein each of the views comprises one or more of operations or movements of the video surveillance camera, and where the remote video surveillance system control interface further comprises a camera view function programming module for permitting a user to program remotely the multiple view function of the video surveillance camera.

11. The video surveillance system front end of claim 10, where the operations of the video surveillance camera comprises at least a zoom operation.

12. The video surveillance system front end of claim 10, where the operations of the video surveillance camera comprises at least a digital zoom operation.

13. The video surveillance system front end of claim 10, where the movements of the video surveillance camera comprises at least a panning movement.

14. The video surveillance system front end of claim 10, where the movements of the video surveillance camera comprises at least a tilting movement.

15. A computer memory medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus of a computer system to perform operations, the operations comprising:

establishing at least one video surveillance camera cluster comprised of at least one video encoder, at least one video analysis engine, and a plurality of video surveillance cameras by assigning unique identifiers to the at least one video surveillance camera cluster and the plurality of video surveillance cameras, wherein the unique identifiers will be used in an electronic database system for managing operations of the at least one video surveillance camera cluster and the plurality of video surveillance cameras;

programming at least one of the plurality of video surveillance cameras in the at least one video surveillance camera cluster;

receiving encoded video data from the at least one video encoder in the at least one video surveillance camera cluster, the encoded video data corresponding to at least one of the plurality of video surveillance cameras in the at least one video surveillance camera cluster;

assigning a unique identifier to the encoded video data gathered by the at least one video surveillance camera and encoded by the corresponding encoder in the at least one video surveillance camera cluster, wherein the unique identifier will be used by the database system to catalogue video data gathered by the corresponding video surveillance camera when saving the video data for later retrieval; and

selecting at least one video analysis operation to be performed on video data gathered by the at least one video surveillance camera in the at least one video surveillance camera cluster, wherein selecting comprises permitting a user to specify video analysis criteria, and wherein the video analysis operation comprises;

performing activity analysis of the video data in accordance with the user-specified video analysis criteria;

identifying and categorizing objects and object activity observed by the at least one video surveillance camera; and

generating data identifying and classifying objects and object activity for use by an entity performing a surveillance function, the data comprising track information comprising at least a pointer providing access to a segment of video data satisfying the user-specified video analysis criteria;

causing the video surveillance camera cluster to perform the operations of performing activity analysis, identifying and categorizing, and generating data; and

causing the video surveillance camera cluster to transmit the track information, including the pointer providing access to the segment of video satisfying the user-specified video analysis criteria, to the computer system;

receiving the track information; and

using at least the track information to display corresponding video to the user.

16. The computer memory medium of claim 15 wherein there are

at least two video surveillance camera clusters, and where the operations further comprise:

listing the unique identifiers of each of the at least two video surveillance camera clusters.

17. The computer memory medium of claim 15 where the operations further comprise:

assigning a description of the properties of the at least one video surveillance camera cluster to the at least one video surveillance camera cluster; and

saving the description in the electronic database system.

18. The computer memory medium of claim 17 where the operations further comprise:

retrieving the description of the properties of the at least one video surveillance camera cluster from the database.

19. The computer memory medium of claim 15 where the operations further comprise:

deleting the at least one video surveillance camera cluster.

20. The computer memory medium of claim 15 where the operations further comprise:

adding a new video surveillance camera to the at least one video surveillance camera cluster by assigning a unique identifier to the new video surveillance camera.

21. The computer memory medium of claim 20 where the operations further comprise:

deleting at least one of the video surveillance cameras from the video surveillance camera cluster.

22. The computer memory medium of claim 15 where the operations further comprise:

assigning a description of the properties of the at least one video surveillance camera comprising the video surveillance camera cluster to that video surveillance camera; and

saving the description in the electronic database system.

23. The computer memory medium of claim 22 where the operations further comprise:

retrieving the description of the properties of the at least one video surveillance camera.

24. The computer memory medium of claim 15 wherein programming the at least one video surveillance camera further comprises:

programming a pan control of the video surveillance camera.

25. The computer memory medium of claim 15 wherein programming the at least one video surveillance camera further comprises:

programming a tilt control of the video surveillance camera.

26. The computer memory medium of claim 15 wherein programming the at least one video surveillance camera further comprises:

programming a zoom control of the video surveillance camera.

27. The computer memory medium of claim 15 wherein programming the at least one video surveillance camera further comprises:

programming a digital zoom control of the video surveillance camera.

28. The computer memory medium of claim 15 wherein programming the at least one video surveillance camera further comprises:

defining at least one view for the at least one video surveillance camera comprising the at least one video surveillance camera cluster, wherein the view corresponds to at least one of programmed operations or movements of the video surveillance camera.

29. The computer memory medium of claim 28 where the operations of the at least one video surveillance camera corresponds to at least a zoom operation.

30. The computer memory medium of claim 28 where the operations of the at least one video surveillance camera corresponds to at least a digital zoom operation.

31. The computer memory medium of claim 28 where the movements of the at least one video surveillance camera corresponds to at least a pan movement.

32. The computer memory medium of claim 28 where the movements of the at least one video surveillance camera corresponds to at least a tilt movement.

33. The computer memory medium of claim 28 wherein programming the at least one video surveillance camera further comprises:

defining multiple views for the at least one video surveillance camera.

34. The computer memory medium of claim 28 where the operations further comprise:

demonstrating the at least one programmed view of the at least one video surveillance camera by displaying real-time feeds of video gathered by the at least one video surveillance camera on a display as the video surveillance camera perform the operations and movements comprising the at least one programmed view.

35. The computer memory medium of claim 15 where the operations further comprise:

starting the at least one selected video analysis operation.

36. The computer memory medium of claim 35 where the operations further comprise:

stopping the at least one selected video analysis operation.

37. The computer memory medium of claim 28 where the operations further comprise:

assigning a unique identifier to the at least one view; and

saving the unique identifier to the electronic database system.

38. The computer memory medium of claim 33 where the operations further comprise:

assigning a unique identifier to each of the multiple views programmed for the at least one video surveillance camera; and

saving the unique identifiers to the electronic database system.

39. The computer memory medium of claim 38 where the operations further comprise:

retrieving the unique identifiers corresponding to the programmed views from the electronic database system.

40. The computer memory medium of claim 28 where the operations further comprise:

assigning a description of the properties of the at least one view for the at least one video surveillance camera to the respective at least one view; and

saving the description of the at least one view to the electronic database system.

41. The computer memory medium of claim 36 where the operations further comprise:

retrieving the description of the at least one view of the at least one video surveillance camera from the electronic database system.

42. The computer memory medium of claim 33 wherein the operations further comprise:

deleting at least one of the multiple programmed views.

43. The computer memory medium of claim 28 wherein the operations further comprise:

updating the at least one view programmed for the at least one video surveillance camera comprising the at least one video surveillance camera cluster by modifying the programmed operations or movements corresponding to the at least one view.

44. The computer memory medium of claim 33 wherein the operations further comprise:

listing the available views for the at least one video surveillance camera.

45. The computer memory medium of claim 15 wherein the operations further comprise:

associating the at least video surveillance camera cluster with address information; and

saving the associated address information to the electronic database system.

46. The computer memory medium of claim 45 wherein the address information comprises campus information.

47. The computer memory medium of claim 45 wherein the address information comprises street address information.

48. The computer memory medium of claim 45 wherein the address information comprises a building name.

49. The computer memory medium of claim 45 wherein the address information comprises city information.

50. The computer memory medium of claim 45 wherein the address information comprises state information.

51. The computer memory medium of claim 45 wherein the address information comprises building entrance information.

52. The computer memory medium of claim 45 wherein the operations further comprise:

deleting the address information.

53. The computer memory medium of claim 15, wherein:

establishing comprises establishing a plurality of video surveillance camera clusters, each cluster comprised of at least one video surveillance camera; and

the video analysis operation is applied to the plurality of video surveillance camera clusters.