Method and system for effectively performing event detection using feature streams of image sequences

• US 8,004,563 B2
• Filed: 07/03/2003
• Issued: 08/23/2011
• Est. Priority Date: 07/05/2002
• Status: Active Grant

• Current Assignee: Aspectus Ltd.
• Original Assignee: Agent Vi
• Inventors: Talmon, Gad, Ashani, Zvi
• Primary Examiner(s): Patel; Jayanti K
• Assistant Examiner(s): TORRENTE, RICHARD T
• Application Number:
  US10/501,949
• Publication Number:
  US 20050036659A1
• Time in Patent Office: 2,973 Days
• Field of Search: 348/154, 348/155, 375/240.08
• US Class Current: 348/155
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19652 Systems using zones in a single scene defined for different treatment, e.g. outer zone gives pre-alarm, inner zone gives alarm
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19667 Details realated to data compression, encryption or encoding, e.g. resolution modes for reducing data volume to lower transmission bandwidth or memory requirements
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Method and system for performing event detection and object tracking in image streams by installing in field, a set of image acquisition devices, where each device includes a local programmable processor for converting the acquired image stream that consist of one or more images, to a digital format, and a local encoder for generating features from the image stream. These features are parameters that are related to attributes of objects in the image stream. The encoder also transmits a feature stream, whenever the motion features exceed a corresponding threshold. Each image acquisition device is connected to a data network through a corresponding data communication channel. An image processing server that determines the threshold and processes the feature stream is also connected to the data network. Whenever the server receives features from a local encoder through its corresponding data communication channel and the data network, the server provides indications regarding events in the image streams by processing the feature stream and transmitting these indications to an operator.

Claims

1. Method for performing event detection and object tracking in an image stream, wherein an event is defined as the occurrence of a type of activity in field which requires some type of action in response thereto, said method comprising:

a) installing in the field, an image acquisition device for acquiring an image stream and comprising a local programmable processor for converting the acquired image stream, consisting of one or more images, to a digital format, and a local encoder, preprogrammed to operate with an image processing server via a data network on the basis of distribution of image processing algorithms between the local encoder and the server, said local encoder being operable to apply low-level feature extraction to the digital format of said image stream, and upon identifying that a number and type of features in the image stream exceed corresponding threshold, selectively generating, from said image stream, a feature stream including parameters related to attributes of areas in said image stream, for transmitting said feature stream via the data network to the image processing server; and

b) connecting said image acquisition device to the data network through a corresponding data communication channel, to enable the operation of the encoder by the image processing server and transmission of the feature stream from the image acquisition device to the image processing server, thereby actuating high-level processing applications at the image processing server applied to the feature stream to detect the event and obtain an indication about the detected event; and

c) transmitting the indications from the image processing server to an operator.

2. Method according to claim 1, wherein the local encoder is a composite encoder, which comprises circuitry for compressing the image stream, and further comprising:

a) operating said composite encoder in a first mode, during which it generates and transmits said feature stream to said image processing server, and

b) operating said composite encoder in a second mode responsive to detecting said events, during which it transmits to said image processing server, in addition to said feature stream, at least a portion of said image stream in a desired compression level, according to commands sent from said image processing server.

3. Method according to claim 2, further comprising recording one or more compressed image streams obtained while said local encoder operates in said second mode.

4. Method according to claim 2, further comprising dynamically allocating additional image processing resources, in the server, to data communication channels that receive image streams.

5. Method according to claim 4, further comprising generating and displaying a graphical polygon that encompasses an object of interest, being within at least one of a frame of an image and an area of interest in said image.

6. Method according to claim 5, further comprising generating and displaying a graphical trace indicating the history of movement of an object of interest, being within at least one of the frame of an image and an area of interest in said image.

7. Method according to claim 1, further comprising decoding one or more compressed image streams containing events by said server, and transmitting the decoded image streams to the display of an operator, for viewing.

8. Method according to claim 1, wherein the image stream is selected from a group of images that comprises video streams, still images, computer generated images, pre-recorded digital video data and analog video data.

9. Method according to claim 1, wherein the image streams are video streams, compressed using MPEG format.

10. Method according to claim 1, wherein the feature stream includes at least one of:

a motion feature;

foreground feature

a color feature;

edge data; and

frequency related information.

11. Method according to claim 1, further comprising performing, by the server, at least one of:

License Plate Recognition (LPR);

Facial Recognition (FR);

detection of traffic rules violations;

behavior recognition;

fire detection;

traffic flow detection; and

smoke detection,

using the feature stream received from the image acquisition device, through the data communication channel.

12. Method according to claim 1 wherein said feature stream further comprises a motion feature, said motion feature being encoded in said feature stream only when said motion feature exceeds a predetermined threshold.

13. System for use in performing event detection and object tracking in image streams, wherein an event is defined as the occurrence of a type of activity in field which requires some type of action in response thereto, said system comprising:

a) an image acquisition device for installation in the field, the image acquisition device comprising;

a.

     1) a local programmable processor for converting the acquired image stream to a digital format; and

a.

     2) a local encoder, preprogrammed to operate with an image processing server via a data network on the basis of distribution of image processing algorithms between the image acquisition device and the server, said local encoder being operable for applying a low-level feature extraction to the image stream and, upon identifying that a number and type of features in the image stream exceed corresponding threshold, selectively generating, from said image stream, a feature stream including parameters related to attributes of areas in said image stream;

b) a data communication channel connecting the image acquisition device to the data network and operable for transmitting the feature stream generated by the local encoder to the image processing server, thereby actuating high-level processing applications at the image processing server to detect the event in the field;

c) a remote image processing server connected to said data network, said remote image processing server receiving said feature stream transmitted by said local encoder; and

d) said remote image processing server applying a high-level processing to the received feature stream and detecting said event from analyzing said feature stream, and transmitting indications regarding said event to an operator.

14. System according to claim 13, wherein said local encoder is a composite encoder and further comprises circuitry for compressing said image stream, said composite encoder having a first operating mode, during which it generates and transmits said feature stream to said remote image processing server, and having a second operating mode, during which said composite encoder transmits to said remote image processing server, in addition to said feature stream, at least a portion of said image stream in a desired compression level, according to commands sent from said remote image processing server.

15. System according to claim 14, further comprising a network video recorder for recording one or more image streams obtained while said composite encoder operates in said second mode.

16. System according to claim 14, in which said portion of said image stream obtained while operating in the second mode, comprises only a portion of the image that corresponds to a desired area of interest.

17. System according to claim 14, further comprising the remote image processing server decoding said portion of said image stream and determining said portion of said image stream contains an event;

and

an operator display, for receiving said portion of said image stream, containing the event from the remote image processing server.

18. System according to claim 17, in which the remote image processing server is capable of dynamically allocating image processing resources to said data communication channel when said data communication channel receives said portion of said image stream.

19. System according to claim 17, in which the remote image processing server further comprises processing means for generating and displaying a graphical polygon that encompasses an object of interest, being within at least one of a frame of an image and an area of interest in said image when said composite encoder is in said second operating mode.

20. System according to claim 19, in which the remote image processing server further comprises processing means for generating and displaying a graphical trace indicating the history of movement of an object of interest, being within at least one of the frame of said image and the area of interest of said image.

21. System according to claim 13, in which the image stream is selected from the group of images that comprises video streams, still images, computer generated images, and pre-recorded digital or analog video data.

22. System according to claim 13, in which the image stream is a video stream compressed using MPEG format.

23. System according to claim 13, in which the feature stream includes at least one of:

a motion feature;

a color feature;

edge data; and

frequency related information.

24. System according to claim 17, in which the remote image processing server further comprises processing means for performing at least one of:

License Plate Recognition (LPR);

Facial Recognition (FR);

detection of traffic rules violations;

behavior recognition;

fire detection;

traffic flow detection; and

smoke detection;

using the feature stream received from the local encoder of the image acquisition device, through the data communication channel.

25. System according to claim 13 wherein said feature stream further comprises a motion feature, said motion feature being encoded in said feature stream only when said motion feature exceeds a predetermined threshold.

26. A distributed image processing method for effectively performing event detection in a large number of concurrent image sequences, wherein an event is defined as the occurrence of a type of activity in fields which requires some type of action in response thereto, said method comprising:

a) performing low level feature extraction by encoders associated with image acquisition devices installed in the fields, said low level feature extraction comprising;

i) receiving image streams from the image acquisition devices installed in the fields and generating digital data indicative thereof;

ii) processing the digital data indicative of the image stream, said processing comprising feature extraction and, upon identifying that a number and type of features in the image stream exceed a corresponding threshold, selectively generating a corresponding feature stream having a bandwidth narrower than that of the corresponding image stream, features in said feature stream including parameters related to attributes of areas in the respective image stream;

b) transmitting said reduced bandwidth feature stream from the image acquisition devices to a common remote image processing server, thereby actuating the image processing server for applying high level image processing to the feature stream and detecting events in the fields, and transmitting indications regarding said events to an operator.

27. The method of claim 26 further comprising transmitting to said remote image processing server at least a portion of an image stream from any of said plurality of image acquisition devices associated with the reduced bandwidth feature stream in which said remote image processing server detects an event.

28. The method of claim 27, wherein said high level image processing comprises:

receiving said reduced bandwidth feature stream transmitted from each of said plurality of image acquisition devices, analyzing each said reduced bandwidth feature stream at said remote image processing server, and detecting events in each said image stream based upon said analyzing of each said reduced bandwidth feature stream by said remote image processing server, and displaying said image stream in which an event is indicated on a display screen of an operator at said remote image processing server.

29. The method of claim 28 wherein said high level image processing comprises performing, by said remote image processing server, at least one of:

License Plate Recognition (LPR);

Facial Recognition (FR);

detection of traffic rules violations;

behavior recognition;

fire detection;

traffic flow detection; and

smoke detection;

using said reduced bandwidth feature stream received from at least one of said plurality of image acquisition devices.

30. The method of claim 26 wherein said selective extraction comprises selectively encoding features in said reduced bandwidth feature stream only when at least one of a number and type of said features exceed a predetermined threshold, wherein said threshold is applied to said image acquisition device to control when said encoder generates and transmits said feature stream, and said number and type of features exceeding said threshold are indicative of activity to be further analyzed by said remote image processing server in order to detect said event.

31. The method of claim 30 wherein said transmitting said reduced bandwidth feature stream to said remote image processing server occurs only when said number and type of features exceed said predetermined threshold.

32. The method of claim 30 wherein said features further comprise motion features, and said motion features are encoded in said reduced bandwidth feature stream only when said motion features exceed said predetermined threshold.

33. The method of claim 30 wherein said predetermined threshold is set by said remote server.

34. The method of claim 30 further comprising transmitting to said remote image processing server at least a portion of an image stream from any of said plurality of image acquisition devices associated with the reduced bandwidth feature stream in which said remote image processing server detects an event.

35. The method of claim 34 further comprising generating and displaying a graphical polygon that encompasses an object of interest within at least one of a frame of said at least a portion of an image and an area of interest in said at least a portion of an image.

36. The method of claim 35 further comprising generating and displaying a graphical trace indicating the history of movement of said object of interest.

37. The method of claim 26 further comprises recording said image stream when said event is detected therein.

38. The method of claim 26 wherein said image stream is selected from a group of images comprising video streams, still images, computer generated images, pre-recorded digital video data and pre-recorded analog video data.

39. The method of claim 26 wherein said features stream comprises at least one of:

a motion feature;

a color feature;

edge data; and

frequency related information.

40. A distributed image processing system for effectively performing event detection in a large number of concurrent image sequences, said distributed image processing system having an in field component and a remote component, wherein an event is defined as the occurrence of a type of activity which requires some type of action in response thereto, said distributed image processing system comprising:

a) a low level feature extraction component located in field, said low level feature extraction component comprising;

a plurality of image acquisition devices installed in field, each producing an image stream;

a processor and an encoder associated with each of said plurality of image acquisition devices;

said processor converting said image stream to a digital format;

said encoder applying a low-level feature extraction to the image stream and upon identifying that a number and type of features in the image stream exceed corresponding threshold selectively generating from the image stream a reduced bandwidth feature stream including parameters related to attributes of areas in said image stream;

b) a remote high level image processing component comprising a remote image processing server;

c) a data network with which said low level feature extraction component communicates with said remote image processing server, each of said plurality of image acquisition devices and associated encoders communicating with said data network through a corresponding data communication channel;

d) said encoder transmitting said reduced bandwidth feature stream to said remote image processing server;

e) said remote image processing server analyzing said reduced bandwidth feature stream and detecting said events associated with said image stream; and

f) wherein, upon detecting said event from the feature stream, said remote image processing server causes at least a portion of the corresponding image stream associated with said events to be transmitted to said remote image processing server.

41. The system of claim 40 wherein said encoder comprises a composite encoder which incorporates said processor, and further comprises circuitry for compressing said image stream, said composite encoder having a first operating mode during which said composite encoder transmits said reduced bandwidth feature stream to said remote image processing server, and a second operating mode during which said composite encoder transmits to said remote image processing server, in addition to said reduced bandwidth feature stream, at least a portion of said image stream in a desired compression level, according to commands sent from said remote image processing server, said remote image processing server controlling said composite encoder to operate in said second operating mode responsive to detecting said events.

42. The system of claim 41 further comprising an operator display for receiving from said remote image processing server said image stream in which said event is detected.

43. The system of claim 42 wherein said remote image processing server further comprises a programmable processor for generating and displaying on said operator display a graphical polygon that encompasses an object of interest within at least one of a frame of said at least a portion of an image and an area of interest in said at least a portion of an image.

44. The system of claim 43 wherein said remote image processing server generates and displays on said operator display a graphical trace indicating a history of movement of said object of interest.

45. The system of claim 41 further comprising a network video recorder associated with each of said plurality of image acquisition devices for recording said image stream when said composite encoder operates in said second operating mode.

46. The system of claim 41 wherein said remote image processing server dynamically allocates additional image processing resources to data communication channels receiving said image stream.

47. The system of claim 40 wherein said features are encoded in said reduced bandwidth feature stream only when at least one of a number and type of said features exceed a predetermined threshold, wherein said threshold is applied to said image acquisition device to control when said encoder generates and transmits said feature stream, and said number and type of features exceeding said threshold are indicative of activity to be further analyzed by said remote image processing server in order to detect said event.

48. The method of claim 47 wherein said reduced bandwidth feature stream is transmitted to said remote image processing server only when said number and type of features exceed said predetermined threshold.

49. The system of claim 47 wherein said features further comprise motion features, and said motion features are encoded in said reduced bandwidth feature stream only when said motion features exceed said predetermined threshold.

50. The system of claim 47 wherein said predetermined threshold is established by said remote image processing server.

51. The system of claim 40 wherein said image stream is selected from a group of images comprising video streams, still images, computer generated images, pre-recorded digital video data and pre-recorded analog video data.

52. The system of claim 40 wherein said features stream comprises at least one of:

a motion feature;

a color feature;

edge data; and

frequency related information.

53. The system of claim 40 wherein said remote image processing server performs at least one of:

License Plate Recognition (LPR);

Facial Recognition (FR);

detection of traffic rules violations;

behavior recognition;

fire detection;

traffic flow detection; and

smoke detection;

using said reduced bandwidth feature stream.

54. Method for use in performing event detection and object tracking in image streams using a system formed by a set of image acquisition devices, each installed in field where an event is to be detected and connected to a central station through a data network, the method comprising:

processing a digital format of an image stream, consisting of one or more images, to detect features, being parameters related to attributes of objects in said image stream, and, based on these features, performing the event detection and object tracking, wherein

providing an image processing server remote from the field, said image processing server being preprogrammed to operate with each of the image acquisition devices on the basis of distribution of image processing algorithms between the image acquisition device and the server, said image processing server being preprogrammed for performing said distribution of image processing algorithms by carrying out the following;

dynamically determining a threshold value for each feature, which is associated with the event to be detected and which is to be extracted by the set of image acquisition devices from image data acquired thereby in the field, and updating each of the image acquisition devices with the corresponding threshold, thereby enabling the image acquisition device to apply low-level feature extraction to the image stream and selectively generate a features stream upon identifying that a number and type of features in the image stream exceed the corresponding threshold, in response to a feature stream, received from each of the image acquisition devices via the network, applying a high-level processing to the received feature stream to identify events in an image stream to which said feature stream corresponds, and controlling an operational mode of the respective image acquisition device, to thereby enable transmission of output data to an operator.

55. A non-transitory computer readable medium for use in performing event detection and extraction and object tracking, the computer readable medium containing instructions which, when executed by a processor, cause the processor to operate with each of multiple remote image acquisition devices via a communication network on the basis of distribution of image processing algorithms between the image acquisition device and said processor, by selectively carrying out the following:

dynamically determining a threshold value for each of a plurality of predetermined features, which are parameters related to attributes of objects in an image stream to be captured by at least one of the remote image acquisition devices, and are associated with the event to be detected and extracted by the respective image acquisition device from image data acquired thereby in the field, and updating each of the image acquisition devices with the corresponding threshold, thereby enabling the image acquisition device to apply low-level feature extraction to the image stream and selectively generate a features stream upon identifying that a number and type of the features in the image stream exceed the corresponding threshold, in response to a feature stream, received from at least one of the image acquisition devices via the network, processing the feature stream by applying a high-level processing to the received feature stream to identify events in an image stream to which said feature stream corresponds, and controlling an operational mode of the respective image acquisition device, to thereby enable transmission of output data to an operator.

56. System for use in performing event detection and object tracking in image streams, the system comprising an image processing server configured for connecting via a data network to a set of image acquisition devices each located in field where the event detection is to be performed, said image processing server being preprogrammed to operate with each of the image acquisition devices on the basis of distribution of image processing algorithms between the image acquisition devices and the image processing server, by determining and providing to each of the image acquisition devices a corresponding threshold for each feature that is associated with the event to be detected and is to be extracted by the image acquisition devices from image data acquired thereby and enabling the image acquisition device to apply low-level feature extraction to at least one of the image streams and selectively generate a feature stream corresponding to the image stream upon identifying that a number and type of the features in the image stream exceed the corresponding threshold, the image processing server being responsive to a feature stream, received from each of the image acquisition devices for applying high-level processing to the received feature stream, to control an operational mode of the respective image acquisition device and to identify event in the image stream corresponding to the feature stream, to thereby enable transmission of output data to an operator.

57. A non-transitory computer program, embodied on a computer readable medium, for use in performing event detection and object tracking in image streams, comprising computer program code means which, when run on a computer causes the computer to:

connect, via a data network, an image processing server to a set of image acquisition devices, each located in field where the event detection is to be performed; and

cause the image processing server to operate with each of the image acquisition devices on the basis of distribution of image processing algorithms between the image acquisition devices and the image processing server by determining and providing to each of the image acquisition devices a corresponding threshold for each feature that is associated with the event to be detected and is to be extracted by the image acquisition devices from at least one of the image streams acquired thereby and enabling the image acquisition device to apply low-level feature extraction to the image stream and selectively generate a feature stream corresponding to one of the image streams upon identifying that a number and type of the features in the image stream exceed the corresponding threshold, the image processing server being responsive to a feature stream, received from each of the image acquisition devices for applying high-level processing to the received feature stream, to control an operational mode of the respective image acquisition device and to identify the event in the image stream corresponding to the feature stream, to thereby enable transmission of output data to an operator.

58. The computer program as claimed in claim 57 embedded on a computer readable medium.