Method and apparatus for video frame sequence-based object tracking

• US 9,052,386 B2
• Filed: 07/24/2008
• Issued: 06/09/2015
• Est. Priority Date: 02/06/2002
• Status: Active Grant

• Current Assignee: Qognify Ltd. (PSBU Midco SARL)
• Original Assignee: Nice Systems Limited (Nice Ltd)
• Inventors: Yeredor, Arie, Dvir, Igal, Koren-Blumstein, Guy, Lachover, Boaz
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US12/220,354
• Publication Number:
  US 20090033745A1
• Time in Patent Office: 2,511 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G07C 9/00 Individual registration on entry or exit
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19615 wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

An apparatus and method for the analysis of a sequence of captured images covering a scene for detecting and tracking of moving and static objects (86) and for matching (88) the patterns of object behavior in the captured images to object behavior in predetermined scenarios.

Claims

1. An apparatus for the analysis of a sequence of captured images covering a scene having a background, for detecting and tracking of moving and static objects and for matching the patterns of object behavior in the captured images to object behavior in predetermined scenarios, the apparatus comprising the elements of:

at least one image sequence source for transmitting a sequence of images to an object tracking program; and

an object tracking program for tracking an object irrespective of the aspect ratio thereof, comprising;

a configuration layer for initializing the apparatus in accordance with user pre-defined parameters, comprising a timing parameters definer component that provides a number of time units to be elapsed before a static object is recognized;

a pre-processing application layer for constructing a long term difference image between a currently captured video frame and a long term reference image, the long term reference image showing the background of the scene, and for constructing a short term difference image between the currently captured video frame and a short term reference image, the short term reference image showing the background of the scene and at least one static object within the scene, wherein the short term reference image is defined by having the least one static object as not belonging to the background of the scene and being stationary, that is, not moving, in the scene for a pre-determined period as provided by the timing parameters definer component;

an object clustering application layer for generating at least one new or updated object from the long term difference image and the short term difference image; and

a background updating application layer for updating the long term reference image and the short term reference image with background changes prior to processing of a new frame.

2. The apparatus of claim 1 wherein the configuration application layer comprises a reference image constructor, the reference image constructor comprising a current frame capture module for assigning a captured image as the short term reference image or as the long term reference image.

3. The apparatus of claim 1 wherein the configuration application layer comprises a reference image constructor, the reference image constructor comprising a reference image loading module for loading an existing short term reference image or an existing long term reference image located on file as the reference image.

4. The apparatus of claim 1 wherein the configuration application layer comprises a reference image constructor, the reference image constructor comprising a reference image learning module for generating a reference image from a consecutive sequence of captured images.

5. The apparatus of claim 1 wherein the pre-processing application layer comprises:

a current frame handler for obtaining a captured frame;

a short term reference image handler for loading an existing short-term reference image;

a long term reference image handler loads an existing long-term reference image;

a pre-processor module for generating a new short term and long term reference images;

a short term reference image handler for updating the short term reference image with the new short term reference image; and

a long term reference image handler for updating the long term reference image with the new long term reference image.

6. The apparatus of claim 1 wherein the clustering application layer comprises:

an object merger module for correcting clustering errors by successive merging of at least two partially overlapping objects having the same motion vector for a pre-defined period of time;

an objects group builder module for creating at east one group of at least two close objects;

an object group adjuster module for determining the spatial parameters of each object in the at least one group; and

a new objects constructor module for constructing a new object based on the long term difference image and on the short term difference image.

7. The apparatus of claim 6 wherein the clustering application layer further comprises an object searcher module for locating discarded objects having spatial parameters similar to the parameters of the new object.

8. The apparatus of claim 6, wherein the pixels of a group of overlapping objects are scored based on the respective contribution to the group.

9. The apparatus of claim 1 wherein the objects clustering application layer generates at least one new or updated object from the tong term difference image and the short term difference image and at least one existing object.

10. The apparatus of claim 9 wherein the object tracking program further comprises scene characterization application layer for describing the scene and for triggering an alarm, based on comparing a behavior pattern of the at least one existing object to at least one pre-defined behavior pattern or characteristic.

11. The apparatus of claim 10 wherein the scene characterization application layer comprises an object movement measurement module for analyzing changes in the parameters of the at least one existing object and determining the at least one existing object movement.

12. The apparatus of claim 10 wherein the scene characterization application layer comprises an object merger module for correcting errors the at least one existing object and an alarm triggering mechanism for determining whether an alarm is to be triggered based on the at least one existing object patterns.

13. The apparatus of claim 1 wherein the background updating application layer further comprises a short term reference image updater module and a long term reference image updater module for maintaining the updated short term and long term reference images.

14. The apparatus of claim 1 further comprising an object tracking control database, the database comprising;

at least one long term reference image, the at least one long term reference image comprising a background image of the scene without dynamic or static objects tracked by the apparatus; and

at least one short term reference image, the at least one short term reference image comprising a background image of the scene with the dynamic or static objects tracked by the apparatus.

15. The apparatus of claim 14 wherein the object tracking control database further comprising;

an objects table comprising a list of the dynamic or static objects tracked by the apparatus, each object is associated with object data and object meta data;

a short-term difference image representing a difference between a currently captured frame and a short-term reference image and a long term difference image representing a difference between a currently captured frame and a long-term reference image; and

a background draft comprising a changing image of the scene and making up the short term reference image and the long-term reference image.

16. The apparatus of claim 15 wherein the object tracking control database further comprising a discarded objects archive for storing discarded objects.

17. The apparatus of claim 1, further comprising generation of two sophisticated absolute distance (SAD) maps, corresponding to the long term difference image and short term difference image, respectively, wherein each pixel in the SAD maps is provided with a value reflecting the probability of a map belonging either to an object or the background scene.

18. The apparatus of claim 17, wherein for each specific pixel in a currently captured frame the absolute difference between the corresponding pixel in a reference image is calculated taking account of an average value of each corresponding pixel, thereby obtaining probability of a map belonging either to an object or the background scene.

19. The apparatus of claim 17, wherein for each specific pixel in a currently captured frame the absolute difference between the corresponding pixel in a reference image is calculated based on an average of a pixel and pre-determined neighboring pixels, thereby obtaining probability of a map belonging either to an object or the background scene.

20. The apparatus of claim 1, wherein updating the long term reference image and the short term reference image, respectively, is based on pre-determined groups of pixels in respective temporary (draft) images of as current frame in which there was no motion for a plurality of frames, and wherein, consequently, the updating of each pixel in the short-term reference image is performed in accordance with the values of corresponding pixels in a sophisticated absolute distance (SAD) map and an objects map.

21. The apparatus of claim 20, wherein the score is based on the value of a pixel in the group relative to a plurality of pre-determined threshold values.

22. The apparatus of claim 20, wherein an object in the group is discarded based the scores of pixels thereof, area and overlap ratio with other objects in the group.

23. The apparatus of claim 1, wherein a new object is identified based on adjustable concentric virtual rays that reach a pixel having a pre-determined value of a non-recorded object.

24. The apparatus of claim 1, wherein an object is regarded as a static object based on a counter associated with the object relative to a threshold.

25. The apparatus of claim 1, wherein a false alarm is determined in case an object is moving in a bounding structure for a pre-set number of frames out of another pre-set number of frames.

26. A method for the analysis of a sequence of captured images covering a scene having a background, for detecting and tracking of moving and static objects and for matching the patterns of object behavior in the captured images to object behavior in predetermined scenarios, the method comprising:

transmitting a sequence of images to an object tracking program; and

tracking an object irrespective of the aspect ratio thereof, by an object tracking program that comprises;

a configuration layer for initializing the object tracking program in accordance with user pre-defined parameters, comprising a timing parameters definer component that provides a number of time units to be elapsed before a static object is recognized;

a pre-processing application layer for constructing a long term difference image between a currently captured video frame and a long term reference image, the long term reference image showing the background of the scene, and for constructing a short term difference image between the currently captured video frame and a short term reference image, the short term reference image showing the background of the scene and at least one static object within the scene, wherein the short term reference image is defined by having the least one static object as not belonging to the background of the scene and being stationary, that is, not moving, in the scene for a pre-determined period as provided by the timing parameters definer component;

an object clustering application layer for generating at least one new or updated object from the long term difference image and the short term difference image; and

a background updating application layer for updating the long term reference image and the short term reference image with background changes prior to processing of a new frame.