Method and apparatus for video frame sequence-based object tracking

• US 20090033745A1
• Filed: 07/24/2008
• Published: 02/05/2009
• Est. Priority Date: 02/06/2002
• Status: Active Grant

• Current Assignee: Qognify Ltd. (PSBU Midco SARL)
• Original Assignee: Nice Systems Limited (Nice Ltd)
• Inventors: Yeredor, Arie, Koren-Blumstein, Guy, Lachover, Boaz, Dvir, Igal
• Granted Patent: US 9,052,386 B2
• Time in Patent Office: Days
• US Class Current: 348/152
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G07C 9/00 Individual registration on entry or exit
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19615 wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

An apparatus and method for the analysis of a sequence of captured images covering a scene for detecting and tracking of moving and static objects (86) and for matching (88) the patterns of object behavior in the captured images to object behavior in predetermined scenarios.

Claims

1. An apparatus for the analysis of a sequence of captured images covering a scene having a background, for detecting and tracking of moving and static objects and for matching the patterns of object behavior in the captured images to object behavior in predetermined scenarios, the apparatus comprising the elements of:

at least one image sequence source for transmitting a sequence of images to an object tracking program; and

an object tracking program comprising;

a pre-processing application layer for constructing a long term difference image between a currently captured video frame and a long term reference image, the long term reference image showing the background of the scene, and for constructing a short term difference image between the currently captured video frame and a short term reference image, the short term reference image showing the background of the scene and at least one static object within the scene;

an object clustering application layer for generating at least one new or updated object from the long term difference image and the short term difference image; and

a background updating application layer for updating the long term reference image and the short term reference image with background changes prior to processing of a new frame.

2. The apparatus of claim 1 wherein the object tracking program further comprises a configuration application layer for initializing the apparatus in accordance with user pre-defined parameters.

3. The apparatus of claim 2 wherein the configuration application layer comprises a reference image constructor, the reference image constructor comprising a current frame capture module for assigning a captured image as the short term reference image or as the long term reference image.

4. The apparatus of claim 2 wherein the configuration application layer comprises a reference image constructor, the reference image constructor comprising a reference image loading module for loading an existing short term reference image or an existing long term reference image located on file as the reference image.

5. The apparatus of claim 2 wherein the configuration application layer comprises a reference image constructor, the reference image constructor comprising a reference image learning module for generating a reference image from a consecutive sequence of captured images.

6. The apparatus of claim 2 wherein the configuration application layer comprises a timing parameters definer for providing time setting information.

7. The apparatus of claim 2 wherein the configuration application layer comprises an element of a visual parameters definer for providing the geometry of the scene.

8. The apparatus of claim 7 wherein the visual parameters definer comprises a camera tilt setting module for deriving camera tilt in accordance with measurements of an object located at different locations in the scene.

9. The apparatus of claim 7 wherein the visual parameters definer comprises a camera zoom setting module for defining the maximum, the minimum and the typical size of the objects to be tracked.

10. The apparatus of claim 7 wherein the visual parameters definer comprises a region location definition module for defining the location of at least one region-of-interest within the scene.

11. The apparatus of claim 7 wherein the visual parameters definer comprises a region type definition module for defining a region of interest in the scene.

12. The apparatus of claim 7 wherein the visual parameters definer comprises an alarm type definition module for defining a region of interest as a trigger alarm region.

13. The apparatus of claim 1 wherein the pre-processing application layer comprises:

a current frame handler for obtaining a captured frame;

a short term reference image handler for loading an existing short-term reference image;

a long term reference image handler loads an existing long-term reference image;

a pre-processor module for generating a new short term and long term reference images;

a short term reference image handler for updating the short term reference image with the new short term reference image; and

a long term reference image handler for updating the long term reference image with the new long term reference image.

14. The apparatus of claim 13 wherein the short and long term reference image handlers further provide the moments of the short and long term reference images.

15. The apparatus of claim 1 wherein the clustering application layer comprises:

an object merger module for correcting clustering errors by successive merging of at least two partially overlapping objects having the same motion vector for a pre-defined period of time;

an objects group builder module for creating at least one group of at least two close objects;

an object group adjuster module for determining the spatial parameters of each object in the at least one group; and

a new objects constructor module for constructing a new object based on the long term difference image and on the short term difference image.

16. The apparatus of claim 15 wherein the clustering application layer further comprises an object searcher module for locating discarded objects having spatial parameters similar to the parameters of the new object.

17. The apparatus of claim 15 wherein the clustering application layer further comprises a Kalman filtering module.

18. The apparatus of claim 15 wherein the clustering application layer further comprises an object status updater module for modifying the status of an object.

19. The apparatus of claim 1 wherein the objects clustering application layer generates at least one new or updated object from the long term difference image and the short term difference image and at least one existing object.

20. The apparatus of claim 19 wherein the object tracking program further comprises a scene characterization application layer for describing the scene and for triggering an alarm, based on comparing a behavior pattern of the at least one existing object to at least one pre-defined behavior pattern or characteristic.

21. The apparatus of claim 20 wherein the scene characterization application layer comprises an object movement measurement module for analyzing changes in the parameters of the at least one existing object and determining the at least one existing object movement.

22. The apparatus of claim 20 wherein the scene characterization application layer comprises an object merger module for correcting errors the at least one existing object and an alarm triggering mechanism for determining whether an alarm is to be triggered based on the at least one existing object patterns.

23. The apparatus of claim 1 wherein the background update application layer comprises a background draft updater module for updating the short term reference image and the long term reference image from the currently captured video frame.

24. The apparatus of claim 23 wherein the background update application layer further comprises a short term reference image updater module and a long term reference image updater module for maintaining the updated short term and long term reference images.

25. The apparatus of claim 1 further comprising an object tracking control database, the database comprising;

at least one long term reference image, the at least one long term reference image comprising a background image of the scene without dynamic or static objects tracked by the apparatus; and

at least one short term reference image, the at least one short term reference image comprising a background image of the scene with the dynamic or static objects tracked by the apparatus.

26. The apparatus of claim 25 wherein the object tracking control database further comprising;

an objects table comprising a list of the dynamic or static objects tracked by the apparatus, each object is associated with object data and object meta data;

a short-term difference image representing a difference between a currently captured frame and a short-term reference image and a long term difference image representing a difference between a currently captured frame and a long-term reference image; and

a background draft comprising a changing image of the scene and making up the short term reference image and the long-term reference image.

27. The apparatus of claim 26 wherein the object tracking control database further comprising a discarded objects archive for storing discarded objects.

28. A method for the analysis of a sequence of captured images showing a scene for detecting and tracking at least two separate objects merging or two merged objects separating, the method comprising the steps of:

capturing at least one long term reference image representing the background of the scene, at least one short term reference image representing the background of the scene and at least two static or moving objects within the scene, and at least one current image of the scene;

pre-processing the at least one current image of the scene and generating a long term difference image between the at least one long term reference image and the at least one current image of the scene, and a short term difference image between the short term reference image and the at least one current image of the scene;

clustering the at least two moving or static object from the short term difference image and the long term difference image and generating at least one new object and at least one existing object; and

merging the at least two moving or static objects.

29. A method for the analysis of a sequence of captured images showing a scene for detecting and tracking at least two separate objects merging or two merged objects separating, the method comprising the steps of:

capturing at least one long term reference image representing the background of the scene, at least one short term reference image representing the background of the scene and at least two static or moving objects within the scene, and at least one current image of the scene;

pre-processing the at least one current image of the scene and generating a long term difference image between the at least one long term reference image and the at least one current image of the scene, and a short term difference image between the short term reference image and the at least one current image of the scene;

clustering the at least two moving or static object from the short term difference image and the long term difference image and generating at least one new object and at least one existing object; and

separating the at least two moving or static objects.