Detection and Tracking of Moving Objects from a Moving Platform in Presence of Strong Parallax

• US 20080273751A1
• Filed: 10/16/2007
• Published: 11/06/2008
• Est. Priority Date: 10/16/2006
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Medioni, Gerard, Cohen, Isaac, Yuan, Chang, Kang, Jinman
• Granted Patent: US 8,073,196 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

Among other things, methods, systems and computer program products are described for detecting and tracking a moving object in a scene. One or more residual pixels are identified from video data. At least two geometric constraints are applied to the identified one or more residual pixels. A disparity of the one or more residual pixels to the applied at least two geometric constraints is calculated. Based on the detected disparity, the one or more residual pixels are classified as belonging to parallax or independent motion and the parallax classified residual pixels are filtered. Further, a moving object is tracked in the video data. Tracking the object includes representing the detected disparity in probabilistic likelihood models. Tracking the object also includes accumulating the probabilistic likelihood models within a number of frames during the parallax filtering. Further, tracking the object includes based on the accumulated probabilistic likelihood models, extracting an optimal path of the moving object.

Claims

1. A computer-implemented method for detecting motion in a sequence of video frames, the method comprising:

identifying residual pixels from the sequence of video frames, to provide estimated indication of a moving object or a static three-dimensional (3-D) structure; and

sorting the identified residual pixels to isolate pixels that correspond to the moving object comprising;

applying at least two geometric constraints to the identified one or more residual pixels, wherein the at least two geometric constraints represent at least a variation in relative depth of the residual pixels;

detecting a disparity of the residual pixels to the applied at least two geometric constraints;

based on the detected disparity, filtering the residual pixels to remove pixels that do not correspond to the moving object;

2. The method of claim 1, wherein identifying the one or more residual pixels comprises performing affine-based motion compensation and detection on the sequence of video frames.

3. The method of claim 1, wherein identifying the one or more residual pixels comprises performing homography based motion detection.

4. The method of claim 1, wherein tracking a moving object comprises using a spatiotemporal JPDAF.

5. The method of claim 1, wherein applying the at least two geometric constraints comprises applying an epipolar constraint including:

generating a first set of two epipolar lines derived from optical flows;

generating a second set of two epipolar lines derived from a fundamental matrix; and

calculating an average angular difference between the first and second sets of epipolar lines to generate an angular difference map.

6. The method of claim 1, wherein, applying the at least two geometric constraints comprises applying a structure consistency constraint including:

generating at least a pair of relative depth maps for the one or more residual pixels;

deriving a bilinear relationship between the generated at least a pair of relative depth maps; and

comparing the one or more residual pixels with the derived bilinear relationship to determine whether the one or more residual pixels conform to the derived bilinear relationship.

7. The method of claim 1, wherein identifying the one or more residual pixels comprises performing affine-based motion compensation and detection that includes:

extracting a number of feature points in each frame of the video data;

matching the feature points in consecutive frames;

estimating a 2D affine motion model;

8. The method of claim 5, wherein extracting a number of feature points is performed by using the Harris corner detector.

9. The method of claim 5, wherein matching the feature points comprises evaluating a cross-correlation of local windows around the feature points.

10. The method of claim 5, wherein estimating a 2D affine motion model comprises fitting the model to at least three pairs of matched points within a RANSAC-based scheme.

11. The method of claim 1, further comprising tracking the moving object in the video data comprising:

representing the detected disparity in a probabilistic likelihood model;

accumulating the probabilistic likelihood model within a number of frames during the filtering; and

based on the accumulated probabilistic likelihood model, extracting an optimal path of the moving object.

17. The computer program product of claim 11, further operable to cause the data processing apparatus to apply a structure consistency constraint comprising:

generating at least a pair of relative depth maps for the one or more residual pixels;

deriving a bilinear relationship between the generated at least a pair of relative depth maps; and

comparing the one or more residual pixels with the derived bilinear relationship to determine whether the one or more residual pixels conform to the derived bilinear relationship.

24. The method of claim 1, further comprising:

performing homography based motion detection to compensate for motion of a camera used to capture the sequence of video frames.

25. The method of claim 24, further comprising:

detecting an angular difference between epipolar lines.

12. A computer program product, embodied on a computer readable medium, operable to cause a data processing apparatus to perform operations comprising:

identify one or more residual pixels from video data;

apply at least two geometric constraints to the identified one or more residual pixels;

detect a disparity of the one or more residual pixels to the applied at least two geometric constraints;

based on the detected disparity, classify the one or more residual pixels as belonging to parallax or independent motion and filtering the parallax classified residual pixels; and

track a moving object in the video data comprising;

representing the detected disparity in probabilistic likelihood models;

accumulating the probabilistic likelihood models within a number of frames during the parallax filtering; and

based on the accumulated probabilistic likelihood models, extracting an optimal path of the moving object.

13. The computer program product of claim 12, further operable to cause the data processing apparatus to identify the one or more residual pixels by performing affine-based motion compensation and detection on the video data.

14. The computer program product of claim 12, further operable to cause the data processing apparatus to identify the one or more residual pixels by performing homography based motion detection.

15. The computer program product of claim 12, further operable to cause the data processing apparatus to track a moving object comprises using a spatiotemporal JPDAF.

16. The computer program product of claim 12, further operable to cause the data processing apparatus to applying the at least two geometric constraints by applying an epipolar constraint including:

generating a first set of two epipolar lines derived from optical flows;

generating a second set of two epipolar lines derived from a fundamental matrix; and

calculating an average angular difference between the first and second sets of epipolar lines to generate an angular difference map.

18. A pipeline processing system comprising:

a geometric constraint estimation stage to perform an estimation of multi-view geometric constraints based on feature point correspondences extracted from a sequence of video frames;

a homography-based image detection stage to compute a background model image, and

identify one or more residual pixels in the sequence of video frames based on inconsistencies with the background model; and

a parallax filtering stage to filter out parallax pixels from the residual pixels by applying one or more outlier detection methods to disparity values with respect to each geometric constraint

19. The system of claim 18, wherein the geometric constraint estimation stage is configured to:

extract or track the feature points along the sequence of video frames;

reduce image noise;

estimate a homography between two consecutive frames in the sequence of video frames; and

estimate a fundamental matrix.

20. The system of claim 18, further comprising linking 2D motion regions obtained from each frame into motion trajectories by a spatio-temporal tracking algorithm.

21. The system of claim 18, wherein the homography-based image detection stage is further configured to detect intensity differences between the background model image and original images are larger than a threshold

22. The system of claim 18, wherein the parallax filtering state is configured to define a likelihood function.

23. A method for detecting motion in a sequence of video frames comprising:

identifying from the sequence of video frames pixels associated with an object in actual motion or a static object in apparent motion; and

processing the identified pixels to isolate the pixels associated with actual motion, wherein the isolating comprises;

detecting a variation in relative depth of each identified pixels;

based on the detected variation, identifying pixels that exceed a level of variation associated with the static structure; and

filtering out those pixels that do not exceed the level of variation associated with the static structure.