System and method for detecting a passing vehicle from dynamic background using robust information fusion

• US 7,248,718 B2
• Filed: 02/10/2005
• Issued: 07/24/2007
• Est. Priority Date: 02/19/2004
• Status: Active Grant

• Current Assignee: Siemens Corp. (Siemens AG)
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Zhu, Ying, Comaniciu, Dorin, Kohler, Thorsten, Pellkofer, Martin
• Primary Examiner(s): Werner; Brian
• Assistant Examiner(s): Redding; Thomas M
• Application Number:
  US11/054,996
• Publication Number:
  US 20050196020A1
• Time in Patent Office: 894 Days
• Field of Search: None
• US Class Current: 382/104
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A system and method for detecting a passing vehicle is disclosed. A video sequence comprising a plurality of image frames is received. Image intensity is measured and image motion is estimated in each image frame. A hypothesis model describing background dynamics is formulated. The measured image intensity and motion estimation is used to determine if the background dynamics has been violated in a given image frame. If the background dynamics has been violated, motion coherency is used to determine whether the violation of the background dynamics is caused by a passing vehicle.

Claims

1. A method of detecting a passing vehicle comprising the steps of:

receiving a video sequence comprised of a plurality of image frames;

measuring image intensity and estimating image motion in each image frame;

formulating a hypothesis model describing background dynamics;

using the measured image intensity and motion estimation to determine if the background dynamics has been violated in a given image frame by determining if the difference between the measured image intensity and a predicted image intensity is above a threshold and if the difference between the measured image motion and a predicted image motion of the background is above a threshold; and

if the background dynamics has been violated, using motion coherency to determine whether the violation of the background dynamics is caused by a passing vehicle by performing the following steps;

applying a first subwindow and a second subwindow to the image frame having the violated background dynamics to determine the sequence of motion causing the violation, the motion between the first and second subwindows being measured using robust motion estimation which calculates image flow and their covariances at distributed sample points in an analysis window with multiple image frames;

determining the trajectory of motion of the violated background dynamics by determining which subwindow first indicates a change in image motion; and

if the first subwindow experiences a change in image motion first, determining the violation of background dynamics to be caused by a passing vehicle.

2. The method of claim 1 wherein the hypothesis model describing background dynamics comprises hypothesis on image intensity and hypothesis on image motion.

3. The method of claim 1 wherein the predicted image intensity is determined by the hypothesis model describing background dynamics.

4. The method of claim 1 wherein the predicted image motion of the background is determined by ego-motion and camera parameters.

5. The method of claim 1 wherein the measured image motion is obtained through a robust motion estimation method.

6. The method of claim 1 further comprising the steps of:

determining if the measured motion associated with the violated background dynamics has motion coherency; and

if there is motion coherency, determining that the violation is due to a passing vehicle.

7. The method of claim 1 wherein robust motion estimation is done by exponentially weighting image flows and their covariances at distributed sample points.

8. The method of claim 1 wherein the robust motion estimation is done through joint spatial temporal filtering on image flows at distributed sample points.

9. The method of claim 8 wherein the spatial temporal filtering is done using variable density bandwidth fusion and multiscale mean shift.

10. The method of claim 9 wherein the measured motion in the first and second analysis windows are determined by the most significant sample estimate obtained by variable density bandwidth fusion and multiscale mean shift.

11. The method of claim 1 wherein the video sequence is received from a camera mounted to an ego vehicle.

12. The method of claim 1 wherein trajectory of motion is determined using vehicle speed of the ego vehicle and camera parameters.

13. A system for detecting a passing vehicle comprising:

at least one camera for capturing a video sequence of image frames of background dynamics;

a processor associated with the at least one camera, the processor performing the following steps;

i). measuring image intensity and estimating image motion in each image frame;

ii). formulating a hypothesis model describing the background dynamics;

iii). using the measured image intensity and motion estimation to determine if the background dynamics has been violated in a given image frame by determining if the difference between the measured image intensity and a predicted image intensity is above a threshold and if the difference between the measured image motion and a predicted image motion of the background is above a threshold; and

iv). if the background dynamics has been violated, using motion coherency to determine whether the violation of the background dynamics is caused by a passing vehicle by performing the following steps;

a). applying a first subwindow and a second subwindow to the image frame having the violated background dynamics to determine the sequence of motion causing the violation, the motion between the first and second subwindows being measured using robust motion estimation which calculates image flows and their covariances at distributed sample points in an analysis window with multiple image frames;

b). determining the trajectory of motion of the violated background dynamics by determining which subwindow first indicates a change in image motion; and

c). if the first subwindow experiences a change in image motion first, determining the violation of background dynamics to be caused by a passing vehicle.

14. The system of claim 13 wherein the hypothesis model describing background dynamics comprises hypothesis on image intensity and hypothesis on image motion.

15. The system of claim 13 wherein the predicted image intensity is determined by the hypothesis model describing background dynamics.

16. The system of claim 13 wherein the predicted image motion is determined by ego-motion and camera parameters.

17. The system of claim 13 wherein the measured image motion is obtained through a robust motion estimation method.

18. The system of claim 13 wherein the processor further performs the steps of:

determining if the measured motion associated with the violated background dynamics has motion coherency; and

if there is motion coherency, determining that the violation is due to a passing vehicle.

19. The system of claim 13 wherein robust motion estimation is done by exponentially weighting image flows and their covariances at distributed sample points.

20. The system of claim 13 wherein the robust motion estimation is done through joint spatial temporal filtering on image flows at distributed sample points.

21. The system of claim 20 wherein the spatial temporal filtering is done using variable density bandwidth fusion and multiscale mean shift.

22. The system of claim 21 wherein the measured motion in the first and second analysis windows are determined by the most significant sample estimate obtained by variable density bandwidth fusion and multiscale mean shift.

23. The system of claim 13 wherein the video sequence is received from a camera mounted to an ego vehicle.

24. The system of claim 13 wherein trajectory of motion is determined using vehicle speed of the ego vehicle and camera parameters.