System and method for on-road detection of a vehicle using knowledge fusion

• US 7,639,841 B2
• Filed: 12/19/2005
• Issued: 12/29/2009
• Est. Priority Date: 12/20/2004
• Status: Active Grant

• Current Assignee: VDO Automotive AG (Continental AG), Siemens Corp. (Siemens AG)
• Original Assignee: Siemens VDO Automotive AG (Continental AG), Siemens Corp. (Siemens AG)
• Inventors: Zhu, Ying, Comaniciu, Dorin, Pellkofer, Martin, Köhler, Thorsten
• Primary Examiner(s): Mehta; Bhavesh M
• Assistant Examiner(s): MACKOWEY, ANTHONY M
• Application Number:
  US11/311,500
• Publication Number:
  US 20060177099A1
• Time in Patent Office: 1,471 Days
• Field of Search: 382/103, 382/104, 382/107, 382155-160, 382/203, 382/209, 382214-220, 382224-228, 348/113, 348/116, 348/117, 348/118, 348/119, 348/148, 348/149, 701/23, 701/26, 701/27, 701/28, 701/41, 701/44, 701/45, 701/48, 701/70, 701/77, 701/300, 701/301, 340/435, 340/436
• US Class Current: 382/104
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention is directed to a system and method for on-road vehicle detection. A video sequence is received that is comprised of a plurality of image frames. A potential vehicle appearance is identified in an image frame. Known vehicle appearance information and scene geometry information are used to formulate initial hypotheses about vehicle appearance. The potential vehicle appearance is tracked over multiple successive image frames. Potential motion trajectories for the potential vehicle appearance are identified over the multiple image frames. Knowledge fusion of appearance, scene geometry and motion information models are applied to each image frame containing the trajectories. A confidence score is calculated for each trajectory. A trajectory with a high confidence score is determined to represent a vehicle appearance.

Claims

1. A method for on-road vehicle detection comprising the steps of:

receiving a video sequence comprised of a plurality of image frames;

identifying a potential vehicle appearance in an image frame;

using known vehicle appearance information and scene geometry information to formulate initial hypotheses about vehicle appearance;

tracking the potential vehicle appearance over multiple successive image frames;

identifying potential motion trajectories for the potential vehicle appearance over the multiple image frames;

applying knowledge fusion by assigning a confidence score to each trajectory of vehicle appearance, where the confidence score is defined as the product of probabilities obtained from appearance model, scene geometry model and motion model determined for each image frame containing the trajectories; and

determining that a trajectory with a high confidence score represents a vehicle appearance.

2. The method of claim 1 wherein a potential vehicle appearance is determined by hypothesis testing using a probability model.

3. The method of claim 2 wherein the probability model is defined as the probability that a vehicle appears in an observed image patch.

4. The method of claim 2 wherein the probability model is obtained from known vehicle and non-vehicle training samples.

5. The method of claim 2 wherein the probability model is formed by using a set of image features as weak classifiers that characterize various vehicle aspects.

6. The method of claim 1 wherein the scene geometry models impose strong constraints on a location of a vehicle in a given image frame.

7. The method of claim 6 wherein the scene geometry model is a probability model defined as the joint probability distribution function of vehicle location and vehicle size inside an image frame.

8. The method of claim 7 wherein the joint probability distribution function of vehicle location and vehicle size in a given image frame is obtained as the product of probability distribution function of vehicle location in a give image frame and conditional probability distribution function of vehicle size given its location in a given image frame.

9. The method of claim 8 wherein the probability distribution function of vehicle location in a give image frame is a uniform distribution.

10. The method of claim 7 wherein the vehicle size in a given image frame is a function of vehicle location in a given image frame, vehicle size in the world coordinate system, and elevation of the road surface where the vehicle touches the road.

11. The method of claim 10 wherein given a vehicle location in a given image frame, the vehicle size in the world coordinate system is described as a random variable with a normal distribution, and elevation of the road surface is described as a random variable with a normal distribution.

12. The method of claim 8 wherein the conditional probability distribution function of vehicle size given its location in a given image frame is defined as a normal distribution.

13. The method of claim 12 wherein mean and covariance for the normal distribution are derived from mean and covariance of vehicle size in the world coordinate system and mean and covariance of elevation of the road surface where the vehicle touches the road.

14. The method of claim 1 wherein the motion models impose constraints on the movement of the potential vehicle appearance in a subsequent frame.

15. The method of claim 14 wherein the motion model is defined as conditional probability distribution function of vehicle location and vehicle size in an image frame given its location and size in the previous image frame and the two consecutive image frames.

16. The method of claim 15 wherein the conditional probability of the conditional probability distribution function of vehicle location and vehicle size is a normal distribution.

17. The method of claim 16 wherein mean of the normal distribution is an unbiased estimate of vehicle location and size given its location and size in a previous image frame and two consecutive image frames.

18. The method of claim 16 wherein covariance of the normal distribution is the covariance of an unbiased estimate of the vehicle location and size given its location and size in a previous image frame and two consecutive image frames.

19. The method of claim 1 wherein the video sequence is received from a camera mounted to a vehicle.

20. The method of claim 1 wherein trajectories of vehicle appearance are extended into each subsequent image frames using estimated image motion of the vehicle appearance.

21. The method of claim 1 wherein the step of applying knowledge fusion is applied to trajectories of vehicle appearance in each image frame of the video sequence.

22. The method of claim 1 wherein the appearance, scene geometry and motion information models are used to update the confidence score for each trajectory in each subsequent image frame in the video sequence.

23. The method of claim 1 wherein if the confidence score for a particular trajectory falls below a predetermined value, the particular trajectory is not tracked in subsequent image frames.

24. A system for on-road vehicle detection comprises:

at least one camera for capturing a video sequence of image frames of background dynamics;

a processor associated with the at least one camera, the processor performing the following steps;

i). identifying a potential vehicle appearance in an image frame;

ii). using known vehicle appearance information and scene geometry information to formulate initial hypotheses about vehicle appearance;

iii). tracking the potential vehicle appearance over multiple successive image frames;

iv). identifying potential motion trajectories for the potential vehicle appearance over the multiple image frames;

v). applying knowledge fusion by assigning a confidence score to each trajectory of vehicle appearance, where the confidence score is defined as the product of probabilities obtained from appearance model, scene geometry model and motion model determined for each image frame containing the trajectories;

and

vi). determining that a trajectory with a high confidence score represents a vehicle appearance.

25. The system of claim 24 wherein a potential vehicle appearance is determined by hypothesis testing using a probability model.

26. The system of claim 25 wherein the probability model is defined as the probability that a vehicle appears in an observed image patch.

27. The system of claim 25 wherein the probability model is obtained from known vehicle and non-vehicle training samples.

28. The system of claim 25 wherein the probability model is formed by using a set of image features as weak classifiers that characterize various vehicle aspects.

29. The system of claim 24 wherein the scene geometry models impose strong constraints on a location of a vehicle in a given image frame.

30. The system of claim 29 wherein the scene geometry model is a probability model defined as the joint probability distribution function of vehicle location and vehicle size inside an image frame.

31. The system of claim 30 wherein the joint probability distribution function of vehicle location and vehicle size in a given image frame is obtained as the product of probability distribution function of vehicle location in a give image frame and conditional probability distribution function of vehicle size given its location in a given image frame.

32. The system of claim 31 wherein the probability distribution function of vehicle location in a give image frame is a uniform distribution.

33. The system of claim 31 wherein the vehicle size in a given image frame is a function of vehicle location in a given image frame, vehicle size in the world coordinate system, and elevation of the road surface where the vehicle touches the road.

34. The system of claim 33 wherein given vehicle location in a given image frame, the vehicle size in the world coordinate system is described as a random variable with a normal distribution, and elevation of the road surface is described as a random variable with a normal distribution.

35. The system of claim 31 wherein the conditional probability distribution function of vehicle size given its location in a given image frame is defined as a normal distribution.

36. The system of claim 35 wherein mean and covariance for the normal distribution are derived from mean and covariance of vehicle size in the world coordinate system and mean and covariance of elevation of the road surface where the vehicle touches the road.

37. The system of claim 24 wherein the motion models impose constraints on the movement of the potential vehicle appearance in a subsequent frame.

38. The system of claim 37 wherein the motion model is defined as conditional probability distribution function of vehicle location and vehicle size in an image frame given its location and size in the previous image frame and the two consecutive image frames.

39. The system of claim 38 wherein the conditional probability distribution function of vehicle location and vehicle size is a normal distribution.

40. The system of claim 39 wherein mean of the normal distribution is an unbiased estimate of vehicle location and size given its location and size in a previous image frame and two consecutive image frames.

41. The system of claim 39 wherein covariance of the normal distribution is the covariance of an unbiased estimate of the vehicle location and size given its location and size in a previous image frame and two consecutive image frames.

42. The system of claim 24 the at least one camera is mounted to a vehicle.

43. The system of claim 24 wherein trajectories of vehicle appearance are extended into each subsequent image frames using estimated image motion of the vehicle appearance.

44. The system of claim 24 wherein the step of applying knowledge fusion is applied to trajectories of vehicle appearance in each image frame of the video sequence.

45. The system of claim 24, wherein the appearance, scene geometry and motion information models are used to update the confidence score for each trajectory in each subsequent image frame in the video sequence.

46. The system of claim 24 wherein if the confidence score for a particular trajectory falls below a predetermined value, the particular trajectory is not tracked in subsequent image frames.