System and method for vehicle detection and tracking

• US 7,764,808 B2
• Filed: 03/22/2004
• Issued: 07/27/2010
• Est. Priority Date: 03/24/2003
• Status: Active Grant

• Current Assignee: Continental Automotive GmbH (Continental AG)
• Original Assignee: Siemens AG, Siemens Corp. (Siemens AG)
• Inventors: Zhu, Ying, Comaniciu, Dorin, Kohler, Thorsten, Pellkofer, Martin
• Primary Examiner(s): Le; Vu
• Assistant Examiner(s): WANG, CLAIRE X
• Application Number:
  US10/805,946
• Publication Number:
  US 20040234136A1
• Time in Patent Office: 2,318 Days
• Field of Search: 382/104, 382/103
• US Class Current: 382/104
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes

Abstract

A system and method for detecting and tracking an object is disclosed. A camera captures a video sequence comprised of a plurality of image frames. A processor receives the video sequence and analyzes each image frame to determine if an object is detected. The processor applies one or more classifiers to an object in each image frame and computes a confidence score based on the application of the one or more classifiers to the object. A database stores the one or more classifiers and vehicle training samples. A display displays the video sequence.

Claims

1. A method for detecting one or more objects belonging to the same object class comprising the steps of:

a) receiving a video sequence from a video camera comprised of a plurality of image frames;

b) applying one or more component classifiers to detect components of objects in an image frame in the video sequence, wherein the component classifiers include classifiers for detecting object components of different sizes at multiple scales;

c) computing a confidence score based in part on the response from the one or more component detectors;

d) repeating steps b) and c) to detect components of objects belonging to the same object class in additional images frames in the video sequence; and

e) accumulating confidence scores from the component detectors to determine if an object is detected,

wherein said method is adapted for detecting moving and stationary objects from a moving video camera.

2. The method of claim 1 wherein if accumulated confidence scores indicate high confidence of a presence of an object, the method further comprising the step of:

identifying the detected components to be an object of a particular object class.

3. The method of claim 1 wherein the object class is a vehicle.

4. The method of claim 1 further comprising the step of:

if an object is detected, outputting a detection signal and object position.

5. The method of claim 1 further comprising the steps of:

testing geometry constraints on a spatial arrangement of detected components in an image; and

applying whole-appearance classifiers an image patch that contains the detected components and which is aligned according to the position of the detected components.

6. The method of claim 5 wherein the geometry constraints are derived from camera parameters.

7. The method of claim 5 wherein the geometry constraints are derived from object size.

8. The method of claim 5 wherein the geometry constraints are derived from a location of an object appearance in the image.

9. The method of claim 5 wherein the whole appearance classifiers detect entire or partial object appearance, the entire or partial object appearance being aligned according to positioning of at least two components.

10. The method of claim 1 wherein component classifiers are defined by discriminant features and decision rules which are learned through boosted training.

11. The method of claim 10 wherein the discriminant features include corners.

12. The method of claim 10 wherein the discriminant features include horizontal edges.

13. The method of claim 10 wherein the discriminant features include vertical edges.

14. The method of claim 10 wherein the discriminant features include horizontal stripes.

15. The method of claim 10 wherein the discriminant features include vertical stripes.

16. The method of claim 10 wherein the discriminant features include diagonal stripes.

17. The method of claim 10 further comprising the step of:

performing an online adaptation to adapt a classifier structure to an online pattern.

18. The method of claim 17 wherein the step of performing an online adaptation further comprises the step of:

applying a dynamic switching strategy to direct the detector to take appropriate weak classifiers as discriminants according to auxiliary information about the online pattern.

19. The method of claim 1 wherein the one or more classifiers include overlapping component classifiers.

20. The method of claim 19 wherein the overlapping component classifiers comprises four corners representing a rear profile of a vehicle.

21. The method of claim 19 wherein the overlapping component classifiers comprises four corners representing a frontal profile of a vehicle.

22. The method of claim 19 wherein the overlapping component classifiers comprises four corners representing a side profile of a vehicle.

23. The method of claim 20 wherein one of the overlapping component classifiers detects the bottom left corner of a vehicle.

24. The method of claim 20 wherein one of the overlapping component classifiers detects the bottom right corner of a vehicle.

25. The method of claim 20 wherein one of the overlapping component classifiers detects the top left corner of a vehicle.

26. The method of claim 20 wherein one of the overlapping component classifiers detects the top right corner of a vehicle.

27. The method of claim 20 wherein positioning of the four corners of the rear profile for a vehicle differ for different classes of vehicles.

28. The method of claim 27 wherein a class of vehicle includes sedans.

29. The method of claim 27 wherein a class of vehicle includes sports utility vehicles.

30. The method of claim 27 wherein a class of vehicle includes vans.

31. The method of claim 27 wherein a class of vehicle includes tractor trailers.

32. The method of claim 27 wherein a class of vehicle includes trucks.

33. The method of claim 20 wherein a distance between any two corners of the vehicle is constrained.

34. The method of claim 33 wherein the constraint between any two corners of the vehicle is scaled based on a distance between the vehicle and a camera capturing the video sequence and camera parameters.

35. The method of claim 1 wherein an image pyramid of multiple resolutions is used to detect objects of size 2*x, 4*x and so on with the classifier for the size x.

36. The method of claim 1 wherein the accumulated confidence scores are inferred from confidence scores across multiple frames using a recursive filter.

37. The method of claim 36 wherein when the accumulated confidence score is a linear combination of the confidence scores of multiple component classifiers and the whole-appearance classifiers.

38. The method of claim 37 wherein when the confidence score for a principal component classifier is sufficiently high, the confidence score of the remaining component classifiers and the whole-appearance classifier are computed.

39. The method of claim 2 wherein if an object is detected the method comprising the step of:

tracking the object over subsequent image frames.

40. The method of claim 39 wherein the step of tracking the object further comprises the step of:

restricting an area of search in each subsequent image frame based on the location of the object in a current image frame.

41. The method of claim 39 wherein the step of tracking the object further comprises the step of:

determining the optimal classifier scale based on a distance between the object and a camera detecting the object and camera parameters.

42. The method of claim 1 wherein the confidence scores of component classifiers are computed in a coarse to fine framework.

43. The method of claim 1 wherein detection is performed on an image pyramid of multiple resolutions.

44. The method of claim 1 wherein an object class includes pedestrians.

45. The method of claim 1 wherein an object class includes bicycles.

46. The method of claim 1 wherein an object class includes motorcycles.

47. The method of claim 1 wherein object class includes different types of traffic signs.

48. A system for detection and tracking an object comprising:

a camera for capturing a video sequence comprised of a plurality of image frames;

a processor for receiving the video sequence and analyzing each image frame to determine if an object is detected, said processor applying one or more component classifiers to detect components of objects in each image frame and computing a confidence score based on the response from the one or more component detectors and the result of additional validation, wherein the accumulated confidence scores is inferred from confidence scores across multiple frames using a recursive filter; and

a database for storing the one or more classifiers and object training samples,

wherein said system is adapted for detecting moving and stationary objects from a moving video camera.

49. The method of claim 48 wherein the object class is a vehicle.

50. The method of claim 48 wherein the object class is a pedestrian.

51. The method of claim 48 wherein the object class is a bicycle.

52. The method of claim 48 wherein the object class is a motorbike.

53. The method of claim 48 wherein the object class includes different types of traffic signs.

54. The system of claim 48 wherein the detected components are determined to be an object of a particular object class if the confidence scores are high.

55. The system of claim 54 wherein if an object is detected, the processor outputs a warning signal.

56. The system of claim 48 further comprising:

a display for displaying the video sequence.

57. The system of claim 48 wherein the processor further comprises:

means for testing geometry constraints on a spatial arrangement of detected components in an image; and

means for applying whole-appearance classifiers an image patch that contains the detected components and which is aligned according to the position of the detected components.

58. The system of claim 57 wherein the geometry constraints are derived from camera parameters.

59. The system of claim 57 wherein the geometry constraints are derived from object size.

60. The system of claim 57 wherein the geometry constraints are derived from a location of an object appearance in the image.

61. The system of claim 57 wherein the whole appearance classifiers detect entire or partial object appearance, the entire or partial object appearance being aligned according to positioning of at least two components.

62. The system of claim 48 wherein the component classifiers include classifiers for detecting object components of different sizes at multiple scales.

63. The system of claim 48 wherein component classifiers are defined by discriminant features and decision rules which are learned through boosted training.

64. The system of claim 63 wherein the discriminant features include corners.

65. The system of claim 63 wherein the discriminant features include horizontal edges.

66. The system of claim 63 wherein the discriminant features include vertical edges.

67. The system of claim 63 wherein the discriminant features include horizontal stripes.

68. The system of claim 63 wherein the discriminant features include vertical stripes.

69. The system of claim 63 wherein the discriminant features include diagonal stripes.

70. The system of claim 63 further comprising the step of:

performing an online adaptation to adapt a classifier structure to an online pattern.

71. The system of claim 70 wherein the step of performing an online adaptation further comprises the step of:

applying a dynamic switching strategy to direct the detector to take appropriate weak classifiers as discriminants according to auxiliary information about the online pattern.

72. The system of claim 48 wherein the one or more classifiers include overlapping component classifiers.

73. The system of claim 72 wherein the overlapping component classifiers comprises four corners representing a rear profile of a vehicle.

74. The system of claim 73 wherein one of the overlapping component classifiers detects the bottom left corner of a vehicle.

75. The system of claim 73 wherein one of the overlapping component classifiers detects the bottom right corner of a vehicle.

76. The system of claim 73 wherein one of the overlapping component classifiers detects the top left corner of a vehicle.

77. The system of claim 73 wherein one of the overlapping component classifiers detects the top right corner of a vehicle.

78. The system of claim 73 wherein positioning of the four corners of the rear profile for a vehicle differ for different classes of vehicles.

79. The system of claim 78 wherein a class of vehicle includes sedans.

80. The system of claim 78 wherein a class of vehicle includes sports utility vehicles.

81. The system of claim 78 wherein a class of vehicle includes vans.

82. The system of claim 78 wherein a class of vehicle includes tractor trailers.

83. The system of claim 73 wherein a distance between any two corners of the vehicle is constrained.

84. The system of claim 83 wherein the constraint between any two corners of the vehicle is scaled based on a distance between the vehicle and a camera capturing the video sequence as well as camera parameters.

85. The system of claim 84 wherein an image pyramid of multiple resolutions is used to detect objects of size 2*x, 4*x and so on with the classifier for the size x.

86. The system of claim 48 wherein when the accumulated confidence score is a linear combination of the confidence scores of multiple component classifiers and the whole-appearance classifiers.

87. The system of claim 48 wherein when the confidence score for a principal component classifier is sufficiently high, the confidence score of the remaining component classifiers and the whole-appearance classifier are computed.

88. The system of claim 48 wherein the processor comprises:

means for tracking a detected object over subsequent image frames.

89. The system of claim 88 wherein tracking means further comprises:

means for restricting an area of search in each subsequent image frame based on the location of the object in a current image frame.

90. The system of claim 88 wherein the tracking means further comprises:

means for determining the optimal classifier scale based on a distance between the object and a camera detecting the object and camera parameters.

91. The system of claim 48 wherein the confidence scores of component classifiers are computed in a coarse to fine framework.

92. The system of claim 48 wherein detection and tracking is performed on an image pyramid of multiple resolutions.

93. A method for detecting one or more objects belonging to the same object class comprising the steps of:

a) receiving a video sequence from a video camera comprised of a plurality of image frames;

b) applying one or more component classifiers to detect components of objects in an image frame in the video sequence, wherein the one or more component classifiers include overlapping component classifiers, and wherein component classifiers are defined by discriminant features and decision rules which are learned through boosting;

c) computing a confidence score based in part on the response from the one or more component detectors;

d) repeating steps b) and c) to detect components of objects belonging to the same object class in additional images frames in the video sequence; and

e) accumulating confidence scores from the component detectors to determine if an object is detected,

wherein said method is adapted for detecting moving and stationary objects from a moving video camera.