System and method for vehicle detection and tracking

• US 20040234136A1
• Filed: 03/22/2004
• Published: 11/25/2004
• Est. Priority Date: 03/24/2003
• Status: Active Grant

• Current Assignee: Continental Automotive GmbH (Continental AG)
• Original Assignee: Siemens Corp. (Siemens AG)
• Inventors: Zhu, Ying, Comaniciu, Dorin, Kohler, Thorsten, Pellkofer, Martin
• Granted Patent: US 7,764,808 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes

Abstract

A system and method for detecting and tracking an object is disclosed. A camera captures a video sequence comprised of a plurality of image frames. A processor receives the video sequence and analyzes each image frame to determine if an object is detected. The processor applies one or more classifiers to an object in each image frame and computes a confidence score based on the application of the one or more classifiers to the object. A database stores the one or more classifiers and vehicle training samples. A display displays the video sequence.

Claims

1. A method for detecting one or more objects belonging to the same object class comprising the steps of:

a). receiving a video sequence comprised of a plurality of image frames;

b). applying one or more classifiers to detect components of objects in an image frame in the video sequence;

c). computing a confidence score based in part on the response from the one or more component detectors;

d). repeating steps b). and c). to detect components of objects belonging to the same object class in additional images frames in the video sequence; and

e). accumulating confidence scores from the component detectors to determine if an object is detected.

2. The method of claim 1 wherein if accumulated confidence scores indicate high confidence of a presence of an object, the method further comprising the step of:

identifying the detected components to be an object of a particular type.

3. The method of claim 1 wherein the object class is a vehicle.

4. The method of claim 1 further comprising the step of:

if an object is detected, outputting a detection signal and object position.

5. The method of claim 2 further comprising the steps of:

testing geometry constraints on a spatial arrangement of detected components in an image; and

applying whole-appearance classifiers an image patch that contains the detected components and which is aligned according to the position of the detected components.

6. The method of claim 5 wherein the geometry constraints are derived from camera parameters.

7. The method of claim 5 wherein the geometry constraints are derived from object size.

8. The method of claim 5 wherein the geometry constraints are derived from a location of an object appearance in the image.

9. The method of claim 5 wherein the whole appearance classifiers detect entire or partial object appearance, the entire or partial object appearance being aligned according to positioning of at least two components.

10. The method of claim 1 wherein the component classifiers include classifiers for detecting components at multiple scales.

11. The method of claim 1 wherein component classifiers are defined by discriminant features and decision rules which are learned through boosted training.

12. The method of claim 11 wherein the discriminant features include corners.

13. The method of claim 11 wherein the discriminant features include horizontal edges.

14. The method of claim 11 wherein the discriminant features include vertical edges.

15. The method of claim 11 wherein the discriminant features include horizontal stripes.

16. The method of claim 11 wherein the discriminant features include vertical stripes.

17. The method of claim 11 wherein the discriminant features include diagonal stripes.

18. The method of claim 11 further comprising the step of:

performing an online adaptation to adapt a classifier structure to an online pattern.

19. The method of claim 18 wherein the step of performing an online adaptation further comprises the step of:

applying a dynamic switching strategy to direct the detector to take appropriate weak classifiers as discriminants according to auxiliary information about the online pattern.

20. The method of claim 2 wherein the one or more classifiers include overlapping component classifiers.

21. The method of claim 20 wherein the overlapping component classifiers comprises four corners representing a rear profile of a vehicle.

22. The method of claim 20 wherein the overlapping component classifiers comprises four corners representing a frontal profile of a vehicle.

23. The method of claim 20 wherein the overlapping component classifiers comprises four corners representing a side profile of a vehicle.

24. The method of claim 21 wherein one of the overlapping component classifiers detects the bottom left corner of a vehicle.

25. The method of claim 21 wherein one of the overlapping component classifiers detects the bottom right corner of a vehicle.

26. The method of claim 21 wherein one of the overlapping component classifiers detects the top left corner of a vehicle.

27. The method of claim 21 wherein one of the overlapping component classifiers detects the top right corner of a vehicle.

28. The method of claim 21 wherein positioning of the four corners of the rear profile for a vehicle differ for different classes of vehicles.

29. The method of claim 28 wherein a class of vehicle includes sedans.

30. The method of claim 28 wherein a class of vehicle includes sports utility vehicles.

31. The method of claim 28 wherein a class of vehicle includes vans.

32. The method of claim 28 wherein a class of vehicle includes tractor trailers.

33. The method of claim 28 wherein a class of vehicle includes trucks.

34. The method of claim 21 wherein a distance between any two corners of the vehicle is constrained.

35. The method of claim 34 wherein the constraint between any two corners of the vehicle is scaled based on a distance between the vehicle and a camera capturing the video sequence and camera parameters.

36. The method of claim 35 wherein an image pyramid of multiple resolutions is used to detect objects of size 2*x, 4*x and so on with the classifier for the size x.

37. The method of claim 1 wherein the accumulated confidence scores are inferred from confidence scores across multiple frames using a recursive filter.

38. The method of claim 37 wherein when the accumulated confidence score is a linear combination of the confidence scores of multiple component classifiers and the whole-appearance classifiers.

39. The method of claim 38 wherein when the confidence score for a principal component classifier is sufficiently high, the confidence score of the remaining component classifiers and the whole-appearance classifier are computed.

40. The method of claim 2 wherein if an object is detected the method comprising the step of:

tracking the object over subsequent image frames.

41. The method of claim 40 wherein the step of tracking the object further comprises the step of:

restricting an area of search in each subsequent image frame based on the location of the object in a current image frame.

42. The method of claim 40 wherein the step of tracking the object further comprises the step of:

determining the optimal classifier scale based on a distance between the object and a camera detecting the object and camera parameters.

43. The method of claim 1 wherein the confidence scores of component classifiers are computed in a coarse to fine framework.

44. The method of claim 1 wherein detection is performed on an image pyramid of multiple resolutions.

45. The method of claim 1 wherein an object class includes pedestrians.

46. The method of claim 1 wherein an object class includes bicycles.

47. The method of claim 1 wherein an object class includes motorcycles.

48. The method of claim 1 wherein object class includes different types of traffic signs.

49. A system for detection and tracking an object comprising:

a camera for capturing a video sequence comprised of a plurality of image frames;

a processor for receiving the video sequence and analyzing each image frame to determine if an object is detected, said processor applying one or more classifiers to detect components of objects in each image frame and computing a confidence score based on the response from the one or more component detectors and the result of additional validation; and

a database for storing the one or more classifiers and object training samples

50. The method of claim 49 wherein the object class is a vehicle.

51. The method of claim 49 wherein the object class is a pedestrian.

52. The method of claim 49 wherein the object class is a bicycle.

53. The method of claim 49 wherein the object class is a motorbike.

54. The method of claim 49 wherein the object class includes different types of traffic signs.

55. The system of claim 49 wherein the detected components are determined to be an object if the confidence scores are high.

56. The system of claim 55 wherein if an object is detected, the processor outputs a warning signal.

57. The system of claim 49 further comprising:

a display for displaying the video sequence.

58. The system of claim 49 wherein the processor further comprises:

means for testing geometry constraints on a spatial arrangement of detected components in an image; and

means for applying whole-appearance classifiers an image patch that contains the detected components and which is aligned according to the position of the detected components.

59. The system of claim 58 wherein the geometry constraints are derived from camera parameters.

60. The system of claim 58 wherein the geometry constraints are derived from object size.

61. The system of claim 58 wherein the geometry constraints are derived from a location of an object appearance in the image.

62. The system of claim 58 wherein the whole appearance classifiers detect entire or partial object appearance, the entire or partial object appearance being aligned according to positioning of at least two components.

63. The system of claim 49 wherein the component classifiers include classifiers for detecting components at multiple scales.

64. The system of claim 49 wherein component classifiers are defined by discriminant features and decision rules which are learned through boosted training.

65. The system of claim 64 wherein the discriminant features include corners.

66. The system of claim 64 wherein the discriminant features include horizontal edges.

67. The system of claim 64 wherein the discriminant features include vertical edges.

68. The system of claim 64 wherein the discriminant features include horizontal stripes.

69. The system of claim 64 wherein the discriminant features include vertical stripes.

70. The system of claim 64 wherein the discriminant features include diagonal stripes.

71. The system of claim 64 further comprising the step of:

performing an online adaptation to adapt a classifier structure to an online pattern.

72. The system of claim 71 wherein the step of performing an online adaptation further comprises the step of:

applying a dynamic switching strategy to direct the detector to take appropriate weak classifiers as discriminants according to auxiliary information about the online pattern.

73. The system of claim 49 wherein the one or more classifiers include overlapping component classifiers.

74. The system of claim 73 wherein the overlapping component classifiers comprises four corners representing a rear profile of a vehicle.

75. The system of claim 74 wherein one of the overlapping component classifiers detects the bottom left corner of a vehicle.

76. The system of claim 74 wherein one of the overlapping component classifiers detects the bottom right corner of a vehicle.

77. The system of claim 74 wherein one of the overlapping component classifiers detects the top left corner of a vehicle.

78. The system of claim 74 wherein one of the overlapping component classifiers detects the top right corner of a vehicle.

79. The system of claim 74 wherein positioning of the four corners of the rear profile for a vehicle differ for different classes of vehicles.

80. The system of claim 79 wherein a class of vehicle includes sedans.

81. The system of claim 79 wherein a class of vehicle includes sports utility vehicles.

82. The system of claim 79 wherein a class of vehicle includes vans.

83. The system of claim 79 wherein a class of vehicle includes tractor trailers.

84. The system of claim 74 wherein a distance between any two corners of the vehicle is constrained.

85. The system of claim 84 wherein the constraint between any two corners of the vehicle is scaled based on a distance between the vehicle and a camera capturing the video sequence as well as camera parameters.

86. The system of claim 85 wherein an image pyramid of multiple resolutions is used to detect objects of size 2*x, 4*x and so on with the classifier for the size x.

87. The system of claim 49 wherein the accumulated confidence scores is inferred from confidence scores across multiple frames using a recursive filter.

88. The system of claim 87 wherein when the accumulated confidence score is a linear combination of the confidence scores of multiple component classifiers and the whole-appearance classifiers.

89. The system of claim 87 wherein when the confidence score for a principal component classifier is sufficiently high, the confidence score of the remaining component classifiers and the whole-appearance classifier are computed.

90. The system of claim 49 wherein the processor comprises:

means for tracking a detected object over subsequent image frames.

91. The system of claim 90 wherein tracking means further comprises:

means for restricting an area of search in each subsequent image frame based on the location of the object in a current image frame.

92. The system of claim 90 wherein the tracking means further comprises:

means for determining the optimal classifier scale based on a distance between the object and a camera detecting the object and camera parameters.

93. The system of claim 49 wherein the confidence scores of component classifiers are computed in a coarse to fine framework

94. The system of claim 49 wherein detection and tracking is performed on an image pyramid of multiple resolutions.

95. A method for tracking one or more objects depicted in a video sequence comprising:

receiving a video sequence comprised of a plurality of image frames;

detecting an object to be tracked in one of the image frames;

computing one or more appearance trajectories based on the position of the tracked object to estimate the direction in which the tracked object is traveling; and

determining if the object is the tracked object based on the appearance trajectory.

96. The method of claim 95 wherein said step of determining if the object is the tracked object further comprises the steps of:

applying one or more classifiers to the computed appearance trajectory;

computing a confidence score for the computed appearance trajectory; and

determining the object to be the tracked object if the confidence score is high.

97. The method of claim 95 wherein the computation of the appearance trajectory considers physically well-founded motion constraints.

98. The method of claim 95 wherein the trajectories are computed by accumulating detection results over multiple frames.

99. The method of claim 98 wherein the computed appearance trajectory determines changes in position, scale, location and aspect conditions in consecutive frames.

100. The method of claim 95 wherein the tracked object is a vehicle.

101. The method of claim 95 wherein the tracked object is a pedestrian.

102. The method of claim 95 wherein the tracked object is a bicycle.

103. The method of claim 95 wherein the tracked object is a motorbike.

104. The method of claim 95 wherein the tracked object includes different types of traffic signs.