Method and apparatus for pedestrian detection

• US 6,956,469 B2
• Filed: 04/07/2004
• Issued: 10/18/2005
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: Ford Motor Company, SRI International, Inc.
• Original Assignee: Ford Motor Company, Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hirvonen, David, Camus, Theodore Armand
• Primary Examiner(s): Crosland, Donnie L.
• Application Number:
  US10/819,870
• Publication Number:
  US 20040258279A1
• Time in Patent Office: 559 Days
• Field of Search: 340/903, 340/435, 340/436, 340/425.5, 340/937, 340/925, 340/961, 340/555, 340/565, 340/944, 382/103, 382/104, 382/123, 382/171, 382/209, 382/217, 382/154, 348/42, 348/47, 348/51, 348/77, 348/135, 348/143, 348/148, 348/149, 348/152, 250330-334
• US Class Current: 340/435
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A vehicle vision system that identifies pedestrians located proximate a vehicle. The system comprises a sensor array that produces imagery that is processed to generate disparity images. A depth map can be produced from the disparity images. Either the depth map or the disparity images are processed and compared to pedestrian templates. A pedestrian list is produced from the results of that comparison. The pedestrian list is processed to eliminate false pedestrians as determined by inverse eccentricity. The pedestrians are then tracked, and if a collision is possible, that information is provided to the driver and/or to an automated collision avoidance or damage or injury mitigation system.

Claims

1. A method of vision processing comprising:

producing a depth map of a scene proximate a vehicle;

comparing the depth map to a pedestrian template;

identifying a match between the depth map and the pedestrian template, wherein objects having inverse eccentricities less than a threshold are not identified as a match; and

detecting the presence of a pedestrian.

2. The method of claim 1 further comprising:

imaging the scene with a stereo camera to produce stereo images; and

processing the stereo images to produce the depth map.

3. The method of claim 1 wherein the comparing step further comprises:

differencing each of the pixels in the depth map and each similarly positioned pixel in the pedestrian template; and

if the difference at each pixel is less than a predefined amount, the pixel is deemed a match.

4. The method of claim 3 wherein the identifying step comprises:

summing the number of pixels deemed a match and dividing the sum by a total number of pixels in the pedestrian template to produce a match score;

spatially and/or temporally filtering the match score values to produce a new match score; and

if the match score is greater than a predefined match score amount, the pedestrian template is deemed a match.

5. The method of claim 1 wherein the comparing step further comprises:

dividing the depth map and pedestrian template into a first half and a second half;

differencing each of the pixels in the first half of the depth map and each similarly positioned pixel in the first half of the pedestrian template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a first match;

summing numbers of pixels deemed a first match and dividing the sum by a total number of pixels in the first half of the pedestrian template to produce a first match score

differencing each of the pixels in the second half of the depth map and each similarly positioned pixel in the second half of the pedestrian template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a second match;

summing numbers of pixels deemed a second match and dividing the sum by a total number of pixels in the second half of the pedestrian template to produce a second match score;

multiplying the first match score with the second match score to determine a final match score;

if the final match score is greater than a predefined match score amount, the pedestrian template is deemed a match.

6. The method of claim 1 further comprising:

accessing at least one pedestrian template from a database comprising a plurality of pedestrian templates.

7. The method of claim 6 wherein the plurality of pedestrian templates represent pedestrians at varying positions and poses relative to the vehicle.

8. The method of claim 7 further including tracking a pedestrian across image frames.

9. The method of claim 1 further comprising:

receiving information regarding at least one pedestrian within the scene from a secondary sensor; and

using the information to limit a number of pedestrian templates that are compared to the depth map.

10. The method of claim 9 wherein the secondary sensor is at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

11. The method of claim 1 further comprising:

receiving information regarding at least one pedestrian within the scene from a secondary sensor; and

using the information to validate the match.

12. The method of claim 11 wherein the secondary sensor is at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

13. The method of claim 1 further comprising:

tracking the pedestrian across a plurality of frames to determine the position and velocity and direction of the pedestrian.

14. The method of claim 1 wherein adjusting a parameter of a vehicle includes warning of an impending collision with a pedestrian.

15. The method of claim 1 wherein the threshold is 0.4.

16. The method of claim 1 further including adjusting a parameter of a vehicle in response to the match of a pedestrian template.

17. A method of vision processing comprising:

producing a multi-resolution disparity image of a scene proximate a vehicle;

comparing the multi-resolution disparity image to a pedestrian template;

identifying a match between the multi-resolution disparity image and the pedestrian template, wherein objects having inverse eccentricities less than a threshold are not identified as a match; and

detecting the presence of a pedestrian.

18. The method of claim 17 further comprising:

imaging the scene with a stereo camera to produce stereo images; and

processing the stereo images to produce the multi-resolution disparity image.

19. The method of claim 17 wherein the comparing step further comprises:

differencing each of the pixels in the multi-resolution disparity image and each similarly positioned pixel in the pedestrian template; and

if the difference at each pixel is less than a predefined amount, the pixel is deemed a match.

20. The method of claim 17 wherein the identifying step comprises:

summing the number of pixels deemed a match and dividing the sum by a total number of pixels in the pedestrian template to produce a match score;

spatially and/or temporally filtering the match score values to produce a new match score; and

if the match score is greater than a predefined match score amount, the pedestrian template is deemed a match.

21. The method of claim 17 wherein the comparing step further comprises:

dividing the multi-resolution disparity image and pedestrian template into a first half and a second half;

differencing each of the pixels in the first half of the multi-resolution disparity image and each similarly positioned pixel in the first half of the pedestrian template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a first match;

summing numbers of pixels deemed a first match and dividing the sum by a total number of pixels in the first half of the pedestrian template to produce a first match score

differencing each of the pixels in the second half of the multi-resolution disparity image and each similarly positioned pixel in the second half of the pedestrian template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a second match;

summing numbers of pixels deemed a second match and dividing the sum by a total number of pixels in the second half of the pedestrian template to produce a second match score;

multiplying the first match score with the second match score to determine a final match score;

if the final match score is greater than a predefined match score amount, the pedestrian template is deemed a match.

22. The method of claim 17 further including accessing at least one pedestrian template from a database comprising a plurality of pedestrian templates.

23. The method of claim 22 wherein the plurality of pedestrian templates represent pedestrians at varying positions and poses relative to the vehicle.

24. The method of claim 23 further including tracking a pedestrian across image frames.

25. The method of claim 17 further comprising:

receiving information regarding at least one pedestrian within the scene from a secondary sensor; and

using the information to limit a number of pedestrian templates that are compared to the multi-resolution disparity image.

26. The method of claim 25 wherein the secondary sensor is at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

27. The method of claim 17 further comprising:

receiving information regarding at least one pedestrian within the scene from a secondary sensor; and

using the information to validate the match.

28. The method of claim 27 wherein the secondary sensor is at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

29. The method of claim 17 further including tracking the pedestrian across a plurality of multi-resolution disparity image to determine the position and velocity of the pedestrian.

30. The method of claim 17 wherein the threshold is 0.4.

31. The method of claim 17 further including adjusting a parameter of a vehicle in response to the match of a pedestrian template.

32. The method of claim 31 wherein adjusting a parameter of a vehicle includes warning of an impending collision with a pedestrian.

33. Apparatus for performing vision processing comprising:

a stereo image preprocessor for producing a multi-resolution disparity image;

a depth map generator for processing the multi-resolution disparity image to form a depth map; and

a target processor for comparing the depth map to a plurality of pedestrian templates to identify a match between at least one pedestrian template within the plurality of pedestrian templates and the depth map, wherein the target processor does not identify objects having inverse eccentricities less than a threshold.

34. The apparatus of claim 33 further including a secondary sensor that provides information regarding objects in the scene.

35. The apparatus of claim 34 wherein the secondary sensor comprises at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

36. A pedestrian detection system comprising:

a platform;

a stereo camera pair attached to said platform;

a stereo image preprocessor for producing a multi-resolution disparity image from said stereo camera pair;

a depth map generator for processing the multi-resolution disparity image disparity image to form a depth map; and

a pedestrian processor having a pedestrian template database, said pedestrian processor for comparing the depth map to a plurality of pedestrian templates in said pedestrian template database to identify a match between at least one pedestrian template and the depth map, wherein the pedestrian processor does not identify objects having inverse eccentricities less than a threshold as a pedestrian.

37. A pedestrian detection system comprising:

a vehicle;

a stereo camera pair attached to said vehicle;

a stereo image preprocessor for producing a multi-resolution disparity image from said stereo camera pair; and

a pedestrian processor having a pedestrian template database, said pedestrian processor for comparing the multi-resolution disparity image to a plurality of pedestrian templates in said pedestrian template database to identify a match between at least one pedestrian template and the multi-resolution disparity image, wherein the pedestrian processor does not identify objects having inverse eccentricities less than 0.4 as a match as a pedestrian.

38. A computer readable medium for storing a computer program that directs a computer to:

produce a depth map of a scene proximate a platform;

compare the depth map to a pedestrian template;

identify a match between the depth map and the pedestrian template, wherein objects having inverse eccentricities less than a threshold are not identified as a match; and

detect the presence of a pedestrian.

39. A computer readable medium according to claim 38 wherein the map is a depth map.

40. A computer readable medium according to claim 38 wherein the map is a disparity map.