Method and apparatus for real-time pedestrian detection for urban driving

• US 8,861,842 B2
• Filed: 02/07/2011
• Issued: 10/14/2014
• Est. Priority Date: 02/05/2010
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Jung, Sang-Hack, Bansal, Mayank, Matei, Bogdan Calin Mihai, Eledath, Jayan, Sawhney, Harpreet Singh, Kumar, Rakesh, Hadsell, Raia
• Primary Examiner(s): Bali, Vikkram
• Application Number:
  US13/022,118
• Publication Number:
  US 20110255741A1
• Time in Patent Office: 1,345 Days
• Field of Search: None
• US Class Current: 382/155
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A computer implemented method for detecting the presence of one or more pedestrians in the vicinity of the vehicle is disclosed. Imagery of a scene is received from at least one image capturing device. A depth map is derived from the imagery. A plurality of pedestrian candidate regions of interest (ROIs) is detected from the depth map by matching each of the plurality of ROIs with a 3D human shape model. At least a portion of the candidate ROIs is classified by employing a cascade of classifiers tuned for a plurality of depth bands and trained on a filtered representation of data within the portion of candidate ROIs to determine whether at least one pedestrian is proximal to the vehicle.

Claims

1. A computer implemented method for detecting presence of one or more pedestrians in a vicinity of a vehicle, comprising the steps of:

receiving imagery of a scene from at least one image capturing device;

deriving a depth map from the imagery;

detecting a plurality of pedestrian candidate regions of interest (ROIs) from the depth map by matching each of the plurality of pedestrian candidate ROIs with a 3D human shape model; and

classifying at least a portion of the pedestrian candidate ROIs by employing a cascade of classifiers tuned for a plurality of depth bands and trained on a filtered representation of data within the portion of pedestrian candidate ROIs to determine whether at least one pedestrian is proximal to the vehicle.

2. The method of claim 1, wherein the filtered representation is a combination of pedestrian contour and gradient features.

3. The method of claim 1, wherein the filtered representation is based on deep learning classifiers.

4. The method of claim 3, wherein the deep learning classifiers form a multi-layer convolutional network.

5. The method of claim 1, wherein contour-based classifiers are employed for a depth of a candidate ROI closer than or equal to a predetermined distance and HOG-based classifiers are employed for a depth greater than the predetermined distance.

6. The method of claim 5, wherein the contour-based classifiers are each trained based on the steps of:

dividing a candidate ROI into a plurality of sub-ROIs;

refining each of the sub-ROIs by means of contour template matching to produce refined sub-ROIs;

composing a foreground mask from contour template matching; and

applying an HOG-based classifier to the sub-ROIs based on the refined sub-ROIs and the foreground mask.

7. The method of claim 6, wherein contour template matching is achieved by chamfer matching.

8. The method of claim 1, wherein the pedestrian candidate ROIs are pruned and labeled by classification into one of a plurality of human or non-human structures based in part on vertical support histograms.

9. The method of claim 8, wherein classification into one of a plurality of human or non-human structures comprises:

creating a 2D grid of tiles along image coordinates;

for each tile, fitting a patch grid from a planar total least squares estimate;

rejecting patches as invalid from the patch grid that having less than a predetermined number of 3D points or having less than a predetermined number of inliers;

rejecting patches as ground or below ground according to their position and normal direction; and

classifying and labeling each of the remaining patches as one of a tall vertical structure, an overhanging structure, and a candidate based on the vertical support histograms.

10. The method of claim 1, wherein an output of matching each of the plurality of pedestrian candidate ROIs with a 3D human, shape model is a correlation score map.

11. The method of claim 10, wherein a plurality of peaks in the correlation score map is selected by non-maximal suppression.

12. The method of claim 11, wherein an area about each of the plurality of peaks of the correlation score map is projected into the image to obtain an initial pedestrian candidate ROI set.

13. The method of claim 12, wherein candidate ROIs corresponding to non-pedestrians are partially pruned from the initial pedestrian ROI candidate set by removing regions that have more than a predetermined percentage of overlap with existing detections.

14. The method of claim 13, wherein the partially pruned candidate ROIs are further pruned by examining a combination of depth and edge features of two individual pedestrian detections.

15. The method of claim 14, wherein examining a combination of depth and edge features of two individual pedestrian detections further comprises the steps of:

computing a Canny edge map for each pedestrian candidate ROI;

rejecting edge pixels that are further than a predetermined distance from an expected disparity;

obtaining a vertical projection of remaining edges, resulting in a 1D profile from which peaks are detected using mean-shift;

at each detected peak, initializing a new pedestrian ROI and refined it horizontally and then vertically to obtain a centered bounding box about a pedestrian candidate ROI; and

removing any resulting overlapped pedestrian candidate ROIs from the initial pedestrian ROI candidate set.

16. The method of claim 1, wherein the plurality of depth bands is obtained at a plurality of disjoint image resolutions.

17. A non-transitory computer-readable medium comprising computer-executable instructions for detecting a presence of one or more pedestrians in a vicinity of a vehicle, the instructions configured for:

receiving a imagery of a scene from at least one image capturing device;

deriving a depth map from the imagery;

detecting a plurality of pedestrian candidate regions of interest (ROIs) from the depth map by matching each of the plurality of pedestrian candidate ROIs with a 3D human shape model; and

classifying at least a portion of the pedestrian candidate ROIs by employing a cascade of classifiers tuned for a plurality of depth bands and trained on a filtered representation of data within the portion of pedestrian candidate ROIs to determine whether at least one pedestrian is proximal to the vehicle.

18. The computer-readable medium of claim 17, wherein the filtered representation is a combination of pedestrian contour and gradient features.

19. The computer-readable medium of claim 17, wherein the filtered representation is based on deep learning classifiers.

20. The computer-readable medium of claim 19, wherein the deep learning classifiers form a multi-layer convolutional network.

21. The computer-readable medium of claim 17, wherein contour-based classifiers are employed for a depth of a candidate ROI closer than or equal to a predetermined distance and HOG-based classifiers are employed for a depth greater than the predetermined distance.

22. The computer-readable medium of claim 21, wherein the contour-based classifiers are each trained based on the steps of:

dividing a candidate ROI into a plurality of sub-ROIs;

refining each of the sub-ROIs by means of contour template matching to produce refined sub-ROIs;

composing a foreground mask from contour template matching; and

applying an HOG-based classifier to the sub-ROIs based on the refined sub-ROIs and the foreground mask.

23. The computer-readable medium of claim 22, wherein contour template matching is achieved by chamfer matching.

24. The computer-readable medium of claim 17, wherein the pedestrian candidate ROIs are pruned and labeled by classification into one of a plurality of human or non-human structures based in part on vertical support histograms.

25. The computer-readable medium of claim 24, wherein classification into one of a plurality of human or non-human structures comprises:

creating a 2D grid of tiles along image coordinates;

for each tile, fitting a patch grid from a planar total least squares estimate;

rejecting patches as invalid from the patch grid that having less than a predetermined number of 3D points or having less than a predetermined number of inliers;

rejecting patches as ground or below ground according to their position and normal direction; and

classifying and labeling each of the remaining patches as one of a tall vertical structure, an overhanging structure, and a candidate based on the vertical support histograms.

26. A system for detecting a presence of one or more pedestrians in a vicinity of a vehicle, comprising:

at least one image capturing device for receiving imagery of a scene;

at least one processor communicatively connected to the at least one image capturing device for;

receiving the imagery of the scene from the image capturing device;

deriving a depth map from said imagery;

detecting a plurality of pedestrian candidate regions of interest (ROIs) from the depth map by matching each of the pedestrian candidate plurality of ROIs with a 3D human shape model; and

classifying at least a portion of the pedestrian candidate ROIs by employing a cascade of classifiers tuned for a plurality of depth bands and trained on a filtered representation of data within the portion of pedestrian candidate ROIs to determine whether at least one pedestrian is proximal to the vehicle.

27. The system of claim 26, wherein the at least one image capturing device is two or more cameras, one or more LIDAR devices, one or more radar devices, one or more SONAR devices, Or one or more passive or active ranging devices.