Stereo-vision based imminent collision detection

• US 7,660,436 B2
• Filed: 01/29/2004
• Issued: 02/09/2010
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: Autoliv Electronic SAS (Autoliv Incorporated), SRI International, Inc.
• Original Assignee: Autoliv Electronic SAS (Autoliv Incorporated), Sarnoff Corporation (SRI International, Inc.)
• Inventors: Chang, Peng, Camus, Theodore Armand
• Primary Examiner(s): Le, Brian Q
• Application Number:
  US10/766,976
• Publication Number:
  US 20040252863A1
• Time in Patent Office: 2,203 Days
• Field of Search: 382/104, 382/199, 382/281, 382/291, 382/189, 701/65, 701/208, 701/209, 701300-302, 340/931, 340/932, 340/961
• US Class Current: 382/104
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A stereo vision based collision avoidance systems having stereo cameras that produce imagery that is processed to produce a depth map of a scene. A potential threat is detected in the depth map. The size, position, and velocity of the detected potential threat are then estimated, and a trajectory analysis of the detected potential threat is determined using the estimated position and the estimated velocity. A collision prediction based on the trajectory analysis is determined, and then a determination is made as to whether a collision is imminent based on the collision prediction and on the estimated size of the potential threat.

Claims

1. A computer implemented method of detecting an imminent collision comprising the steps of:

capturing and preprocessing imagery of a scene proximate a platform using an image processor;

producing from the imagery a depth map using the image processor, wherein each pixel in the depth map has associated 3D position data;

performing by the image processor the steps of tessellating the depth map into a number of patches and selecting a plurality of the patches of the depth map for processing, wherein said

processing comprise fitting a plane to each patch of said selected plurality the patches, obtaining a normal vector to each said plane, and classifying the selected plurality of patches of the depth map into a plurality of classes based on the obtained normal vector for each said selected patch and on said 3D position data; and

detecting a potential threat in the tessellated depth map during the processing of the selected plurality of the patches.

2. The method of claim 1, further comprising:

estimating position of the detected potential threat;

estimating velocity of the detected potential threat;

filtering the estimated position and filtering the estimated velocity;

performing a trajectory analysis of the detected potential threat using the filtered estimated position and the filtered estimated velocity; and

performing a collision prediction based on the trajectory analysis.

3. The method of claim 2, further comprising:

estimating size of the detected potential threat; and

determining if a collision is imminent based on the collision prediction and on the estimated size of the potential threat.

4. The method of claim 2 wherein the filtering includes Kalman filtering.

5. The method of claim 2 wherein estimating the velocity of the detected potential threat includes the step of identifying 2D feature correspondences from imagery produced in different time frames.

6. The method of claim 5 wherein estimating the velocity of the detected potential threat further includes the step of obtaining 3D correspondences from the 2D feature correspondences and from the depth map.

7. The method of claim 6 wherein estimating the velocity of the detected potential threat further includes the step of estimating velocity using Random Sample Consensus.

8. The method of claim 1

said plurality of classes comprise one class representing the patches that are likely to represent a potential threat, another class representing the patches that possibly represent a potential threat, and another class representing the patches that are unlikely to represent a potential threat.

9. The method of claim 8 further including the step of grouping patches that are likely to represent a potential threat together.

10. The method of claim 9 further including the step of creating a bounding box that represents a potential threat, wherein the bounding box is created in accord with the grouping of patches.

11. The method of claim 8 wherein detecting a potential threat in the tessellated depth map includes the steps of moving each patch after local tessellation to find the region of maximum depth map data density near the original patch location, of discarding a patch if the region of maximum depth map data density does not meet a predetermined criterion, and adding the patch to the plurality of the patches if the patch density meets the predetermined criterion.

12. The method of claim 1 wherein obtaining a normal vector includes the steps of calculating the third Eigen-vector of a matrix of patch values using a singular valued decomposition of the matrix, and then estimating the normal vector as the third Eigen-vector.

13. A collision detection system, comprising:

an imaging device for providing imagery of a scene proximate a platform;

an image preprocessor for preprocessing said imagery;

a depth map generator for producing a depth map from said preprocessed imagery wherein each pixel in the depth map has associated 3D position data; and

a collision detector for tessellating the depth map into a number of patches, selecting a plurality of the patches of the depth map for processing, wherein said processing comprise fitting a plane to each patch of said selected plurality of the patches, obtaining a normal vector to each said plane, and classifying the selected plurality of patches of the depth map into a plurality of classes based on the obtained normal vector for each said selected patch and on said 3D position data; and

detecting a potential threat in said tessellated depth map during the processing of the selected plurality of the patches.

14. The system of claim 13, wherein said collision detector estimates position and velocity of said detected potential threat;

said collision detector includes a filter for filtering image noise and outliers from said estimated position and from said estimated velocity;

said collision detector performs a trajectory analysis of said detected potential threat using said estimated position and said estimated velocity;

said collision detector predicts a collision based on said trajectory analysis.

15. The system of claim 14 wherein said collision detector estimates said velocity by identifying 2D feature correspondences from imagery produced in different time frames.

16. The system of claim 13, further including a host vehicle, wherein said image preprocessor comprises a stereo image preprocessor and said imaging device comprises a stereo camera pair mounted in fixed locations relative to said host vehicle.

17. The system of claim 13

wherein said plurality of classes comprise as likely to represent a potential threat, as possibly representing a potential threat, or as being unlikely to represent a potential threat;

and

said collision detector detects the potential threat by grouping patches that are likely to represent a potential threat together; and

forming a bounding box around said potential threat based on said patch groupings.

18. The system of claim 17 wherein said collision detector detects the potential threat by moving each patch after local tessellation to find the region of maximum depth map data density near the original patch location, discarding said patch if the region of maximum depth map data density does not meet a predetermined criterion, and adding said patch to said plurality of patches if said patch density meets said predetermined criterion.

19. The system of claim 13 wherein obtaining normal vectors includes the steps of calculating for each patch a third Eigen-vector of a matrix of patch values using a singular valued decomposition of said matrix, and then estimating said normal vector as said third Eigen-vector.

29. The system of claim 15 wherein the collision detector further estimates said velocity by obtaining 3D correspondences from the 2D feature correspondences and from the depth map.

30. The system of claim 14 wherein said collision detector estimates size of said detected potential threat and determines if a collision is imminent based on said collision prediction and on said estimated size.

20. A computer readable medium having stored thereon a plurality of instructions, the plurality of instruction including instructions which, when executed by a processor causes the processor to perform the steps comprising:

capturing and preprocessing an imagery of a scene proximate a platform;

producing from the imagery a depth map, wherein each pixel in the depth map has associated 3D position data;

tessellating the depth map into a number of patches and selecting a plurality of the patches of the depth map for processing, wherein said processing comprise fitting a plane to each patch of said selected plurality of patches, obtaining a normal vector to each said plane, and classifying the selected plurality of patches of the depth map into a plurality of classes based on the obtained normal vector for each said selected patch and on said 3D position data;

detecting a potential threat in the tessellated depth map during the processing of the selected plurality of the patches.

21. The computer readable medium of claim 20 that further causes the processor to estimate the size of the detected potential threat;

estimate the position of the detected potential threat;

estimate the velocity of the detected potential threat;

filter the estimated position and the estimated velocity perform a trajectory analysis of the detected potential threat using the estimated position and the estimated velocity; and

perform a collision prediction based on the trajectory analysis.

22. The computer readable medium of claim 21 that further causes the processor to estimate the velocity by identifying 2D feature correspondences from imagery produced in different time frames.

23. The computer readable medium of claim 22 that further causes the processor to estimate velocity by obtaining 3D correspondences from the 2D feature correspondences and from the depth map.

24. The computer readable medium of claim 20

wherein said plurality of classes comprise as likely to represent a potential threat, as possibly representing a potential threat, or as being unlikely to represent a potential threat.

25. The computer readable medium of claim 24 that further controls a computer to group patches together that are likely to represent said potential threat.

26. The computer readable medium of claim 25 that further causes the processor to create a bounding box that represents the potential threat, wherein the bounding box is created in accord with the patch groupings.

27. The computer readable medium of claim 20 that further causes the processor to detect a potential threat in said tessellated depth map by the steps of moving each patch after local tessellation to find the region of maximum depth map data density near the original patch location, discarding said patch if the region of maximum depth map data density does not meet a predetermined criterion, and adding said patch to said plurality of patches if said patch density meets said predetermined criterion.

28. The computer readable medium of claim 20 that further causes the processor to obtain normal vectors by the steps of calculating for each patch a third Eigen-vector of a matrix of patch values using a singular valued decomposition of said matrix, and then estimating said normal vector as said third Eigen-vector.