Stereo-vision based imminent collision detection

• US 20040252863A1
• Filed: 01/29/2004
• Published: 12/16/2004
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: Autoliv Electronic SAS (Autoliv Incorporated), SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Chang, Peng, Camus, Theodore Armand
• Granted Patent: US 7,660,436 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A stereo vision based collision avoidance systems having stereo cameras that produce imagery that is processed to produce a depth map of a scene. A potential threat is detected in the depth map. The size, position, and velocity of the detected potential threat are then estimated, and a trajectory analysis of the detected potential threat is determined using the estimated position and the estimated velocity. A collision prediction based on the trajectory analysis is determined, and then a determination is made as to whether a collision is imminent based on the collision prediction and on the estimated size of the potential threat.

Claims

1. A method of detecting an imminent collision comprising the steps of:

producing from imagery a depth map of a scene proximate a platform;

detecting a potential threat in the depth map;

estimating the size of the detected potential threat;

estimating the position of the detected potential threat;

estimating the velocity of the detected potential threat;

performing a trajectory analysis of the detected potential threat using the estimated position and the estimated velocity; and

performing a collision prediction based on the trajectory analysis.

2. The method of claim 1, further including determining if a collision is imminent based on the collision prediction and on the estimated size of the potential threat.

3. The method of claim 1, further including filtering the estimated position and filtering the estimated velocity before performing trajectory analysis.

4. The method of claim 2 wherein the filtering includes Kalman filtering.

5. The method of claim 1 wherein estimating the velocity of the detected potential threat includes the step of identifying 2-dimensional feature correspondences from imagery produced in different time frames.

6. The method of claim 5 wherein estimating the velocity of the detected potential threat further includes the step of obtaining 3D correspondences from the 2-dimensional feature correspondences and from the depth map.

7. The method of claim 6 wherein estimating the velocity of the detected potential threat further includes the step of estimating velocity using Random Sample Consensus.

8. The method of claim 1 wherein detecting a potential threat in the depth map includes the steps of:

tessellating the depth map into a number of patches;

selecting a plurality of patches for subsequent processing;

fitting a plane to each patch of said plurality of the patches;

obtaining a normal vector to each plane; and

classifying each patch of said plurality of patches into a class, where one class represents the patches that are likely represent a potential threat, another class represents the patches that possibly representing a potential threat, and another class represents the patches that are unlikely to represent a potential threat, wherein classifying is based on the obtained normal vector for each patch.

9. The method of claim 8 further including the step of grouping patches that are likely to represent a potential threat together.

10. The method of claim 9 further including the step of creating a bounding box that represents a potential threat, wherein the bounding box is created in accord with the grouping of patches.

11. The method of claim 8 wherein detecting a potential threat in the depth map includes the steps of moving each patch after local tessellation to find the region of maximum stereo density near the original patch location, of discarding a patch if the region of maximum stereo density does not meet a predetermined criteria, and adding the patch to the plurality of the patches if the patch density meets the predetermined criteria.

12. The method of claim 8 wherein obtaining a normal vector includes the steps of calculating the third Eigen-vector of a matrix of patch values using a singular valued decomposition of the matrix, and then estimating the normal vector as the third Eigen-vector.

13. A collision detection system, comprising:

a stereo camera pair for producing imagery of a scene;

a stereo image preprocessor for preprocessing said imagery;

a depth map generator for producing a depth map from said preprocessed imagery; and

a collision detector for determining from said depth map and from said imagery if a collision is imminent;

wherein said collision detector detects a potential threat in said depth map;

wherein said collision detector estimates size, position, and velocity of said detected potential threat;

wherein said collision detector performs a trajectory analysis of said detected potential threat using said estimated position and said estimated velocity;

wherein said collision detector predicts a collision based on said trajectory analysis; and

wherein said collision detector determines if a collision is imminent based on said collision prediction and on said estimated size.

14. The system of claim 13, wherein said collision detector includes a filter for filtering image noise and outliers from said estimated position and from said estimated velocity before performing trajectory analysis.

15. The system of claim 13 wherein said collision detector estimates said by identifying 2-dimensional feature correspondences from imagery produced in different time frames and then obtains 3D correspondences of said 2-dimensional feature correspondences using said depth map.

16. The system of claim 13, further including a host vehicle, wherein said stereo camera pair is mounted in fixed locations relative to said host vehicle.

17. The system of claim 13 wherein said collision detector detects a potential threat in the depth map by:

tessellating said depth map into a number of patches;

fitting a plane to a plurality of said patches;

obtaining normal vectors to said plurality of patches;

classifying each of said plurality of patches, based on said normal vector for that patch, as likely to represent a potential threat, as possibly representing a potential threat, or as being unlikely to represent a potential threat; and

.

grouping patches that are likely to represent a potential threat together; and

forming a bounding box around said potential threat based on said patch groupings.

18. The system of claim 17 wherein said collision detector searches each patch after tessellation to find a densest part of said patch, discards said patch if said patch density does not meet a predetermined criteria, and adds said patch to said plurality of patches if said patch density meets said predetermined criteria.

19. The system of claim 17 wherein obtaining normal vectors includes the steps of calculating for each patch a third Eigen-vector of a matrix of patch values using a singular valued decomposition of said matrix, and then estimating said normal vector as said third Eigen-vector.

20. A computer readable medium having stored thereon a plurality of instructions, the plurality of instruction including instructions which, when executed by a processor causes the processor to perform the steps comprising:

producing from imagery a depth map of a scene proximate a platform;

detecting a potential threat in the depth map;

estimating the size of the detected potential threat;

estimating the position of the detected potential threat;

estimating the velocity of the detected potential threat;

performing a trajectory analysis of the detected potential threat using the estimated position and the estimated velocity; and

performing a collision prediction based on the trajectory analysis.

21. The computer readable medium of claim 20 that further causes the processor to filter the estimated position and the estimated velocity before performing trajectory analysis.

22. The computer readable medium of claim 20 that further causes the processor to determine the velocity by identifying 2-dimensional feature correspondences from imagery produced in different time frames.

23. The computer readable medium of claim 22 that further causes the processor to determine velocity by obtaining 3D correspondences from the 2-dimensional feature correspondences and from the depth map.

24. The computer readable medium of claim 20 that further causes the processor to detect the potential threat by the steps of:

tessellating said depth map into a number of patches;

fitting a plane to each patch of a plurality of patches;

obtaining a normal vector to each plane; and

classifying each of said plurality of patches, based on said normal vector for that patch, as likely to represent a potential threat, as possibly representing a potential threat, or as being unlikely to represent a potential threat.

25. The computer readable medium of claim 24 that further controls a computer to group patches together that are likely to represent said potential threat.

26. The computer readable medium of claim 25 that further causes the processor to create a bounding box that represents the potential threat, wherein the bounding box is created in accord with the patch groupings.

27. The computer readable medium of claim 20 that further causes the processor to detect a potential threat in said depth map by the steps of moving each patch after local tessellation to find the region of maximum stereo density near the original patch location, discarding said patch if the region of maximum stereo density does not meet a predetermined criteria, and adding said patch to said plurality of patches if said patch density meets said predetermined criteria.

28. The computer readable medium of claim 20 that further causes the processor to obtain normal vectors by the steps of calculating for each patch a third Eigen-vector of a matrix of patch values using a singular valued decomposition of said matrix, and then estimating said normal vector as said third Eigen-vector.