Enhanced obstacle detection and tracking for three-dimensional imaging systems used in motor vehicles

• US 7,741,961 B1
• Filed: 09/28/2007
• Issued: 06/22/2010
• Est. Priority Date: 09/29/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Tracey, Colin, Rafii, Abbas, Hsu, Stephen Charles
• Primary Examiner(s): GOINS, DAVETTA WOODS
• Application Number:
  US11/906,046
• Time in Patent Office: 998 Days
• Field of Search: 340/435, 340/436, 340/923.2, 340/937, 340/934, 382/103, 382/224, 348/142, 348/148, 701/96, 701/209, 701/211
• US Class Current: 340/435
• CPC Class Codes: B60Q 9/005 using a video camera
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 1/31 providing stereoscopic vision
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/70 characterised by an event-triggered choice to display a specific image among a selection of captured images
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

An obstacle detection and tracking system identifies objects in the path of a vehicle equipped with the system and issues a visual, audible, and/or control system warning. The system includes a depth imaging system that acquires depth data from objects in the field of view of a detection zone encompassing at least a portion of the road to be driven upon. It is assumed most of the acquired data represents road plane information. Statistical analysis of the depth image data identifies in (X,Y,Z) space at least one plane of the road being driven, after which identification threshold normal heights above and below the road plane are defined. Imaged objects within the detection zone that are higher or lower than a threshold normal are deemed of potential concern and will generate a warning to the vehicle operator or vehicle.

Claims

1. An obstacle detection system to detect objects in the path of a motor vehicle moving on a road surface and identify which of said objects are of potential concern as obstacles, the obstacle detection system comprising:

a depth imaging system that acquires depth data and is attachable to said motor vehicle so as to have a field of view encompassing at least a portion of a desired detection zone in which an object of potential concern to said motor vehicle may be present, said depth imaging system including an emitter of optical energy, at least one detector that detects at least a fraction of the emitted said optical energy reflected by at least one of said objects, and means for determining said depth data from a comparison of the emitted said optical energy and said fraction of said optical energy reflected by at least one of said objects;

means for processing data acquired by said depth imaging system to model said road surface in at least said detection zone so as to define at least one road plane therein;

means for discerning from data processed by said means for processing objects that are of potential concern to said motor vehicle; and

means for signaling at least one warning mechanism, coupled to said means for discerning, responsive to identification of an object of potential concern.

2. The obstacle detection system of claim 1, wherein said depth imaging system acquires data most of which represents at least one plane of said road surface.

3. The obstacle detection system of claim 1, wherein said means for processing data samples said data.

4. The obstacle detection system of claim 1, wherein said means for processing data carries out a RANSAC routine.

5. The obstacle detection system of claim 1, wherein said depth imaging system is a time-of-flight (TOF) system.

6. The obstacle detection system of claim 1, wherein said depth imaging system includes a stereographic camera system.

7. The obstacle detection system of claim 1, wherein at least one of said means for processing data and said means for discerning includes at least one software routine stored on a machine readable medium, which routine when executed by a processor so processes said data acquired by said depth imaging system.

8. The obstacle detection system of claim 1, wherein said means for discerning employs height detection masking such that an object having height greater than a threshold normal relative to said road plane is identified as being of potential concern.

9. The obstacle detection system of claim 1, wherein said means for discerning includes slope detection masking.

10. A method to detect objects in the path of a motor vehicle moving on a road surface and to identify which of said objects are of potential concern as obstacles, the method comprising the following steps:

(a) providing a depth imaging system that acquires depth data and is attachable, to said motor vehicle and has a field of view encompassing at least a portion of a desired detection zone in which an object of potential concern to said motor vehicle may be present, said depth imaging system including an emitter of optical energy, at least one detector that detects at least a fraction of the emitted said optical energy reflected by at least one of said objects, and means for determining said depth data from a comparison of the emitted said optical energy and said fraction of said optical energy reflected by at least one of said objects;

(b) processing data acquired at step (a) to model said road surface in at least said detection zone so as to define at least one road plane therein;

(c) discerning from data processed at step (b) objects that are of potential concern to said motor vehicle; and

(d) outputting to at least one warning mechanism at least one signal responsive to identification of an object discerned at step (c) as being of potential concern.

11. The method of claim 10, wherein said depth imaging system acquires data most of which represents at least one plane of said road surface.

12. The method of claim 10, wherein step (b) includes sampling said data.

13. The method of claim 10, wherein step (b) includes processing data using a RANSAC routine.

14. The method of claim 10, wherein said depth imaging system is a time-of-flight (TOF) system.

15. The method of claim 10, wherein said depth imaging system includes a stereographic camera system.

16. The method of claim 10, wherein at least a portion of step (b) is implemented using at least one software routine stored on a machine readable medium, which routine when executed by a processor so processes said data acquired by said depth imaging system.

17. The method of claim 10, wherein step (c) employs height detection masking such that an object having height greater than a threshold normal relative to said road plane is identified as being of potential concern.

18. A method to reduce occurrence of false-positive in detection of objects in the path of a motor vehicle moving on a road surface and to identify which of said objects are of potential concern as obstacles, the method comprising the following steps:

(a) providing said motor vehicle with a time-of-flight (TOF) imaging system that acquires depth data and has a field of view encompassing at least a plane of said road surface representing a portion of a desired detection zone in which an object of potential concern to said motor vehicle may be present such that depth data acquired includes a plane of said road surface, said TOF imaging system including an emitter of optical energy, at least one detector that detects at least a fraction of the emitted said optical energy reflected by at least one of said objects, and means for determining said depth data from a comparison of the emitted said optical energy and said fraction of said optical energy reflected by at least one of said objects;

(b) processing data acquired at step (a) to model said road surface in at least said detection zone so as to define at least one road plane therein;

(c) discerning from data processed at step (b) objects that are of potential concern to said motor vehicle and;

(d) outputting at least one warning signal responsive to identification of an object discerned at step (c) as being of potential concern.

19. The method of claim 18, wherein step (c) includes at least one of (i) height detection masking such that an object having height greater than a threshold normal relative to said road plane is identified as being of potential concern, and (ii) slope detection masking.

20. The method of claim 18, wherein step (b) includes processing data using a RANSAC routine.