Vehicular vision system

• US 7,263,209 B2
• Filed: 06/13/2003
• Issued: 08/28/2007
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: Ford Motor Company, SRI International, Inc.
• Original Assignee: Ford Motor Company, Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hill, Ian Gregory, Rieder, Andre, Southall, John Benjamin, Camus, Theodore Armand, Hirvonen, David, Mandelbaum, Robert, Call, Billie Joe
• Primary Examiner(s): Bali, Vikkram
• Application Number:
  US10/461,699
• Publication Number:
  US 20040252862A1
• Time in Patent Office: 1,537 Days
• Field of Search: 358/124, 382/158, 382/104, 382/103, 382/123, 382/171, 382/154, 382/209, 382/217, 348/159, 348/42, 348/147, 348/148, 340/425.5, 340/435, 340/436, 340/901, 340/903, 340/925, 340/944
• US Class Current: 382/104
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A vision system for a vehicle that identifies and classifies objects (targets) located proximate a vehicle. The system comprises a sensor array that produces imagery that is processed to generate depth maps of the scene proximate a vehicle. The depth maps are processed and compared to pre-rendered templates of target objects that could appear proximate the vehicle. A target list is produced by matching the pre-rendered templates to the depth map imagery. The system processes the target list to produce target size and classification estimates. The target is then tracked as it moves near a vehicle and the target position, classification and velocity are determined. This information can be used in a number of ways. For example, the target information may be displayed to the driver, the information may be used for an obstacle avoidance system that adjusts the trajectory or other parameters of the vehicle to safely avoid the obstacle. The orientation and/or configuration of the vehicle may be adapted to mitigate damage resulting from an imminent collision, or the driver may be warned of an impending collision.

Claims

1. A method of performing vehicular vision processing comprising:

producing a depth image of a scene proximate a vehicle, wherein said depth image comprise a two-dimensional array of pixels;

comparing the depth image to a plurality of templates of objects, wherein said comparing comprises differencing each of the pixels in the depth image and each similarly positioned pixel in the template, and if the difference at each pixel is less than a predefined amount, the pixel is deemed a match;

identifying a match between the depth image and at least one template, wherein said identifying comprise summing the number of pixels deemed a match and dividing the sum by a total number of pixels in the template to produce a match score, spatially and/or temporally filtering the match score values to produce a new match score; and

if the new match score is greater than a predefined match score amount, the template is deemed a match; and

adjusting a parameter of the vehicle in response to the match of said template.

2. The method of claim 1 further comprising:

imaging the scene with a stereo camera to produce stereo images; and

processing the stereo images to produce the depth image.

3. The method of claim 1 wherein the comparing step further comprises:

dividing the depth image and template into a first half and a second half;

differencing each of the pixels in the first half of the depth image and each similarly positioned pixel in the first half of the template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a first match;

summing numbers of pixels deemed a first match and dividing the sum by a total number of pixels in the first half of the template to produce a first match score;

differencing each of the pixels in the second half of the depth image and each similarly positioned pixel in the second half of the template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a second match;

summing numbers of pixels deemed a second match and dividing the sum by a total number of pixels in the second half of the template to produce a second match score;

multiplying the first match score with the second match score to determine a final match score;

if the final match score is greater than a predefined match score amount, the template is deemed a match.

4. The method of claim 1 further comprising:

accessing at least one template from a database comprising a plurality of templates.

5. The method of claim 4 wherein the plurality of templates represent objects at varying positions and poses relative to the vehicle.

6. The method of claim 5 wherein at least one template in the plurality of templates is a model of another vehicle.

7. The method of claim 1 further comprising:

receiving information regarding at least one object within the scene from a secondary sensor; and

using the information to limit a number of templates that are compared to the depth image.

8. The method of claim 7 wherein the secondary sensor is at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

9. The method of claim 1 further comprising:

receiving information regarding at least one object within the scene from a secondary sensor; and

using the information validate the match.

10. The method of claim 9 wherein the secondary sensor is at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

11. The method of claim 1 further comprising:

tracking the match across a plurality of depth images to determine a velocity and direction of the match.

12. The method of claim 1 further comprising:

classifying the match with regard to size and shape.

13. The method of claim 1 wherein the parameter of the vehicle comprises at least one parameter selected from the group of an air bag deployment parameter, a vehicle height parameter, a vehicle direction control parameter.

14. The method of claim 1, wherein the comparing step further comprises:

dividing both the depth image and the template into N regions, where each region of the depth image is associated with a corresponding region in the template;

differencing the pixels in the each of the N depth image regions and similarly positioned pixels in the N template regions;

if the difference at a pixel is less than a predefined amount, that pixel is deemed a match;

summing the numbers of pixels in each of the N template regions that are deemed a match, and then dividing that sum by a total number of pixels in each of the N template regions to produce a match score;

multiplying the match scores to determine a final match score; and

if the final match score is greater than a predefined match score amount, the template is deemed a match.

15. The method of claim 1 wherein the depth image is produced from a disparity image.

16. The method of claim 15 wherein the disparity image includes information as to what a pixel in the disparity image is invalid.

17. The method of claim 1, further including the step of:

validating a target based at least partially on the match identified between the depth image and at least one template.

18. The method of claim 17, wherein validating a target further depends on the output of a secondary sensor.

19. Apparatus for performing vehicular vision processing comprising:

a stereo image preprocessor for producing a multi-resolutional disparity image of a scene proximate a vehicle;

a depth map generator for processing the multi-resolutional disparity image to form a depth map, wherein said depth map comprise a two-dimensional array of pixels; and

a target processor for comparing the depth map to a plurality of templates and identifying a match between at least one template within the plurality of templates and the depth map, wherein said comparing comprises differencing each of the pixels in the depth map and each similarly positioned pixel in the template, and if the difference at each pixel is less than a predefined amount, the pixel is deemed a match, and said identifying comprise summing the number of pixels deemed a match and dividing the sum by a total number of pixels in the template to produce a match score, spatially and/or temporally filtering the match score values to produce a new match score and if the new match score is greater than a predefined match score amount, the template is deemed a match.

20. The apparatus of claim 19 wherein the target processor classifies the match.

21. The apparatus of claim 19 further comprises:

a secondary sensor that provides information regarding the scene.

22. The apparatus of claim 21 wherein the secondary sensor comprises at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.

23. A method of performing vehicular vision processing, comprising:

producing a disparity image of a scene proximate a vehicle, wherein said disparity image comprise a two-dimensional array of pixels;

comparing the disparity image to a plurality of templates of objects, wherein said comparing comprises differencing each of the pixels in the disparity image and each similarly positioned pixel in the template, and if the difference at each pixel is less than a predefined amount, the pixel is deemed a match;

identifying a match between the disparity image and at least one template, wherein said identifying comprise summing the number of pixels deemed a match and dividing the sum by a total number of pixels in the template to produce a match score, spatially and/or temporally filtering the match score values to produce a new match score; and

if the new match score is greater than a predefined match score amount, the template is deemed a match; and

adjusting a parameter of the vehicle in response to the match of said template.