Vehicular vision system

• US 7,974,442 B2
• Filed: 08/10/2007
• Issued: 07/05/2011
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company), SRI International, Inc.
• Inventors: Southall, John Benjamin, Rieder, Andre, Hill, Ian Gregory, Call, Billie Joe, Mandelbaum, Robert, Hirvonen, David, Camus, Theodore Armand
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US11/837,114
• Publication Number:
  US 20080159620A1
• Time in Patent Office: 1,425 Days
• Field of Search: 382103-107, 382/209, 382216-219, 382/154, 348113-120, 348143-149, 348169-172
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A vision system for a vehicle that identifies and classifies objects (targets) located proximate a vehicle. The system comprises a sensor array that produces imagery that is processed to generate depth maps of the scene proximate a vehicle. The depth maps are processed and compared to pre-rendered templates of target objects that could appear proximate the vehicle. A target list is produced by matching the pre-rendered templates to the depth map imagery. The system processes the target list to produce target size and classification estimates. The target is then tracked as it moves near a vehicle and the target position, classification and velocity are determined. This information can be used in a number of ways. For example, the target information may be displayed to the driver, the information may be used for an obstacle avoidance system that adjusts the trajectory or other parameters of the vehicle to safely avoid the obstacle. The orientation and/or configuration of the vehicle may be adapted to mitigate damage resulting from an imminent collision, or the driver may be warned of an impending collision.

Claims

1. A method of performing vehicular vision processing comprising:

generating a plurality of image templates, wherein said image templates are generated from a scene not proximate a vehicle, wherein each of said image templates comprise a two dimensional array of pixels, and value of each of the pixel is a function of a specific distance to a point in the scene not proximate the vehicle;

producing an image of a scene proximate a vehicle, wherein said image comprise a two dimensional array of pixels and value of each of the pixel is a function of a specific distance to a point in the scene proximate the vehicle;

comparing said image to said plurality of generated image templates; and

identifying a match between said image and at least one of said plurality of generated image templates.

2. The method of claim 1 wherein the function of the specific distance to a point in the scene not proximate the vehicle is disparity and the function of the specific distance to a point in the scene proximate the vehicle is disparity.

3. The method of claim 1 wherein the function of the specific distance to a point in the scene not proximate the vehicle is depth and the function of the specific distance to a point in the scene proximate the vehicle is depth.

4. The method of claim 1 wherein the point in the scene not proximate the vehicle is a point on an object.

5. The method of claim 1 wherein the point in the scene proximate the vehicle is a point on an object.

6. The method of claim 1 wherein the generating step further comprises rendering at least one object model according to at least one object parameter.

7. The method of claim 6 wherein the object model is a vehicle model.

8. The method of claim 6 wherein the object model is a pedestrian model.

9. The method of claim 6 wherein the object model is a cyclist model.

10. The method of claim 6 wherein said at least one object parameter is a position dimension in a 3D volume relative to the vehicle.

11. The method of claim 6 wherein said at east one object parameter is a pose dimension in a 3D volume relative to the vehicle.

12. The method of claim 6 wherein said at least one object parameter is a position dimension in a 3D volume relative to the vehicle and said at least one other object parameter is of a pose dimension in a 3D volume relative to the vehicle.

13. The method of claim 6 wherein the object model is a 3D model.

14. The method of claim 13 wherein the 3D object model is a cuboid.

15. The method of claim 1 further comprising adjusting a parameter of the vehicle in response to the match of said image template.

16. The method of claim 1 further comprising:

imaging the scene proximate the vehicle with a stereo camera to produce stereo images; and

processing the stereo images to produce the image of a scene proximate a vehicle.

17. The method of claim 1 wherein the comparing step further comprises:

differencing each of the pixels in the image and each similarly positioned pixel in the image template; and

if the difference at each pixel is less than a predefined amount, the pixel is deemed a match.

18. The method of claim 17 wherein the identifying step comprises:

summing the number of pixels deemed a match and dividing the sum by a total number of pixels in the image template to produce a match score; and

if the match score is greater than a predefined match score amount, the image template is deemed a match.

19. The method of claim 18 further comprising:

spatially and/or temporally filtering the match score to produce a new match score; and

if the new match score is greater than a predefined match score amount, the image template is deemed a match.

20. The method of claim 1 wherein the comparing step further comprises:

dividing the image and the image template into a first half and a second half;

differencing each of the pixels in the first half of the image and each similarly positioned pixel in the first half of the image template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a first match;

summing numbers of pixels deemed a first match and dividing the sum by a total number of pixels in the first half of the image template to produce a first match score;

differencing each of the pixels in the second half of the image and each similarly positioned pixel in the second half of the image template;

if the difference at each pixel is less than a predefined amount, the pixel is deemed a second match;

summing numbers of pixels deemed a second match and dividing the sum by a total number of pixels in the second half of the image template to produce a second match score;

multiplying the first match score with the second match score to determine a final match score;

if the final match score is greater than a predefined match score amount, the image template is deemed a match.

21. The method of claim 1 further comprising:

receiving information from a secondary sensor regarding at least one object in the scene proximate the vehicle; and

using the information to limit a number of the image templates that are compared to the image.

22. The method of claim 1 further comprising:

receiving information from a secondary sensor regarding at least one object in the scene proximate the vehicle; and

using the information to validate the match.

23. The method of claim 1 further comprising:

tracking the match across a plurality of images to determine a velocity and direction of the match.

24. The method of claim 1 further comprising:

classifying the match with regard to size and shape.

25. The method of claim 15 wherein the parameter of the vehicle comprises at least one parameter selected from the group comprising of an air bag deployment parameter, a vehicle height parameter, a vehicle direction control parameter.

26. The method of claim 1, wherein the comparing step further comprises:

dividing both the image and the image template into N regions, where each region of the image is associated with a corresponding region in the image template;

differencing the pixels in the each of the N image regions and similarly positioned pixels in the N image template regions;

if the difference at a pixel is less than a predefined amount, that pixel is deemed a match;

summing the numbers of pixels in each of the N image template regions that are deemed a match, and then dividing that sum by a total number of pixels in each of the N image template regions to produce a match score;

multiplying the match scores to determine a final match score; and

if the final match score is greater than a predefined match score amount, the image template is deemed a match.

27. The method of claim 1 wherein said image of a scene proximate a vehicle include information as to which pixels in the image are invalid.

28. The method of claim 27 wherein said image is a disparity image.

29. The method of claim 28 further comprising producing a depth image from the disparity image.

30. The method of claim 1, further including the step of validating a target based at least partially on a match between the image and at least one of the image template.

31. The method of claim 30, wherein validating a target further depends on the output of a secondary sensor.

32. Apparatus for performing vehicular vision processing comprising:

a stereo image preprocessor for producing at least one image of a scene proximate a vehicle, wherein said image comprise a two dimensional array of pixels, and value of each pixel is a function of a specific distance to a point in the scene proximate the vehicle;

a database comprising a plurality of image templates, wherein said image templates are generated from a scene not proximate a vehicle, wherein each of said image templates comprise a two dimensional array of pixels, and value of each of the pixel is a function of a specific distance to point in the scene not proximate the vehicle;

a target processor for comparing said at least one image of a scene proximate a vehicle to said plurality of generated image templates to identify a match between said image and at least one of said plurality of generated image templates.

33. The apparatus of claim 32 wherein the function of the specific distance to a point in the scene not proximate the vehicle is disparity and the function of the specific distance to a point in the scene proximate the vehicle is disparity.

34. The apparatus of claim 32 wherein the function of the specific distance to a point in the scene not proximate the vehicle is depth and the function of the specific distance to a point in the scene proximate the vehicle is depth.

35. The apparatus of claim 32 wherein the point in the scene not proximate the vehicle is a point on an object.

36. The apparatus of claim 32 wherein the point in the scene proximate the vehicle is a point on an object.

37. The apparatus of claim 32 further comprising a depth map generator for processing the at least one image to form a depth map of said scene proximate the vehicle, wherein said depth map comprise a two dimensional array of pixels, said generator configured to generate the plurality of image templates of objects derived from the scene not proximate a vehicle.

38. The apparatus of claim 32 wherein at least one object model is rendered according to at least one object parameter, said object model is rendered to generate said plurality templates of objects.

39. The apparatus of claim 32 wherein the target processor classifies the match.

40. The apparatus of claim 32 further comprises:

a secondary sensor that provides information regarding the scene.

41. The apparatus of claim 40 wherein the secondary sensor comprises at least one sensor selected from a group comprising a radar sensor, an active infrared sensor, a LIDAR sensor, or a SONAR sensor.