Object recognition system

• US 6,963,657 B1
• Filed: 08/17/2000
• Issued: 11/08/2005
• Est. Priority Date: 09/24/1999
• Status: Expired due to Fees

• Current Assignee: Honda Giken Kogyo Kabushiki Kaisha (Honda Motor Company)
• Original Assignee: Honda Giken Kogyo Kabushiki Kaisha (Honda Motor Company)
• Inventors: Saka, Masakazu, Yuhara, Hiromitsu, Aoki, Tomoyoshi, Nishigaki, Morimichi
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US09/640,928
• Time in Patent Office: 1,909 Days
• Field of Search: 382103-104, 382106-108, 356/3, 348/169, 348/118, 318/580, 318/587, 701/23, 701/28, 340/901, 340/903
• US Class Current: 382/106
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

An object recognition system mounted on a vehicle comprises a camera and a controller. The camera captures an image in front of the vehicle. The controller divides the image into a plurality of windows. The controller measures distance to the road surface for each of the plurality of windows and thereby obtains a plurality of distances. The controller estimates a relative inclination of the road surface against the vehicle based on the plurality of distances. The controller judges, for each of the plurality of windows, based on the estimated inclination, whether the object is an obstacle or the road surface and thereby obtains a judgment result. The controller recognizes the object based on the judgment result. The system recognizes objects in front of the vehicle even when the vehicle is lopsided by pitching, rolling or else.

Claims

1. An object recognition system mounted on a vehicle, comprising:

one or more sensors for capturing an image of a road surface;

measuring means for dividing the image into a plurality of windows and measuring distance to the road surface for each of the plurality of the windows;

means for storing, for each of the plurality of the windows, estimated distance to the road surface;

means for comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

inclination estimation means for estimating a relative inclination of the road surface against the vehicle based on the measured distances that are determined to be of the road surface; and

means for modifying the estimated distances based on the inclination estimated by said inclination estimation means.

2. The system of claim 1 wherein said inclination estimation means comprises pitch estimating means for estimating slope of pitching of the vehicle as it travels and roll estimating means for estimating slope of roll of the vehicle as it travels.

3. The system of claim 2, further comprising:

distance estimating means for estimating the distance from each window to the road surface based on the slope of the pitch estimated by the pitch estimating means and the slope of the roll estimated by the roll estimating means; and

estimated distance memory for storing for each of the windows the distance estimated by said distance estimating means.

4. The system of claim 3 further comprising judging means for comparing the distance measured by said measuring means for each window and the estimated distance estimated by said distance estimating means to determine relative to each window represents the road surface;

and

recognition means for recognizing the object based on a result from the judging means.

5. The system of claim 4 wherein said judging means extracts windows that represent object other than the road surface for transfer to said recognition means.

6. The system of claim 2 wherein said pitch estimating means determines pitch angle Ø

according to the equation;

$\tan <br><br><br><br>\mathrm{\varnothing <br><br>}=\frac{\mathrm{n\Sigma <br><br>ZiYi}-\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Zi\Sigma <br><br>Yi}}{{\mathrm{n\Sigma <br><br>Zi}}^2-{\left(\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Zi}\right)}^2}$

where Yi and Zi are y-axis and z-axis positions respectively of the i-th sample and n indicates the number of samples, y-axis being the direction of height of the vehicle and z-axis being the direction of travel of the vehicle.

7. The system of claim 2, wherein said roll estimating means determines roll angle or α

according to the equation;

$\tan <br><br><br><br>\mathrm{\alpha <br><br>}=\frac{\mathrm{n\Sigma <br><br>ZiYi}-\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Xi\Sigma <br><br>Yi}}{{\mathrm{n\Sigma <br><br>Xi}}^2-{\left(\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Xi}\right)}^2}$

where Xi and Yi are x-axis and y-axis positions respectively of the i-th sample and n indicates the number of samples, x-axis being the direction of breadth of the vehicle and y-axis being the direction of height of the vehicle.

8. An object recognition system mounted on a vehicle, comprising:

one or more sensors for capturing an image of an object;

measuring means for measuring a distance to the road surface for each of a plurality of windows dividing the image;

means for storing, for each of the plurality of the windows, estimated distance to the road surface;

means for comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

inclination estimation means for estimating a relative inclination of the road surface against the vehicle based on the plurality of distances that are determined to be of the road surface; and

correction means for correcting positioning error of said one or more sensors based on a long time average of the estimated inclination.

9. The system of claim 8 wherein said inclination estimation means comprises:

pitch estimating means for estimating slope of pitching of the vehicle as it travels; and

roll estimating means for estimating slope of roll of the vehicle as it travels;

wherein said correction means comprises;

distance estimating means for estimating the distance from each window to the road surface based on the slope of pitch estimated by the pitch estimating means and the slope of roll estimated by the roll estimating means;

estimated distance memory for storing for each of the windows the distance estimated by said distance estimating means;

wherein data held by said distance memory is made to change at a large time constant such that secular change of positioning of said one or more sensors is corrected.

10. The system of claim 8 wherein said correction means comprises:

pitch estimating means for estimating slope of pitching of the vehicle as it travels;

roll estimating means for estimating slope of roll of the vehicle as it travels; and

sensor position estimating means for estimating deviation of the position of said one or more sensors from their specified position based on the pitch estimated by said pitch estimating means and the roll estimated by said roll estimating means;

wherein the deviation of the position of said one or more sensors is determined based on moving average of the estimated pitch and the estimated roll.

11. An object recognition method for recognizing an object in front of a vehicle, comprising steps of:

capturing an image in front of the vehicle;

measuring a distance to the road surface for each of a plurality of windows dividing the image and thereby obtaining a plurality of distances;

storing, for each of the plurality of the windows, estimated distance to the road surface;

comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

estimating a relative inclination of the road surface against the vehicle based on the measured distances that are determined to be of the road surface; and

modifying the estimated distances based on the estimated relative inclination.

12. The method of claim 11, further comprising a step, performed between the steps of measuring and inclination estimation, of:

extracting windows having captured the road surface from the plurality of windows, and wherein the step of inclination estimation estimates the inclination utilizing distances of the plurality of distances, the utilized distances corresponding to the extracted windows.

13. The method of claim 12, further comprising a step of:

estimating distances to the road surface respectively for the plurality of windows based on the estimated inclination, wherein windows are extracted based on the estimated distances in the step of extraction.

14. The method of claim 11, further comprising the steps of:

judging, for each of the plurality of windows, based on the estimated inclination, whether the object is an obstacle or the road surface and thereby giving a judgment result;

recognizing the object based on the judgment result;

obtaining, when the object has been judged as an obstacle in the judging step, a relative speed and a relative speed and a relative distance between the vehicle and the obstacle utilizing the image;

sensing possible collision with the obstacle based on at least one of the relative speed and the relative distance; and

performing collision avoidance action.

15. An object recognition system mounted on a vehicle, comprising:

a camera for capturing an image of an object; and

a controller configured to perform the functions of;

dividing the image into a plurality of windows;

measuring a distance to the road surface for each of the plurality of windows and thereby obtaining a plurality of distances;

storing, for each of the plurality of the windows, estimated distance to the road surface;

comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

estimating a relative inclination of the road surface against the vehicle based on the measured distances that are determined to be of the road surface; and

modifying the estimated distances based on the estimated relative inclination.

16. The system of claim 15, wherein the controller further performs functions of:

extracting windows having captured the road surface from the plurality of windows, and

estimating the inclination utilizing distances of the plurality of distances, the utilized distance corresponding to the extracted windows.

17. The system of claim 16 wherein:

the controller further performs functions of;

estimating distances to the road surface respectively for the plurality of windows based on the estimated inclination; and

storing the estimated distances in a memory; and

wherein the function of extracting the windows is based on the estimated distances stored in the memory.

18. The system of claim 15, wherein the controller further performs function of:

judging, for each of the plurality of windows, based on the estimated inclination, whether the object is an obstacle or the road surface and thereby obtaining a judgment result;

recognizing the object based on the judgment result;

calculating, when the object has been judged as an obstacle, a relative speed and a relative distance between the vehicle and the obstacle utilizing the image; and

sensing possible collision with the obstacle based on at least one of the relative speed and the relative distance and performing a collision avoidance action.

19. An object recognition system mounted on a vehicle, comprising:

one or more sensors for capturing an image of a road surface;

measuring means for dividing the image into a plurality of windows and measuring distance to the road surface for each of the plurality of the windows;

means for storing, for each of the plurality of the windows, estimated distance to the road surface, the initial values of said estimated distance having been calculated under a condition that the vehicle is positioned parallel to the road surface;

means for comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

inclination estimation means for estimating a relative inclination of the road surface against the vehicle based on the measured distances that are determined to be of the road surface; and

means for modifying the estimated distances based on the inclination estimated by said inclination estimation means.

20. The system of claim 19 wherein said inclination estimation means comprises pitch estimating means for estimating slope of pitching of the vehicle as it travels and roll estimating means for estimating slope of roll of the vehicle as it travels.

21. The system of claim 20, further comprising:

distance estimating means for estimating the distance from each window to the road surface based on the slope of the pitch estimated by the pitch estimating means and the slope of the roll estimated by the roll estimating means; and

estimated distance memory for storing for each of the windows the distance estimated by said distance estimating means.

22. The system of claim 21 further comprising judging means for comparing the distance measured by said measuring means for each window and the estimated distance estimated by said distance estimating means to determine relative to each window represents the road surface;

and

recognition means for recognizing the object based on a result from the judging means.

23. The system of claim 22 wherein said judging means extracts windows that represent object other than the road surface for transfer to said recognition means.

24. The system of claim 19 wherein said pitch estimating means determines pitch angle Ø

according to the equation;

$\tan <br><br><br><br>\mathrm{\varnothing <br><br>}=\frac{\mathrm{n\Sigma <br><br>ZiYi}-\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Zi\Sigma <br><br>Yi}}{{\mathrm{n\Sigma <br><br>Zi}}^2-{\left(\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Zi}\right)}^2}$

where Yi and Zi are y-axis and z-axis positions respectively of the i-th sample and n indicates the number of samples, y-axis being the direction of height of the vehicle and z-axis being the direction of travel of the vehicle.

25. The system of claim 19, wherein said roll estimating means determines roll angle or α

according to the equation;

$\tan <br><br><br><br>\mathrm{\alpha <br><br>}=\frac{\mathrm{n\Sigma <br><br>ZiYi}-\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Xi\Sigma <br><br>Yi}}{{\mathrm{n\Sigma <br><br>Xi}}^2-{\left(\mathrm{\Sigma <br><br>}<br><br><br><br>\mathrm{Xi}\right)}^2}$

where Xi and Yi are x-axis and y-axis positions respectively of the i-th sample and n indicates the number of samples, x-axis being the direction of breadth of the vehicle and y-axis being the direction of height of the vehicle.

26. An object recognition system mounted on a vehicle, comprising:

one or more sensors for capturing an image of an object;

measuring means for measuring a distance to the road surface for each of a plurality of windows dividing the image;

means for storing, for each of the plurality of the windows, estimated distance to the road surface, the initial values of said estimated distance having been calculated under a condition that the vehicle is positioned parallel to the road surface;

means for comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

inclination estimation means for estimating a relative inclination of the road surface against the vehicle based on the plurality of distances that are determined to be of the road surface; and

correction means for correcting positioning error of said one or more sensors based on a long time average of the estimated inclination.

27. The system of claim 26 wherein said inclination estimation means comprises:

pitch estimating means for estimating slope of pitching of the vehicle as it travels; and

roll estimating means for estimating slope of roll of the vehicle as it travels;

wherein said correction means comprises;

distance estimating means for estimating the distance from each window to the road surface based on the slope of pitch estimated by the pitch estimating means and the slope of roll estimated by the roll estimating means;

estimated distance memory for storing for each of the windows the distance estimated by said distance estimating means;

wherein data held by said distance memory is made to change at a large time constant such that secular change of positioning of said one or more sensors is corrected.

28. The system of claim 26 wherein said correction means comprises:

pitch estimating means for estimating slope of pitching of the vehicle as it travels;

roll estimating means for estimating slope of roll of the vehicle as it travels; and

sensor position estimating means for estimating deviation of the position of said one or more sensors from their specified position based on the pitch estimated by said pitch estimating means and the roll estimated by said roll estimating means;

wherein the deviation of the position of said one or more sensors is determined based on moving average of the estimated pitch and the estimated roll.

29. An object recognition method for recognizing an object in front of a vehicle, comprising steps of:

capturing an image in front of the vehicle;

measuring a distance to the road surface for each of a plurality of windows dividing the image and thereby obtaining a plurality of distances;

storing, for each of the plurality of the windows, estimated distance to the road surface, the initial values of said estimated distance having been calculated under a condition that the vehicle is positioned parallel to the road surface;

comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

estimating a relative inclination of the road surface against the vehicle based on the measured distances that are determined to be of the road surface; and

modifying the estimated distances based on the estimated relative inclination.

30. The method of claim 29, further comprising a step, performed between the steps of measuring and inclination estimation, of:

extracting windows having captured the road surface from the plurality of windows, and wherein the step of inclination estimation estimates the inclination utilizing distances of the plurality of distances, the utilized distances corresponding to the extracted windows.

31. The method of claim 30, further comprising a step of:

estimating distances to the road surface respectively for the plurality of windows based on the estimated inclination, wherein windows are extracted based on the estimated distances in the step of extraction.

32. The method of claim 29, further comprising the steps of:

judging, for each of the plurality of windows, based on the estimated inclination, whether the object is an obstacle or the road surface and thereby giving a judgment result;

recognizing the object based on the judgment result;

obtaining, when the object has been judged as an obstacle in the judging step, a relative speed and a relative speed and a relative distance between the vehicle and the obstacle utilizing the image;

sensing possible collision with the obstacle based on at least one of the relative speed and the relative distance; and

performing collision avoidance action.

33. An object recognition system mounted on a vehicle, comprising:

a camera for capturing an image of an object; and

a controller configured to perform the functions of;

dividing the image into a plurality of windows;

measuring a distance to the road surface for each of the plurality of windows and thereby obtaining a plurality of distances;

storing, for each of the plurality of the windows, estimated distance to the road surface, the initial values of said estimated distance having been calculated under a condition that the vehicle is positioned parallel to the road surface;

comparing, for each of the plurality of the windows, the measured distance with the estimated distance to determine if the measured distance belongs to the road surface;

estimating a relative inclination of the road surface against the vehicle based on the measured distances that are determined to be of the road surface; and

modifying the estimated distances based on the estimated relative inclination.

34. The system of claim 33, wherein the controller further performs functions of:

extracting windows having captured the road surface from the plurality of windows, and

estimating the inclination utilizing distances of the plurality of distances, the utilized distance corresponding to the extracted windows.

35. The system of claim 34 wherein:

the controller further performs functions of;

estimating distances to the road surface respectively for the plurality of windows based on the estimated inclination; and

storing the estimated distances in a memory; and

wherein the function of extracting the windows is based on the estimated distances stored in the memory.

36. The system of claim 33, wherein the controller further performs function of:

judging, for each of the plurality of windows, based on the estimated inclination, whether the object is an obstacle or the road surface and thereby obtaining a judgment result;

recognizing the object based on the judgment result;

calculating, when the object has been judged as an obstacle, a relative speed and a relative distance between the vehicle and the obstacle utilizing the image; and

sensing possible collision with the obstacle based on at least one of the relative speed and the relative distance and performing a collision avoidance action.