Object recognition apparatus and method

• US 5,638,116 A
• Filed: 09/07/1994
• Issued: 06/10/1997
• Est. Priority Date: 09/08/1993
• Status: Expired due to Term

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Sumitomo Electric Industries Limited (Sumitomo Corporation)
• Inventors: Shimoura, Hiroshi, Tenmoku, Kenji
• Primary Examiner(s): AU, AMELIA MEGAN
• Application Number:
  US08/301,434
• Time in Patent Office: 1,007 Days
• Field of Search: 348/169, 348/170, 348/171, 348/172, 348/113, 348/116, 348/119, 348/135, 348/142, 348/148, 348/149, 348/118, 348/208, 382/106, 382/103, 382/291, 364/424.02, 364/424.01, 364/449, 364/458, 364/460, 364/462, 180/169, 180/167
• US Class Current: 348/118
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An object recognition apparatus that can recognize an object in an image acquired by an on-vehicle camera mounted on a vehicle with high efficiency and accuracy. A recognition processing is carried out in a cut off range, which is cut off from the image, and in which an object to be recognized is supposed to be present. The cut off range is obtained using support information supplied from a position detection device and/or an anti-lock brake device of the vehicle. Since the object in the image changes its position as the vehicle moves, the displacement must be estimated during the object recognition. The estimation requires camera attitude parameters of the on-vehicle camera with respect to the road, and the estimation is performed using support information supplied from the position detection device and/of the anti-lock brake device. This will improve the efficiency and accuracy of obtaining the camera attitude parameters.

Claims

1. An object recognition apparatus mounted on a vehicle comprising:

an on-vehicle camera acquiring an image around the vehicle;

a position detection device including sensors and road map data, said position detection device detecting first support information including a bearing, a position, and a mileage of the vehicle;

means for estimating an image cut off range in the image acquired by said on-vehicle camera on the basis of the first support information supplied from the position detection device, said image cut off range is assumed to include an object to be recognized;

means for recognizing the object in the image cut off range; and

means for calculating a plurality of camera attitude parameters which indicate attitude of said on-vehicle camera with respect to the road.

2. The object recognition apparatus as claimed in claim 1, wherein said means for estimating an image cut off range comprises:

means for estimating, on the basis of the position of the object in the image at a particular time, the position of the object in the image at the next time, the interval between the particular time and the next time being determined in advance; and

means for determining the image cut off range on the basis of the estimated position.

3. The object recognition apparatus as claimed in claim 2, wherein said means for estimating an image cut off range provides the image cut off range with an error range that allows a certain amount of error in determining the image cut off range.

4. The object recognition apparatus as claimed in claim 3, wherein said means for estimating an image cut off range determines said error range on the basis of the difference between the estimated position of the object in the image and an actual position at which the object is actually recognized in the image.

5. The object recognition apparatus as claimed in claim 1, wherein said position detection device has information associated with attributes of the road on which the vehicle is running, and said means for recognizing the object determines the type of the object on the basis of the attributes of the road supplied from the position detection device.

6. The object recognition apparatus as claimed in claim 3, wherein said position detection device provides said means for estimating an image cut off range with information associated with the bearing of the vehicle, and said means for estimating an image cut off range determines the error range in accordance with an amount of change in the bearing of the vehicle.

7. The object recognition apparatus as claimed in claim 3, wherein said position detection device has information associated with the speed of the vehicle, and said means for estimating an image cut off range determines the error range in accordance with the speed of the vehicle.

8. The object recognition apparatus as claimed in claim 7, wherein said means for estimating an image cut off range determines the error range such that the error range is substantially proportional to the speed of the vehicle when the speed of the vehicle is greater than a predetermined value.

9. The object recognition apparatus as claimed in claim 3, wherein said position detection device provides said means for estimating an image cut off range with information associated with the bearing of the vehicle, and said means for estimating an image cut off range determines the error range in accordance with an amount of change in the bearing of the vehicle, and

wherein said means for estimating an image cut off range determines the error range such that the error range is substantially proportional to the change in the bearing of the vehicle when the change of the bearing of the vehicle per unit time or per unit mileage is greater than a predetermined value.

10. The object recognition apparatus as claimed in claim 3, further comprising:

camera attitude parameter generating means for generating camera attitude parameters representing the attitude of the on-vehicle camera with respect to the road; and

reliability obtaining means for obtaining reliability of the camera attitude parameters generated by said camera attitude parameter generating means,

wherein said means for estimating the position of the object estimates the position of the object in the image using the camera attitude parameters, and means for estimating the image cut off range estimates the error range on the basis of the reliability of the camera attitude parameters.

11. The object recognition apparatus as claimed in claim 10, further comprising a brake control device having information for controlling a brake of the vehicle, and controlling the brake on the basis of the information, wherein said reliability obtaining means obtains the reliability on the basis of the information of the brake control device.

12. The object recognition apparatus as claimed in claim 11, wherein said brake control device has information associated with the acceleration of the vehicle, and said reliability obtaining means reduces the reliability when the acceleration of the vehicle is greater than a predetermined value.

13. The object recognition apparatus as claimed in claim 11, wherein said brake control device has information associated with the operation of the brake of the vehicle, and said reliability obtaining means reduces the reliability when the brake of the vehicle is operated.

14. An object recognition apparatus mounted on a vehicle, said apparatus comprising:

an on-vehicle camera acquiring an image around the vehicle;

a position detection device including sensors and road map data, said position detection device detecting first support information including a bearing, a position, and a mileage of the vehicle;

means for estimating an image cut off range in the image acquired by said on-vehicle camera on the basis of the first support information supplied from the position detection device, said image cut off range is assumed to include an object to be recognized; and

means for recognizing the object in the image cut off range,

wherein said means for estimating an image cut off range comprises;

means for estimating, on the basis of the position of the object in the image at a particular time, the position of the object in the image at the next time, the interval between the particular time and the next time being determined in advance; and

means for determining the image cut off range on the basis of the estimated position, said means for estimating an image cut off range providing the image cut off range with an error range that allows a certain amount of error in determining the image cut off range,

wherein said position detection device has information associated with the speed of the vehicle, and said means for estimating an image cut off range determines the error range in accordance with the speed of the vehicle.

15. The object recognition apparatus as claimed in claim 14, wherein said means for estimating an image cut off range determines the error range such that the error range is substantially proportional to the speed of the vehicle when the speed of the vehicle is greater than a predetermined value.

16. An object recognition apparatus mounted on a vehicle, said apparatus comprising:

an on-vehicle camera acquiring an image around the vehicle;

a position detection device including sensors and road map data, said position detection device detecting first support information including a bearing, a position, and a mileage of the vehicle;

means for estimating an image cut off range in the image acquired by said on-vehicle camera on the basis of the first support information supplied from the position detection device, said image cut off range is assumed to include an object to be recognized; and

means for recognizing the object in the image cut off range,

wherein said means for estimating an image cut off range comprises;

means for estimating, on the basis of the position of the object in the image at a particular time, the position of the object in the image at the next time, the interval between the particular time and the next time being determined in advance; and

means for determining the image cut off range on the basis of the estimated position, said means for estimating an image cut off range providing the image cut off range with an error range that allows a certain amount of error in determining the image cut off range, wherein said position detection device provides said means for estimating an image cut off range with information associated with the bearing of the vehicle, and said means for estimating an image cut off range determines the error range in accordance with an amount of change in the bearing of the vehicle, and wherein said means for estimating an image cut off range determines the error range such that the error range is substantially proportional to the change in the bearing of the vehicle when the change of the bearing of the vehicle per unit time or per unit mileage is greater than a predetermined value.

17. An object recognition apparatus mounted on a vehicle, said apparatus comprising:

an on-vehicle camera acquiring an image around the vehicle;

a position detection device including sensors and road map data, said position detection device detecting first support information including a bearing, a position, and a mileage of the vehicle;

means for estimating an image cut off range in the image acquired by said on-vehicle camera on the basis of the first support information supplied from the position detection device, said image cut off range is assumed to include an object to be recognized;

means for recognizing the object in the image cut off range;

camera attitude parameter generating means for generating camera attitude parameters representing the attitude of the on-vehicle camera with respect to the road; and

reliability obtaining means for obtaining reliability of the camera attitude parameters generated by said camera attitude parameter generating means,

wherein said means for estimating the position of the object estimates the position of the object in the image using the camera attitude parameters, and means for estimating the image cut off range estimates the error range on the basis of the reliability of the camera attitude parameters, and wherein said means for estimating an image cut off range comprises;

means for estimating, on the basis of the position of the object in the image at a particular time, the position of the object in the image at the next time, the interval between the particular time and the next time being determined in advance; and

means for determining the image cut off range on the basis of the estimated position, said means for estimating an image cut off range providing the image cut off range with an error range that allows a certain amount of error in determining the image cut of range.

18. The object recognition apparatus as claimed in claim 17, further comprising a brake control device having information for controlling a brake of the vehicle, and controlling the brake on the basis of the information, wherein said reliability obtaining means obtains the reliability on the basis of the information of the brake control device.

19. The object recognition apparatus as claimed in claim 18, wherein said brake control device has information associated with the acceleration of the vehicle, and said reliability obtaining means reduces the reliability when the acceleration of the vehicle is greater than a predetermined value.

20. The object recognition apparatus as claimed in claim 18, wherein said brake control device has information associated with the operation of the brake of the vehicle, and said reliability obtaining means reduces the reliability when the brake of the vehicle is operated.

21. A tracking recognition apparatus performing a tracking recognition of an object on and around a road, on which a vehicle is traveling, on the basis of an image acquired by an on-vehicle camera which picks up an image ahead of or behind the vehicle, said tracking recognition apparatus comprising:

image converting means for converting the image acquired by said on-vehicle camera into a digital image signal at every processing period;

means for generating contours of one or more objects in the image on the basis of the digital image signal supplied from the image converting means;

means for determining coordinates of one or more specified points in the image, said specified points being selected in advance for each object to be recognized so that the position of each object is substantially determined by the specified points;

registration means for registering the specified points to a memory with characteristics of a neighboring image of the specified points;

means for generating camera attitude parameters of the on-vehicle camera, said camera attitude parameters representing the attitude of the on-vehicle camera with respect to the road;

spatial coordinate converting means for converting the coordinates of the specified points registered in said registration means into spatial positions of the specified points by using the camera attitude parameters of the on-vehicle camera;

a position detection device detecting the position of the vehicle, and producing mileage data and bearing data of the vehicle;

first estimating means for estimating spatial positions of the specified points at the next processing period on the basis of the mileage data, the bearing data of the vehicle, and the spatial positions of the specified points obtained by the spatial coordinate converting means;

second estimating means for estimating the coordinates of the specified points at the next processing period in the image associated with the spatial positions;

means for determining an error range around the coordinates of the specified points estimated by said second estimating means when an image of the next processing period is outputted from the image converting means; and

decision means for deciding whether the same specified points are present in the error range in the image of the next processing period on the basis of the characteristics of the neighboring image of the specified points,

wherein if the decision means decides that the same specified points are present, the object is tracked by obtaining the spatial positions of the specified points, whereas if the decision means decides that the same specified points are absent, the registration associated with the object is canceled.

22. The tracking recognition apparatus as claimed in claim 21, wherein if said decision means decides that at least one of the specified points is present in the error range, the tracking of the object is continued, whereas if said decision means decides that none of the specified points is present in the error range, the registration associated with the object is canceled.

23. The tracking recognition apparatus as claimed in claim 21, wherein if said decision means decides that all of the specified points is present in the error range, the tracking of the object is continued, whereas if said decision means decides that at least one of the specified points is absent in the error range, the registration associated with the object is canceled.

24. The tracking recognition apparatus as claimed in claim 21, wherein said registration means stores pattern information for identifying objects to be recognized, and if said decision means decides that the same specified points are present in the error range, matching of the pattern of the object with that of the pattern information registered in said registration means is decided, and if the matching is negated, the registration associated with the object is canceled.

25. The tracking recognition apparatus as claimed in claim 21, wherein said first estimation means obtains spatial positional changes in the specified points on the basis of the spatial positions of the specified points outputted from said spatial position converting means and the spatial positions of the specified points determined at the immediately preceding processing period, and determines that the object associated with the specified points is still if the spatial positional changes are within a predetermined range, or is moving if the spatial positional changes exceed the predetermined range, and

wherein said spatial coordinate converting means obtains the spatial positions of the specified points of the next processing period by considering the spatial positional changes in the specified points if the object associated with the specified points is decided to be moving.

26. The tracking recognition apparatus as claimed in claim 25, wherein said first estimating means decides that the object associated with the specified points is moving if the spatial positional change of one of the specified points exceeds the predetermined range, whereas said first estimation means decides that the object is still if the spatial positional changes of all the specified points are within the predetermined range.

27. The tracking recognition apparatus as claimed in claim 21, wherein said second estimation means obtains, if the spatial positions of the specified points are not obtained in the immediately preceding period, the coordinates of the specified points in the image assuming that the object is still, and said decision means decides whether the same specified points are present in an extended range wider than the error range on the basis of the characteristics of the neighboring image of the specified points registered in the registration means if said decision means decides that the same specified points are not present in the error range.

28. The tracking recognition apparatus as claimed in claim 27, wherein said registration means cancels the registration associated with the object, if the decision means decides that the object is not present in the extended range.

29. An object recognition method for use with a vehicle comprising the steps of:

acquiring an image around the vehicle by an on-vehicle camera;

detecting, by using sensors and road map data, first supporting information including a bearing, a position and a milage of the vehicle;

estimating an image cut off range in said image acquired by said on-vehicle camera, on the basis of the first support information supplied from the position detection device, said image cut off range is assumed to include an object to be recognized;

recognizing the object in the image cut off range; and

calculating a plurality of camera attitude parameters which indicate attitude of said on-vehicle camera with respect to the road.

30. A tracking recognition method performing a tracking recognition of an object on and around a road, on which a vehicle is traveling, on the basis of an image acquired by an on-vehicle camera which picks up an image ahead of or behind the vehicle, said tracking recognition method comprising the steps of:

converting the image acquired by said on-vehicle camera into a digital image signal at every processing period;

generating contours of one or more objects in the image on the basis of the digital image signal;

determining coordinates of one or more specified points in the image, said specified points being selected in advance for each object to be recognized so that the position of each object is substantially determined by the specified points;

registering the specified points to a memory with characteristics of a neighboring image of the specified points;

generating camera attitude parameters of the on-vehicle camera, said camera attitude parameters representing the attitude of the on-vehicle camera with respect to the road;

converting the coordinates of the specified points registered into spatial positions of the specified points by using the camera attitude parameters of the on-vehicle camera;

detecting the position of the vehicle, and producing mileage data and bearing data of the vehicle;

estimating spatial positions of the specified points at the next processing period on the basis of the mileage data, the bearing data of the vehicle, and the spatial positions of the specified points;

estimating the coordinates of the specified points at the next processing period in the image associated with the spatial positions;

determining an error range around the coordinates of the specified points when an image of the next processing period is outputted; and

deciding whether the same specified points are present in the error range in the image of the next processing period on the basis of the characteristics of the neighboring image of the specified points,

wherein if the same specified points are present, the object is tracked by obtaining the spatial positions of the specified points, whereas if the same specified points are absent, the registration associated with the object is canceled.

31. An object recognition apparatus mounted on a vehicle comprising:

an on-vehicle camera acquiring an image around the vehicle;

a position detection device including sensors and road map data, said position detection device detecting first support information including a bearing, a position, and a mileage of the vehicle;

means for estimating an image cut off range in the image acquired by said on-vehicle camera on the basis of the first support information supplied from the position detection device, said image cut off range is assumed to include an object to be recognized; and

means for recognizing the object in the image cut off range, wherein said position detection device has information associated with attributes of the road on which the vehicle is running, and said means for recognizing the object determines the type of the object which is limited in accordance with the attributes of the road supplied from the position detection device.