Vehicle tracking system

• US 20040054473A1
• Filed: 07/31/2003
• Published: 03/18/2004
• Est. Priority Date: 09/17/2002
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Shimomura, Noriko
• Granted Patent: US 6,873,912 B2
• Time in Patent Office: Days
• US Class Current: 701/301
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 3/7865 using correlation of the live video image with a stored image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

A vehicle tracking system for a vehicle is configured to track a remote vehicle by template matching between a reference image and an input image obtained by a camera mounted on the vehicle. The position and distance of a remote vehicle is determined by calculating a position and an enlargement/reduction ratio of a portion of the input image that provide a maximum correlation value between the portion of the input image and the reference template that is greater than a threshold value. The reference template is updated by calculating a dispersion of at least one pixel of the reference template with respect to a corresponding pixel in a vehicle image extracted from the input image that provides the maximum correlation value, and by deleting the at least one pixel when the dispersion of the at least one pixel is equal to or greater than a threshold value.

Claims

1. A vehicle tracking system for a vehicle, comprising:

a first camera mounted on the vehicle;

an image memory configured and arranged to store a first and second input images containing a remote vehicle received from the first camera;

a vehicle detecting section configured and arranged to detect the remote vehicle based on the first input image stored in the image memory;

a reference template creating section configured and arranged to extract an image region including the remote vehicle from the first input image to be used as a reference template;

a reference template storing memory configured and arranged to store the reference template and a position in the first input image where the reference template was extracted;

a vehicle tracking processing section configured and arranged to determine a position and an enlargement/reduction ratio for at least a portion of the second input image that provide a maximum correlation value between the portion of the second input image and the reference template with respect to the remote vehicle, while enlarging or reducing the portion of the second input image in consideration of a change in size of the remote vehicle in the second input image;

a processing result outputting section configured and arranged to determine a position of the remote vehicle relative to the vehicle based on the position and the enlargement/reduction ratio of the portion of the second input image that provide the maximum correlation value;

a vehicle image storage memory configured and arranged to consecutively store a vehicle image extracted from the portion of the second input image at the position and the enlargement/reduction ratio that provide the maximum correlation value; and

a reference template updating section configured and arranged to normalize the vehicle image and the reference template, calculate a dispersion of at least one pixel of the reference template with respect to a pixel at a corresponding position in the vehicle image, and update the reference template by deleting the at least one pixel when the dispersion of the at least one pixel is equal to or greater than a threshold value.

2. The vehicle tracking system as recited in claim 1, further comprising

a tracking reliability determining section configured and arranged to compare the maximum correlation value obtained by the vehicle tracking processing section with a threshold value to determine a reliability of vehicle tracking.

3. The vehicle tracking system as recited in claim 1, wherein

the vehicle tracking processing section is further configured and arranged to detect edges of the remote vehicle in the first and second input images and determine the enlargement/reduction ratio to calculate the correlation based on change in distance between the edges in the first and second input images.

4. The vehicle tracking system as recited in claim 1, further comprising

a distance measurement section configured and arranged to obtain distance data indicative of a distance to the remote vehicle,

the vehicle detecting section being further configured and arranged to detect the remote vehicle based on the distance data obtained by the distance measurement device,

the reference template creating section being further configured and arranged to determine the image region including the remote vehicle based on detection of the remote vehicle by the vehicle detecting section,

the reference template storage memory being further configured and arranged to store the distance data obtained when the first input image was captured, and

the vehicle tracking processing section being further configured and arranged to determine the enlargement/reduction ratio based on a ratio of the distance data obtained when the second input image was captured to the distance data obtained when the first input image was captured.

5. The vehicle tracking system as recited in claim 1, further comprising

a second camera mounted on the vehicle;

and

a distance image creating section configured and arranged to determine a distance between the vehicle and the remote vehicle by creating a distance image including a distance value in each pixel obtained by calculating a parallax of the each pixel with respect to the remote vehicle between the first input image and a third input image simultaneously obtained with the first input image by the second camera,

the image memory being further configured and arranged to store the third input image,

the reference template creating section being further configured and arranged such that extraction of the reference template from the image region is based on the distance image,

the vehicle tracking processing section being further configured and arranged to determine the enlargement/reduction ratio based on a ratio of the distance when the second input image was captured to the distance when the first input image was captured, and

the reference template updating section being further configured and arranged to delete at least one pixel that includes the dispersion that is equal to or greater than a threshold value and a distance value that is different from a distance value indicative of the remote vehicle.

6. The vehicle tracking system as recited in claim 1, wherein

the image memory is further configured and arranged to obtain first and second differential images of the first and second input images, respectively;

the reference template creating section is further configured and arranged to create the reference template using the differential image of the first input image;

the vehicle tracking processing section is further configured and arranged to determine the enlargement/reduction ratio by using the first and second differential images to calculate the correlation value of the first and second input images.

7. The vehicle tracking system as recited in claim 2, wherein

the image memory is further configured and arranged to obtain first and second differential images of the first and second input images, respectively;

the reference template creating section is further configured and arranged to create the reference template using the differential image of the first input image;

the vehicle tracking processing section is further configured and arranged to determine the enlargement/reduction ratio by using the first and second differential images to calculate the correlation value of the first and second input images.

8. The vehicle tracking system as recited in claim 3, wherein

the image memory is further configured and arranged to obtain first and second differential images of the first and second input images, respectively;

the reference template creating section is further configured and arranged to create the reference template using the differential image of the first input image;

the vehicle tracking processing section is further configured and arranged to determine the enlargement/reduction ratio by using the first and second differential images to calculate the correlation value of the first and second input images.

9. The vehicle tracking system as recited in claim 4, wherein

the image memory is further configured and arranged to obtain first and second differential images of the first and second input images, respectively;

the reference template creating section is further configured and arranged to create the reference template using the differential image of the first input image;

the vehicle tracking processing section is further configured and arranged to determine the enlargement/reduction ratio by using the first and second differential images to calculate the correlation value of the first and second input images.

10. The vehicle tracking system as recited in claim 1, wherein

the reference template updating section is further configured and arranged to prohibit deleting the at least one pixel when the dispersion of the at least one pixel is relatively small for a prescribed period.

11. The vehicle tracking system as recited in claim 2, wherein

the reference template updating section is further configured and arranged to prohibit deleting the at least one pixel when the dispersion of the at least one pixel is relatively small for a prescribed period.

12. The vehicle tracking system as recited in claim 3, wherein

the reference template updating section is further configured and arranged to prohibit deleting the at least one pixel when the dispersion of the at least one pixel is relatively small for a prescribed period.

13. The vehicle tracking system as recited in claim 4, wherein

the reference template updating section is further configured and arranged to prohibit deleting the at least one pixel when the dispersion of the at least one pixel is relatively small for a prescribed period.

14. The vehicle tracking system as recited in claim 1, wherein

the reference template updating section is further configured and arranged to cease updating the reference template when a number of tracking processing cycles exceeds a prescribed number of cycles and the dispersion of all pixels of the reference template have been less than or equal to a prescribed value for a prescribed amount of time or longer.

15. The vehicle tracking system as recited in claim 2, wherein

the reference template updating section is further configured and arranged to cease updating the reference template when a number of tracking processing cycles exceeds a prescribed number of cycles and the dispersion of all pixels of the reference template have been less than or equal to a prescribed value for a prescribed amount of time or longer.

16. The vehicle tracking system as recited in claim 3, wherein

the reference template updating section is further configured and arranged to cease updating the reference template when a number of tracking processing cycles exceeds a prescribed number of cycles and the dispersion of all pixels of the reference template have been less than or equal to a prescribed value for a prescribed amount of time or longer.

17. The vehicle tracking system as recited in claim 4, wherein

the reference template updating section is further configured and arranged to cease updating the reference template when a number of tracking processing cycles exceeds a prescribed number of cycles and the dispersion of all pixels of the reference template have been less than or equal to a prescribed value for a prescribed amount of time or longer.

18. A method of tracking a remote vehicle by using image processing of a plurality of images photographed by a camera mounted on a main vehicle, comprising

detecting the remote vehicle in a first input image;

obtaining a reference template by extracting an image region containing the remote vehicle from the first input image when the remote vehicle is detected;

tracking the remote vehicle by determining a position of a portion of a second input image containing the remote vehicle and an enlargement/reduction ratio of the portion of the second input image that provide a maximum correlation value between the reference template and the portion of the second input image, while enlarging or reducing the portion of the second input image in consideration of a change in size of the remote vehicle in the second input image;

outputting a distance and a direction of the remote vehicle with respect to the vehicle as a remote vehicle detection result determined based on the position and the enlargement/reduction ratio of the second image at which the maximum correlation value was obtained;

storing at least one vehicle image extracted from the second image at the position and the enlargement/reduction ratio for which the maximum correlation value was obtained;

normalizing the reference template and the at least one vehicle image stored consecutively;

calculating a dispersion of at least one pixel in the reference template by comparing the reference template and the at least one vehicle image at a pixel that corresponds to the at least one pixel in the reference template;

updating the reference template by consecutively deleting the at least one pixel when the dispersion of the at least one pixel is equal to or greater than a threshold value to obtain an updated reference template; and

using the updated reference template to execute the tracking of the remote vehicle.

19. A vehicle tracking system for a vehicle, comprising

image inputting means for inputting a first and second input images;

image storing means for storing the first and second input images obtained in the image inputting means;

remote vehicle detecting means for detecting a remote vehicle in the first input image;

reference template creating means for creating a reference template including the remote vehicle extracted from the first input image based on a detection result in the remote vehicle detecting means;

vehicle tracking means for tracking the remote vehicle by calculating a position and an enlargement/reduction ratio of a portion of the second input image that provide a maximum correlation between the reference template and the second input image with respect to the remote vehicle;

tracking result outputting means for determining and outputting the position and the enlargement/reduction ratio of the portion of the second input image that provide the maximum correlation;

vehicle image storing means for storing a vehicle image extracted from the second input image at the position and the enlargement/reduction ratio that provide the maximum correlation; and

reference template updating means for updating the reference template by normalizing the vehicle image and the reference template, calculating a dispersion of at least one pixel of the reference template with respect to a pixel at a corresponding position in the vehicle image, and deleting the at least one pixel of the reference template when the dispersion of the at least one pixel is equal to or greater than a threshold value.