SYSTEM AND METHOD FOR USE OF A VEHICLE BACK-UP CAMERA AS A DEAD-RECKONING SENSOR

• US 20100100321A1
• Filed: 10/16/2008
• Published: 04/22/2010
• Est. Priority Date: 10/16/2008
• Status: Active Grant

• Current Assignee: CSR Technology Incorporated (Qualcomm, Inc.)
• Original Assignee: CSR Technology Incorporated (Qualcomm, Inc.)
• Inventors: Koenig, Michael, Brodie, Keith
• Granted Patent: US 8,855,917 B2
• Time in Patent Office: Days
• US Class Current: 701/214
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/04 by terrestrial means G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/10 by using measurements of speed or acceleration G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G01C 21/3461 Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types, segments such as motorways, toll roads, ferries
• CPC Class Codes: G01C 21/3658 Lane guidance
• CPC Class Codes: G01S 19/45 by combining measurements of signals from the satellite radio beacon positioning system with a supplementary measurement
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

Systems and methods are disclosed herein to use a vehicle back-up camera as a cost-effective dead-reckoning sensor in satellite-based vehicle navigation systems. Since the back-up camera may already use a display of the navigation system for display, the data from the back-up camera may be easily obtained and integrated into the navigation system. The data from the camera is received by a navigation receiver wirelessly or through a wired connection. The image is processed to determine the speed, heading, turn-rate of the vehicle to aid the satellite-based navigation system if the satellite signals are inadequate. Thus, enhanced vehicle navigation- performance can be obtained without adding new sensors and/or connecting to a vehicle data bus.

Claims

1. A vehicle navigation system comprising:

a back-up camera mounted on a rear of a vehicle to provide video data;

a video dead-reckoning subsystem to process the video data for generating an estimated motion of the vehicle; and

a satellite-based navigation system to use the estimated motion of the vehicle to aid in generating a navigation solution for the vehicle.

2. The vehicle navigation system of claim 1, wherein the video dead-reckoning subsystem receives the video data from the back-up camera through a wired connection.

3. The vehicle navigation system of claim 1, wherein the video dead-reckoning subsystem receives the video data from the back-up camera through a wireless connection.

4. The vehicle navigation system of claim 1, wherein the video dead-reckoning subsystem further comprises:

a frame capture module to sample the video data in a frame to provide sampled video data;

a memory to store one or more frames of the sampled video data;

one or more correlators to perform image correlation between two frames of the sampled video data to search for one or more displacement vectors; and

a video dead-reckoning state estimator to estimate motion of the vehicle and spatial parameters of the back-up camera from the one or more displacement vectors.

5. The vehicle navigation system of claim 4, wherein the video dead-reckoning subsystem further comprises a downsampling and grayscale conversion module coupled to the frame capture module to down sample the sampled video data and to convert it from a RGB format to a grayscale format.

6. The vehicle navigation system of claim 4, wherein the frame capture module samples the video data at a frame rate.

7. The vehicle navigation system of claim 4, wherein the frame capture module samples the video data in a burst for a short duration.

8. The vehicle navigation system of claim 4, wherein each of the one or more correlators is assigned a corresponding region within each frame of the sampled video data.

9. The vehicle navigation system of claim 8, wherein each of the one or more correlators is configured to perform the image correlation through the acts of:

taking pixel values of the sampled video data within the corresponding region of the correlator from a first frame of the sampled video data;

shifting the corresponding region of the correlator by a displacement to form a shifted region;

taking pixel values of the sampled video data within the shifted region of the correlator from a second frame of the sampled video data;

multiplying a pixel value of each pixel of the sampled video data within the corresponding region of the correlator from the first frame of the sampled video data with a pixel value of a corresponding pixel of the sampled video data within the shifted region of the correlator from the second frame of the sampled video data to form a product; and

summing the product over all pixels taken.

10. The vehicle navigation system of claim 9, wherein a plurality of image correlations is performed for a plurality of displacements of the corresponding region for each correlator.

11. The vehicle navigation system of claim 10, wherein the displacement for shifting the corresponding region of each correlator is restricted to a search space.

12. The vehicle navigation system of claim 11, wherein the displacement vector between two frames of the sampled video data for each correlator is the displacement of the corresponding region corresponding to a largest of the plurality of image correlations.

13. The vehicle navigation system of claim 12, wherein for each correlator the displacement vector found for the image correlation between the first frame and the second frame of the sampled video data is used to modify the search space for a next image correlation.

14. The vehicle navigation system of claim 8, wherein if there is more than one correlator the region corresponding to each correlator does not overlap with the regions corresponding to the remaining correlators.

15. The vehicle navigation system of claim 8, wherein if there is more than one correlator the region corresponding to one correlator overlaps with the region corresponding to another correlator.

16. The vehicle navigation system of claim 4, wherein the video dead-reckoning state estimator comprises:

an image-to-ground transformation module to map a starting position within an image plane to a corresponding starting position in a ground plane in a vehicle frame of reference using an estimate of the spatial parameters of the back-up camera;

a module to estimate an estimated ending position in the ground plane in the vehicle frame of reference using the starting position in the ground plane in the vehicle frame of reference and an estimate of the motion of the vehicle;

a ground-to-image transformation module to map the estimated ending position in the ground plane in the vehicle frame of reference to a corresponding estimated ending position in the image plane using the estimate of the spatial parameters of the back-up camera;

an error module to determine an error between the estimated ending position in the image plane and a measured ending position in the image plane, wherein the measured ending position in the image plane is a displacement of the starting position within the image plane according to the displacement vector; and

an update module to update the estimate of the motion of the vehicle and the estimate of the spatial parameters of the back-up camera using the error to provide an updated estimate of the motion of the vehicle and an updated estimate of the spatial parameters of the back-up camera.

17. The vehicle navigation system of claim 4, wherein the video dead-reckoning state estimator comprises:

an image-to-ground transformation module to map a starting position in an image plane to a corresponding starting position in a ground plane in a vehicle frame of reference using an estimate of the spatial parameters of the back-up camera and to map a measured ending position in the image plane into a corresponding measured ending position in the ground plane in the vehicle frame of reference using the estimate of the spatial parameters of the back-up camera, wherein the measured ending position in the image plane is a displacement of the starting position in the image plane based upon the displacement vector;

a module to estimate an ending position in the ground plane in the vehicle frame of reference using the starting position in the ground plane in the vehicle frame of reference and an estimate of the motion of the vehicle;

an error module to determine an error between the ending position in the ground plane in the vehicle frame of reference and the measured ending position in the ground plane in the vehicle frame of reference; and

an update module to update the estimate of the motion of the vehicle and the estimate of the spatial parameters of the back-up camera using the error to provide an updated estimate of the motion of the vehicle and an updated estimate of the spatial parameters of the back-up camera.

18. The vehicle navigation system of claim 16, wherein the image-to-ground transformation module is configured to perform its mapping as a function of a horizontal view angle, a vertical view angle, a declination angle, a pan angle, a rotation or twist angle, and a vertical offset of the back-up camera.

19. The vehicle navigation system of claim 16, wherein the ground-to-image transformation module is configured to perform its mapping as a function of a horizontal view angle, a vertical view angle, a declination angle, a pan angle, a rotation or twist angle, and a vertical offset of the back-up camera.

20. The vehicle navigation system of claim 16, wherein the updated estimate of the motion of the vehicle and the updated estimate of the spatial parameters of the back-up camera are used for a next iteration of the video dead-reckoning state estimator.

21. The vehicle navigation system of claim 16, wherein the estimate of the motion of the vehicle and the spatial parameters of the back-up camera from a first displacement vector is used to estimate the motion of the vehicle and the spatial parameters of the back-up camera for a second displacement vector.

22. The vehicle navigation system of claim 16, wherein an output from the satellite-based navigation system is fed back to calibrate the video dead-reckoning state estimator.

23. A method of providing dead-reckoning data for a vehicle from a back-up camera to aid a satellite-based navigation system, comprising:

receiving video data from the back-up camera;

sampling the video data in a first frame;

sampling the video data in a second frame;

correlating the video data in the first frame with the video data in the second frame to search for a displacement vector;

estimating motion data of the vehicle from the displacement vector; and

providing the motion data to the satellite-based navigation system.

24. The method of claim 23, further comprising estimating spatial parameters of the back-up camera from the displacement vector.

25. The method of claim 23, wherein correlating the video data in the first frame with the video data in the second frame comprises:

taking pixel values of the video data from a region of the first frame;

shifting the region by a displacement to form a shifted region in the second frame;

taking pixel values of the video data from the shifted region of the second frame;

multiplying the pixel values for each pixel of the video data from the region of the first frame with the pixel values of a corresponding pixel of the video data from the shifted region of the second frame to form a product; and

summing the product over all pixels from the region of the first frame and the shifted region of the second frame to form a correlation.

26. The method of claim 25, wherein correlating the video data in the first frame with the video data in the second frame is performed for a plurality of displacements to generate a plurality of correlations.

27. The method of claim 26, wherein the displacement vector is the displacement corresponding to a largest of the plurality of correlations.

28. The method of claim 23, wherein estimating motion data of the vehicle from the displacement vector comprises:

transforming a starting position in an image plane to a corresponding starting position in a ground plane in a vehicle frame of reference;

estimating an estimated ending position in the ground plane in the vehicle frame of reference using the starting position in the ground plane in the vehicle frame of reference and an estimate of the motion of the vehicle;

transforming the estimated ending position in the ground plane in the vehicle frame of reference to a corresponding estimated ending position in the image plane;

estimating an error between the estimated ending position in the image plane and a measured ending position in the image plane to provide an estimation error, wherein the measured ending position in the image plane is a displacement of the starting position in the image plane according to the displacement vector; and

updating the estimate of the motion of the vehicle based upon the estimation error.

29. The method of claim 23, wherein estimating motion data of the vehicle from the displacement vector comprises:

transforming a starting position in an image plane to a corresponding starting position in a ground plane in a vehicle frame of reference;

estimating an estimated ending position in the ground plane in the vehicle frame of reference using the starting position in the ground plane in the vehicle frame of reference and an estimate of the motion of the vehicle;

transforming a measured ending position in the image plane to a corresponding measured ending position in the ground plane in the vehicle frame of reference, wherein the measured ending position in the image plane is a displacement of the starting position in the image plane according to the displacement vector;

estimating an error between the estimated ending position in the ground plane in the vehicle frame of reference and the measured ending position in the ground plane in the vehicle frame of reference to provide an estimation error; and

updating the estimate of the motion of the vehicle based upon the estimation error.