Processor and processing method for generating a panoramic image for a vehicle

• US 7,782,374 B2
• Filed: 03/02/2006
• Issued: 08/24/2010
• Est. Priority Date: 03/03/2005
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Oizumi, Ken, Akatsuka, Takeshi, Suzuki, Masayasu
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): CHEN, CHIA WEI A
• Application Number:
  US11/366,091
• Publication Number:
  US 20060197761A1
• Time in Patent Office: 1,636 Days
• Field of Search: 348/36, 348/47, 348/48, 345/427
• US Class Current: 348/239
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An image processor is disclosed for a vehicle having a plurality of cameras and a monitor that displays a panoramic image derived from multiple images generated by the plurality of cameras. The processor includes input buffers for storing camera image data captured through respective camera modules and output buffer for storing a panoramic image. The camera image data for generating the panoramic image are retrieved from input buffers. The camera image data are stored in an output buffer using an reconfigured address conversion table. The reconfigured address conversion table is an address conversion table which is obtained by reconfiguring an address conversion table when camera modules are assumed to be free of vehicle installation errors. The reconfiguration is performed on the basis of parameters which are used to correct vehicle installation errors of camera modules.

Claims

1. An image processor for a vehicle having a plurality of cameras and a monitor that displays a panoramic image derived from multiple images generated by the plurality of cameras, the image processor comprising:

input buffers adapted to receive and store camera image data generated by the plurality of cameras;

an output buffer adapted to store panoramic image data for display on the monitor;

a default address conversion table that describes a default correspondence between memory addresses of camera images from an arrangement of cameras assumed to be free of installation errors and memory addresses of a panoramic image generated using the camera images;

a controller adapted to;

detect installation errors in the plurality of cameras based on deviations between a template image and a respective camera image obtained while the vehicle is stationary and at a prescribed position, the template image representing the prescribed position and generated assuming the plurality of cameras are free of installation errors;

prepare a reconfigured address conversion table from the default address conversion table that describes a calibrated correspondence between memory addresses of the respective camera image and memory addresses of a panoramic image generated using the respective camera image to correct for the installation errors;

read the camera image data stored on the input buffers;

translate the camera image data into the panoramic image data using the reconfigured address conversion table; and

store the panoramic image data on the output buffer.

2. The image processor of claim 1, wherein:

the reconfigured address conversion table includes one or more numbers each assigned to a particular camera to designate input data contained in a specified input buffer as having been generated by that particular camera.

3. The image processor of claim 1, wherein the input buffers are part of a single memory unit.

4. The image processor of claim 1, wherein:

the reconfigured address conversion table includes input address information that associates each of a plurality of input addresses in the input buffers with a corresponding output memory address in the output buffer to form at least a portion of the panoramic image; and

the controller is further adapted to read the camera image data located at each of the plurality of input addresses and to store the camera image data so read at the corresponding output memory address in accordance with the input address information.

5. The image processor of claim 4, wherein:

the controller further comprises a table reconfiguration module that corrects the installation errors in the plurality of cameras in the reconfigured address conversion table by shifting the input addresses of the default address conversion table and storing the shifted input address information in the reconfigured address conversion table.

6. The image processor of claim 5, wherein the table reconfiguration module corrects a yaw error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras.

7. The image processor of claim 5, wherein the table reconfiguration module corrects a pitch error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a vertical direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras.

8. The image processor of claim 5, wherein the table reconfiguration module corrects a roll error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a rotational direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras.

9. The image processor of claim 5, wherein the table reconfiguration module corrects a yaw error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras, and corrects a pitch error in the installation of the one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a vertical direction in the image coordinate system.

10. The image processor of claim 5, wherein the table reconfiguration module first corrects a yaw error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras, and corrects a pitch error in the installation of the one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a vertical direction in the image coordinate system;

and then corrects a roll error in the installation of the one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a rotational direction in the image coordinate system.

11. An image processor for a vehicle having a plurality of cameras and a monitor that displays a panoramic image generated from multiple images generated by the plurality of cameras, the image processor comprising:

input buffer means for storing camera image data captured by the cameras;

output buffer means for storing panoramic image data generated from the camera image data retrieved from the input buffer means and for outputting the panoramic image data to the monitor;

a default address conversion table that describes a first correspondence from input memory addresses of camera images from an arrangement of cameras assumed to be free of installation errors to corresponding output memory addresses of a panoramic image generated using the camera images;

means for detecting installation errors in the cameras based on deviations between a template image and a respective camera image obtained when the vehicle is stationary and at a prescribed position, the template image representing the prescribed position and generated assuming the cameras are free of installation errors;

means for preparing a reconfigured address conversion table that describes a second correspondence from input memory addresses of the respective camera image and output memory addresses of a panoramic image generated using the respective camera image to correct for the installation errors; and

means for retrieving the camera image data from the input buffer means and storing the camera image data in the output buffer means according to the reconfigured address conversion table means-to generate the panoramic image data.

12. The image processor of claim 11, wherein:

the reconfigured address conversion table includes one or more camera numbers each assigned to a particular camera to designate input data contained in a specified input buffer as having been generated by that particular camera; and

the reconfigured address conversion table includes input address information for designating input addresses needed for retrieving the camera image data for generating the panoramic image data from the input buffers and output address information for designating output addresses in the output buffer means that contain the camera image data retrieved from the input addresses; and

the retrieving means comprises means for retrieving the camera image data from the input buffers according to the camera numbers and the input address information designated by the reconfigured address conversion table and storing the camera image data in output addresses in the output buffer means according to the output address information designated by the reconfigured address conversion tables.

13. The vehicle image processor of claim 11, wherein the means for preparing the reconfigured address conversion table comprises:

table reconfiguration means for shifting the input addresses of the default address conversion table to correct installation errors of the cameras on the vehicle in order to generate the reconfigured address conversion table.

14. The vehicle image processor of claim 13, wherein the table reconfiguration means corrects for a yaw error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras.

15. The vehicle image processor of claim 13, wherein the table reconfiguration means corrects a pitch error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a vertical direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras.

16. The image processor of claim 13, wherein the table reconfiguration means corrects a roll error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a rotational direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras.

17. The vehicle image processor of claim 13, wherein the table reconfiguration means corrects a yaw error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras, and corrects a pitch error in the installation of the one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a vertical direction in the image coordinate system.

18. The vehicle image processor of claim 13, wherein the table reconfiguration means first corrects a yaw error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras, and corrects a pitch error in the installation of the one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a vertical direction in the image coordinate system;

and then after the yaw error and pitch error are corrected, corrects a roll error in the installation of one of the plurality of cameras in the reconfigured address conversion table by shifting at least one of the plurality of input addresses of the default address conversion table in a direction corresponding to a rotational direction in an image coordinate system represented by the camera image data generated by the one of the plurality of cameras.

19. An image processing method for use in a vehicle having a plurality of cameras and a monitor that displays a panoramic image derived from multiple images generated by the plurality of cameras, comprising:

storing camera image data captured by the cameras on an input buffer;

storing panoramic image data generated from the camera image data retrieved from the input buffer for output to the monitor as a panoramic image display;

determining the existence of an installation error in at least one of the plurality of cameras based on deviations between a template image and a respective camera image in the input buffer obtained while the vehicle is stationary and at a prescribed position, the template image representing the prescribed position and generated assuming the plurality of cameras are free of installation errors;

determining the correction parameter to be applied to a default address conversion table to correct the installation error, the default address conversion table associating a plurality of input addresses of camera images from an arrangement of cameras assumed to be free of installation errors with corresponding output memory addresses of a panoramic image generated using the camera images;

reconfiguring the default address conversion table by shifting memory addresses in the input buffer in accordance with the correction parameter to generate a reconfigured address conversion table; and

retrieving the camera image data from the input buffer and storing the camera image data in the output buffer according to the reconfigured address conversion table to generate the panoramic image data for output to the monitor as the panoramic image display.

20. The image processing method of claim 19, wherein reconfiguring the default address conversion table comprises:

correcting for a yaw error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras.

21. The image processing method of claim 19, wherein reconfiguring the default address conversion table comprises:

correcting for a pitch error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of the default address conversion table in a direction corresponding to a vertical direction in an image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras.

22. The image processing method of claim 19, wherein reconfiguring the default address conversion table comprises:

correcting for a roll error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of the default address conversion table in a direction corresponding to a rotational direction in an image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras.

23. The image processing method of claim 19, wherein reconfiguring the default address conversion table comprises:

correcting for a yaw error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras; and

correcting for a pitch error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of the default address conversion table in a direction corresponding to a vertical direction in the image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras.

24. The image processing method of claim 19, wherein reconfiguring the default address conversion table comprises:

correcting for a yaw error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of the default address conversion table in a direction corresponding to a horizontal direction in an image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras; and

after correcting for the yaw error, correcting for a pitch error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of the default address conversion table in a direction corresponding to a vertical direction in the image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras; and

after correcting for the yaw and pitch errors, correcting for a roll error in the installation of the at least one of the plurality of cameras in the reconfigured address conversion table by shifting the memory addresses in the input buffer of the default address conversion table in a direction corresponding to a rotational direction in the image coordinate system represented by the camera image data generated by the at least one of the plurality of cameras.

25. A method for calibrating an vehicle image processor that incorporates camera image data generated by multiple vehicle-mounted cameras and stored on an input buffer into a panoramic image stored on an output buffer for display on a monitor, comprising:

detecting at least one error in the installation of at least one of the multiple vehicle-mounted cameras based on deviations between a template image and a respective camera image obtained while the vehicle is stationary and at a prescribed position, the template image representing the prescribed position and generated assuming the plurality of cameras are free of installation errors;

creating a reconfigured address conversion table by changing a default address conversion table so that it describes a calibrated association between a different input address in the input buffer with the target output address in the output buffer, the different input address being offset from the selected input address in a direction opposing the at least one detected error in the installation of the at least one vehicle-mounted camera, the default address conversion table describing a default association between at least one selected input address in the input buffer with an output address in the output buffer to create at least a portion of a panoramic image generated using data obtained from the at least one select input address and the default association based on an arrangement of cameras assumed to be free of installation errors; and

loading the reconfigured address conversion table into a memory accessible by the vehicle image processor in a manner that permits the vehicle image processor to retrieve camera image data stored at the different input address and store the retrieved camera image data in the target output address of the output buffer.