System and method of establishing a multi-camera image using pixel remapping

• US 9,900,522 B2
• Filed: 12/01/2011
• Issued: 02/20/2018
• Est. Priority Date: 12/01/2010
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng
• Primary Examiner(s): Kalapodas, Dramos I
• Application Number:
  US13/990,902
• Publication Number:
  US 20130250114A1
• Time in Patent Office: 2,273 Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/306 using a re-scaling of images
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

A camera or vision system for establishing a composite image for displaying in a vehicle includes a first camera and a second camera and a controller. Each camera has a respective field of view that overlaps partially with the respective field of view of the other camera. Each camera has a respective imager for generating a respective preliminary digital image. The cameras together have a combined field of view. The controller is programmed to generate a final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras by using a remapping table to remap selected pixels from each of the preliminary digital images into selected positions of the final composite digital image. A plurality of methods for establishing a composite image are also provided.

Claims

1. A method of establishing a composite image for displaying in a vehicle, comprising:

a) providing a first camera and a second camera, each camera having a field of view;

b) positioning the cameras at the vehicle so that the fields of view of the cameras overlap partially, wherein the cameras together have a combined field of view, and wherein the first camera comprises a rear backup camera having a field of view rearward of the vehicle;

c) recording preliminary digital images from the cameras, each preliminary digital image being made up of a plurality of pixels;

d) sequentially performing at least three functions on the preliminary digital images, wherein the at least three functions comprise (i) dewarping the preliminary digital images to generate intermediate dewarped images and storing the pixel remapping used to generate the intermediate dewarped images in a first intermediate remapping table, (ii) viewpoint adjustment of the intermediate dewarped images to generate dewarped and viewpoint-adjusted images and storing the pixel remapping used to generate the dewarped and viewpoint-adjusted images in a second intermediate remapping table, and (iii) offset correction of the dewarped and viewpoint-adjusted images and storing the pixel remapping used to generate the dewarped and viewpoint-adjusted and offset-corrected images in a third intermediate remapping table;

e) generating a combined remapping table by combining remapping information of the first, second and third intermediate remapping tables;

f) generating a final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras, at least in part by remapping selected pixels from each of the preliminary digital images of the first and second cameras into selected positions of the final composite digital image in accordance with the combined remapping table;

g) displaying, at a video display screen during a backup maneuver of the vehicle, intermediate dewarped images derived from applying the pixel remapping of the first intermediate remapping table on the preliminary digital images of the first camera;

h) displaying, at the video display screen, the final composite digital images when parking the vehicle; and

i) wherein the video display screen is operable to display a split view of images.

2. A method as claimed in claim 1, wherein the preliminary digital images each have associated therewith a preliminary apparent camera position and the final composite digital image has associated therewith a final apparent camera position,

and wherein the selected pixels from the preliminary digital images are selected so that the final apparent camera position associated with the final composite digital image is higher than the preliminary apparent camera positions associated with the preliminary digital images.

3. A method as claimed in claim 1, wherein the selected pixels from the preliminary digital images are selected so that any misalignment between the overlapping portions of the preliminary digital images is substantially eliminated.

4. A method as claimed in claim 1, wherein the selected pixels from the preliminary digital images are selected so that the final composite digital image is dewarped as compared to each of the preliminary digital images.

5. A method as claimed in claim 1, wherein the preliminary digital images each have associated therewith a preliminary apparent camera position and the final composite digital image has associated therewith a final apparent camera position,

and wherein the selected pixels from the preliminary digital images are selected so that the final apparent camera position associated with the final composite digital image is higher than the preliminary apparent camera positions associated with the preliminary digital images,

and wherein the selected pixels from the preliminary digital images are selected so that any misalignment between the overlapping portions of the preliminary digital images is substantially eliminated,

and wherein the selected pixels from the preliminary digital images are selected so that the final composite digital image is dewarped as compared to each of the preliminary digital images.

6. A method as claimed in claim 1, further comprising:

j) reading an input from a user of the vehicle prior to step e), and

k) selecting a remapping table from a plurality of remapping tables based on the input,

wherein in step e) the final composite digital image is generated using the remapping table selected in step k).

7. A method as claimed in claim 1, wherein step e) is carried out in part by generating pixels in the final composite image by interpolating between values of pixels that are adjacent to each other.

8. A method as claimed in claim 1, further comprising:

j) prior to step e), selecting a subset of each preliminary digital image, and wherein selected pixels from each of the preliminary digital images are selected from the subset of each preliminary digital image.

13. A method as claimed in claim 1, wherein positioning the cameras comprises positioning the first camera at a rear portion of the vehicle so that the first camera has a field of view rearward of the vehicle.

14. A method as claimed in claim 1, comprising providing third and fourth cameras, wherein the first, second, third and fourth cameras together have a combined field of view that is a 360 degree field of view around the vehicle.

15. A method as claimed in claim 14, wherein positioning the cameras comprises (i) positioning the second camera at a side of the vehicle so that the second camera has a sideward field of view, (ii) positioning the third camera at another side of the vehicle so that the third camera has a sideward field of view and (iii) positioning the fourth camera at a front of the vehicle so that the fourth camera has a forward field of view ahead of the vehicle.

22. The method of claim 1, comprising;

j) adjusting remapping of selected pixels that correspond to the overlapping region of the field of view for the second camera to accommodate the determined misalignment of the first camera; and

k) adjusting the remapping of selected pixels for the second camera by progressively smaller amounts for pixels adjacent to the overlapping region.

9. A method of establishing a composite image for displaying in a vehicle, comprising:

a) providing a first camera and a second camera, a third camera and a fourth camera, each camera having a field of view, wherein the cameras together have a combined field of view that is a 360 degree field of view around the vehicle;

b) positioning the cameras at the vehicle so that the field of view of each camera overlaps partially with the field of view of two of the other cameras, and wherein the field of view of the first camera overlaps partially with the field of view of the second camera and the field of view of the third camera, and wherein the first camera comprises a rear backup camera having a field of view rearward of the vehicle;

c) recording preliminary digital images from the cameras, each preliminary digital image being made up of a plurality of pixels;

d) sequentially performing at least three functions on the preliminary digital images, wherein the at least three functions comprise (i) dewarping the preliminary digital images to generate intermediate dewarped images and storing the pixel remapping used to generate the intermediate dewarped images in a first intermediate remapping table, (ii) viewpoint adjustment of the intermediate dewarped images to generate dewarped and viewpoint-adjusted images and storing the pixel remapping used to generate the dewarped and viewpoint-adjusted images in a second intermediate remapping table, and (iii) offset correction of the dewarped and viewpoint-adjusted images and storing the pixel remapping used to generate the dewarped and viewpoint-adjusted and offset-corrected images in a third intermediate remapping table;

e) generating a combined remapping table by combining remapping information of the first, second and third intermediate remapping tables;

f) generating a final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras by remapping selected pixels from each of the preliminary digital images into selected positions of the final composite digital image in accordance with the combined remapping table,

wherein the preliminary digital images each have associated therewith a preliminary apparent camera viewpoint and the final composite digital image has associated therewith a final apparent camera viewpoint,

and wherein the selected pixels from the preliminary digital images are selected so that the final apparent camera viewpoint associated with the final composite digital image is higher than the preliminary apparent camera viewpoints associated with the preliminary digital images,

and wherein the selected pixels from the preliminary digital images are selected so that any misalignment between the overlapping portions of the preliminary digital images is substantially eliminated,

and wherein the selected pixels from the preliminary digital images are selected so that the final composite digital image is dewarped as compared to each of the preliminary digital images;

g) displaying, at a video display screen during a backup maneuver of the vehicle, intermediate dewarped images derived from applying the pixel remapping of the first intermediate remapping table on the preliminary digital images of the first camera;

h) displaying, at the video display screen, the final composite digital images when parking the vehicle; and

i) wherein the video display screen is operable to display a split view of images.

16. A method as claimed in claim 9, wherein positioning the cameras comprises (i) positioning the second camera at a side of the vehicle so that the second camera has a sideward field of view, (ii) positioning the third camera at another side of the vehicle so that the third camera has a sideward field of view and (iii) positioning the fourth camera at a front of the vehicle so that the fourth camera has a forward field of view ahead of the vehicle.

23. The method of claim 9, comprising:

j) adjusting the remapping of selected pixels for the second and third cameras by progressively smaller amounts for pixels adjacent to the respective overlapping region.

10. A method of generating a remapping table for use in mapping pixels from a plurality of preliminary digital images into a final composite image, comprising:

a) providing a first camera and a second camera at a vehicle such that each camera has a field of view that overlaps partially with the field of view of the other camera, each camera having an imager for generating one of the preliminary digital image, wherein the cameras together have a combined field of view, and wherein the first camera comprises a rear backup camera having a field of view rearward of the vehicle;

b) detecting a target feature along the path of the vehicle while the vehicle is driven, using a controller at the vehicle;

c) providing a first preliminary digital image from the first camera, wherein the first preliminary digital image contains a first representation of the target feature at a first point time;

d) determining the position of the first representation of the target feature in the first preliminary digital image;

e) providing a second preliminary digital image from the second camera, wherein the second preliminary digital image contains a second representation of the target feature at a second point time;

f) determining the position of the second representation of the target feature in the second preliminary digital image;

g) comparing the positions of the first and second representations of the target feature;

h) generating at least one value for the remapping table based on the result of the comparison in step g);

i) sequentially performing at least three functions on the at least one preliminary digital image, wherein the at least three functions comprise (i) dewarping the at least one preliminary digital image to generate at least one intermediate dewarped image and storing the pixel remapping used to generate the intermediate dewarped images in a first intermediate remapping table, (ii) viewpoint adjustment of the at least one intermediate dewarped image to generate at least one dewarped and viewpoint-adjusted image and storing the pixel remapping used to generate the dewarped and viewpoint-adjusted images in a second intermediate remapping table, and (iii) offset correction of the at least one dewarped and viewpoint-adjusted image and storing the pixel remapping used to generate the dewarped and viewpoint-adjusted and offset-corrected images in a third intermediate remapping table;

j) generating a combined remapping table by combining remapping information of the first, second and third intermediate remapping tables;

k) generating at least one final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras by remapping selected pixels from the at least one preliminary digital image into selected positions of the at least one final composite digital image in accordance with the combined remapping table;

l) displaying, at a video display screen during a backup maneuver of the vehicle, intermediate dewarped images derived from applying the pixel remapping of the first intermediate remapping table on the preliminary digital images of the first camera;

m) displaying, at the video display screen, the final composite digital images when parking the vehicle; and

n) wherein the video display screen is operable to display a split view of images.

11. A method as claimed in claim 10, further comprising:

o) determining a direction of travel of the first representation of the target feature in the first preliminary digital image at the first point in time;

p) determining a direction of travel of the second representation of the target feature in the second preliminary digital image at the second point in time;

q) comparing the directions of travel of the first and second representations of the target feature; and

r) generating at least one value for the remapping table based on the result of the comparison in step q).

12. A method as claimed in claim 11, further comprising:

s) adjusting the values of selected pixels in the remapping table by progressively smaller amounts based on the comparison in step q).

17. A method as claimed in claim 10, wherein the cameras are positioned at a rear portion of the vehicle so that the cameras have respective rearward fields of view rearward of the vehicle.

18. A method as claimed in claim 10, comprising providing a third camera and a fourth camera at the vehicle.

19. A method as claimed in claim 18, wherein the first, second, third and fourth cameras are provided at the vehicle such that they together have a combined field of view that is a 360 degree field of view around the vehicle.

20. A method as claimed in claim 19, wherein (i) the second camera has a sideward field of view sideward of a side of the vehicle, (ii) the third camera has a forward field of view forward of the vehicle and (iii) the fourth camera has a sideward field of view sideward of the other side of the vehicle opposite the second camera.

21. The method of claim 10, comprising:

o) adjusting remapping of selected pixels that correspond to the overlapping region of the field of view for the second camera to accommodate the determined misalignment of the first camera; and

p) adjusting the values of selected pixels in the remapping table for the second camera by progressively smaller amounts for pixels adjacent to the overlapping region.