SYSTEM AND METHOD OF ESTABLISHING A MULTI-CAMERA IMAGE USING PIXEL REMAPPING

• US 20130250114A1
• Filed: 12/01/2011
• Published: 09/26/2013
• Est. Priority Date: 12/01/2010
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng
• Granted Patent: US 9,900,522 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/306 using a re-scaling of images
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

A camera or vision system for establishing a composite image for displaying in a vehicle includes a first camera and a second camera and a controller. Each camera has a respective field of view that overlaps partially with the respective field of view of the other camera. Each camera has a respective imager for generating a respective preliminary digital image. The cameras together have a combined field of view. The controller is programmed to generate a final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras by using a remapping table to remap selected pixels from each of the preliminary digital images into selected positions of the final composite digital image. A plurality of methods for establishing a composite image are also provided.

Claims

1. A method of establishing a composite image for displaying in a vehicle, comprising:

a) providing a first camera and a second camera, each camera having a field of view;

b) positioning the cameras so that the fields of view of the cameras overlap partially, wherein the cameras together have a combined field of view;

c) recording preliminary digital images from the cameras, each preliminary digital image being made up of a plurality of pixels; and

d) generating a final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras, at least in part by remapping selected pixels from each of the preliminary digital images into selected positions of the final composite digital image.

2. A method as claimed in claim 1, wherein the preliminary digital images each have associated therewith a preliminary apparent camera position and the final composite digital image has associated therewith a final apparent camera position, and wherein the selected pixels from the preliminary digital images are selected so that the final apparent camera position associated with the final composite digital image is higher than the preliminary apparent camera positions associated with the preliminary digital images.

3. A method as claimed in claim 1, wherein the selected pixels from the preliminary digital images are selected so that any misalignment between the overlapping portions of the preliminary digital images is substantially eliminated.

4. A method as claimed in claim 1, wherein the selected pixels from the preliminary digital images are selected so that the final composite digital image is dewarped as compared to each of the preliminary digital images.

5. A method as claimed in claim 1, wherein the preliminary digital images each have associated therewith a preliminary apparent camera position and the final composite digital image has associated therewith a final apparent camera position,

and wherein the selected pixels from the preliminary digital images are selected so that the final apparent camera position associated with the final composite digital image is higher than the preliminary apparent camera positions associated with the preliminary digital images,

and wherein the selected pixels from the preliminary digital images are selected so that any misalignment between the overlapping portions of the preliminary digital images is substantially eliminated,

and wherein the selected pixels from the preliminary digital images are selected so that the final composite digital image is dewarped as compared to each of the preliminary digital images.

6. A method as claimed in claim 1, further comprising:

e) reading an input from a user of the vehicle prior to step d), and

f) selecting a remapping table from a plurality of remapping tables based on the input,

wherein in step d) the final composite digital image is generated using the remapping table selected in step f).

7. A method as claimed in claim 1, wherein step d) is carried out in part by generating pixels in the final composite image by interpolating between values of pixels that are adjacent to each other.

8. A method as claimed in claim 1, further comprising:

e) prior to step d), selecting a subset of each preliminary digital image, and wherein selected pixels from each of the preliminary digital images are selected from the subset of each preliminary digital image.

15. A method as claimed in claim 1, wherein positioning the cameras comprises positioning the cameras at one of (i) a front of the vehicle so that the cameras have respective forward fields of view ahead of the vehicle and (ii) a rear portion of the vehicle so that the cameras have respective forward fields of view ahead of the vehicle.

16. A method as claimed in claim 1, comprising providing third and fourth cameras, wherein the first, second, third and fourth cameras together have a combined field of view that is a 360 degree field of view around the vehicle.

17. A method as claimed in claim 16, wherein positioning the cameras comprises (i) positioning the first camera at a front of the vehicle so that the first camera has a forward field of view ahead of the vehicle, (ii) positioning the second camera at a side of the vehicle so that the second camera has a sideward field of view, (iii) positioning the third camera at another side of the vehicle so that the third camera has a sideward field of view and (iv) positioning the fourth camera at a rear of the vehicle so that the fourth camera has a rearward field of view.

9. A method of establishing a composite image for displaying in a vehicle, comprising:

a) providing a first camera and a second camera, a third camera and a fourth camera, each camera having a field of view, wherein the cameras together have a combined field of view that is a 360 degree field of view around the vehicle;

b) positioning the cameras so that the field of view of each camera overlaps partially with the field of view of two of the other cameras;

c) recording preliminary digital images from the cameras, each preliminary digital image being made up of a plurality of pixels; and

d) generating a final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras by remapping selected pixels from each of the preliminary digital images into selected positions of the final composite digital image,

wherein the preliminary digital images each have associated therewith a preliminary apparent camera viewpoint and the final composite digital image has associated therewith a final apparent camera viewpoint,

and wherein the selected pixels from the preliminary digital images are selected so that the final apparent camera viewpoint associated with the final composite digital image is higher than the preliminary apparent camera viewpoints associated with the preliminary digital images,

and wherein the selected pixels from the preliminary digital images are selected so that any misalignment between the overlapping portions of the preliminary digital images is substantially eliminated,

and wherein the selected pixels from the preliminary digital images are selected so that the final composite digital image is dewarped as compared to each of the preliminary digital images.

18. A method as claimed in claim 9, wherein positioning the cameras comprises (i) positioning the first camera at a front of the vehicle so that the first camera has a forward field of view ahead of the vehicle, (ii) positioning the second camera at a side of the vehicle so that the second camera has a sideward field of view, (iii) positioning the third camera at another side of the vehicle so that the third camera has a sideward field of view and (iv) positioning the fourth camera at a rear of the vehicle so that the fourth camera has a rearward field of view.

10-11. -11. (canceled)

12. A method of generating a remapping table for use in mapping pixels from a plurality of preliminary digital images into a final composite image, comprising:

a) driving a vehicle having a first camera and a second camera thereon, each camera having a field of view that overlaps partially with the field of view of the other camera, each camera having an imager for generating one of the preliminary digital image, wherein the cameras together have a combined field of view, wherein the vehicle further includes a controller;

b) detecting a target feature along the path of the vehicle during driving, using the controller;

c) providing a first preliminary digital image from the first camera, wherein the first preliminary digital image contains a first representation of the target feature at a first point time;

d) determining the position of the first representation of the target feature in the first preliminary digital image;

e) providing a second preliminary digital image from the second camera, wherein the second preliminary digital image contains a second representation of the target feature at a second point time;

f) determining the position of the second representation of the target feature in the second preliminary digital image;

g) comparing the positions of the first and second representations of the target feature; and

h) generating at least one value for the remapping table based on the result of the comparison in step g).

13. A method as claimed in claim 12, further comprising:

i) determining a direction of travel of the first representation of the target feature in the first preliminary digital image at the first point in time;

j) determining a direction of travel of the second representation of the target feature in the second preliminary digital image at the second point in time;

k) comparing the directions of travel of the first and second representations of the target feature; and

l) generating at least one value for the remapping table based on the result of the comparison in step k).

14. A method as claimed in claim 13, further comprising:

m) adjusting the values of selected pixels in the remapping table by progressively smaller amounts based on the comparison in step k).

19. A method as claimed in claim 12, wherein the cameras are positioned at one of (i) a front of the vehicle so that the cameras have respective forward fields of view ahead of the vehicle and (ii) a rear portion of the vehicle so that the cameras have respective forward fields of view ahead of the vehicle.

20. A method as claimed in claim 12, wherein driving a vehicle comprises driving a vehicle having a first camera, a second camera, a third camera and fourth camera thereon.

21. A method as claimed in claim 20, wherein the first, second, third and fourth cameras together have a combined field of view that is a 360 degree field of view around the vehicle.

22. A method as claimed in claim 21, wherein the first camera has a forward field of view forward of the vehicle, (ii) the second camera has a sideward field of view sideward of a side of the vehicle, (iii) the third camera has a rearward field of view rearward of the vehicle and (iv) the fourth camera has a sideward field of view sideward of the other side of the vehicle opposite the second camera.