MULTI-CAMERA DYNAMIC TOP VIEW VISION SYSTEM

• US 20140247352A1
• Filed: 02/27/2014
• Published: 09/04/2014
• Est. Priority Date: 02/27/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Rathi, Ghanshyam, Faraji, Hilda, Gupta, Nikhil, Traub, Christian, Schaffner, Michael, Pflug, Goerg
• Granted Patent: US 10,179,543 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A dynamic image stitching system for stitching images captured by multiple cameras of a vision system of a vehicle includes a first camera disposed at a vehicle and having a first field of view exterior the vehicle and a second camera disposed at the vehicle and having a second field of view exterior the vehicle. The first and second fields of view at least partially overlap. A processor is operable to process image data captured by the first and second cameras. The processor processes captured image data to determine characteristics of features or objects present in the overlapping region of the first and second fields of view. The processor is operable to adjust a stitching algorithm responsive to a determination of a difference between a characteristic of a feature as captured by the first camera and the characteristic of the feature as captured by the second camera.

Claims

1. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system of a vehicle, said dynamic image stitching system comprising:

a first camera disposed at a vehicle and having a first field of view exterior the vehicle;

a second camera disposed at the vehicle and having a second field of view exterior the vehicle;

wherein said first and second fields of view at least partially overlap;

a processor operable to process image data captured by said first and second cameras;

wherein said processor processes captured image data to determine characteristics of features present in the overlapping region of said first and second fields of view;

wherein said processor is operable, via a stitching algorithm, to stitch captured image data to form a stitched image;

a display screen operable to display the stitched image for viewing by a driver of the vehicle when normally operating the vehicle; and

wherein said processor is operable to adjust said stitching algorithm responsive to a determination of a difference between a characteristic of a feature as captured by said first camera and the characteristic of the feature as captured by said second camera.

2. The dynamic image stitching system of claim 1, wherein said processor is operable, via said stitching algorithm, to stitch captured image data to form a substantially seamless stitched image.

3. The dynamic image stitching system of claim 1, wherein said processor is operable to determine an object present in the overlapping region of said first and second fields of view, and wherein said processor compares a characteristic of the determined object in image data captured by said first camera with a corresponding characteristic of the determined object in image data captured by said second camera, and wherein said processor adjusts said stitching algorithm responsive to a determination of a difference between the characteristic of the determined object as captured by said first camera and the characteristic of the determined object as captured by said second camera.

4. The dynamic image stitching system of claim 1, wherein said processor processes captured image data over multiple frames of captured image data to track the feature as its image moves through the overlapping region of said first and second fields of view, and wherein said processor determines differences in the characteristic of the feature over multiple frames of captured image data.

5. The dynamic image stitching system of claim 1, wherein the characteristic of the feature comprises one of (i) a generally vertical characteristic and (ii) a generally horizontal characteristic.

6. The dynamic image stitching system of claim 1, wherein said processor processes captured image data to determine whether the characteristic of the feature as captured by said first camera is misaligned with the characteristic of the feature as captured by said second camera, and wherein said processor adjusts said stitching algorithm responsive to said determination.

7. The dynamic image stitching system of claim 1, wherein the characteristic of the feature comprises at least one of (i) a curb at the side of the road on which the vehicle is being driven, (ii) a lane boundary marking at the side of the road on which the vehicle is being driven, (iii) an edge region of the road on which the vehicle is being driven, (iv) a mailbox being approached and passed by the vehicle, (v) a post at the side of the road on which the vehicle is being driven, (vi) a road marking of a road surface on which the vehicle is being driven and (vii) a parking line of a road surface on which the vehicle is being driven.

8. The dynamic image stitching system of claim 1, wherein said processor is operable to adjust said stitching algorithm responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle and in said first and second fields of view.

9. The dynamic image stitching system of claim 8, wherein said processor determines a change in distance to an object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

10. The dynamic image stitching system of claim 8, wherein the determination of a change in vehicle pitch or roll is made at least in part responsive to a suspension system of the vehicle.

11. The dynamic image stitching system of claim 1, wherein said processor is operable to adjust said stitching algorithm at least in part responsive to a non-image-based sensor of the vehicle.

12. The dynamic image stitching system of claim 11, wherein said non-image-based sensor determines a distance to an object exterior the vehicle and wherein said processor, responsive to an output of said non-image-based sensor, determines a change in distance to the object that is indicative of a change in pitch or roll of the vehicle.

13. The dynamic image stitching system of claim 1, comprising a third camera disposed at the vehicle and having a third field of view exterior the vehicle and wherein said third and second fields of view at least partially overlap.

14. The dynamic image stitching system of claim 13, comprising a fourth camera disposed at the vehicle and having a fourth field of view exterior the vehicle and wherein said fourth and third fields of view at least partially overlap and wherein said fourth and first fields of view at least partially overlap.

15. The dynamic image stitching system of claim 14, wherein (i) said first camera comprises a front camera and said first field of view comprises a field of view forward of the vehicle, (ii) said second camera comprises a side camera and said second field of view comprises a field of view sideward of the vehicle, (iii) said third camera comprises a rear camera and said third field of view comprises a field of view rearward of the vehicle and (iv) said fourth camera comprises a side camera and said fourth field of view comprises a field of view sideward of the vehicle at an opposite side of the vehicle from said second field of view.

16. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system of a vehicle, said dynamic image stitching system comprising:

a front camera disposed at a vehicle and having a first field of view forward of the vehicle;

a side camera disposed at a driver side of the vehicle and having a second field of view generally sideward of the vehicle;

wherein said first and second fields of view at least partially overlap;

a processor operable to process image data captured by said front and side cameras;

wherein said processor processes captured image data to determine characteristics of features present in the overlapping region of said first and second fields of view;

wherein said processor is operable, via a stitching algorithm, to stitch captured image data to form a stitched image;

a display screen operable to display the stitched image for viewing by a driver of the vehicle when normally operating the vehicle;

wherein said processor is operable to adjust said stitching algorithm responsive to a determination of a difference between a characteristic of a feature as captured by said front camera and the characteristic of the feature as captured by said side camera; and

wherein said processor processes captured image data to determine whether the characteristic of the feature as captured by said front camera is misaligned with the characteristic of the feature as captured by said side camera, and wherein said processor adjusts said stitching algorithm responsive to said determination.

17. The dynamic image stitching system of claim 16, wherein said processor processes captured image data over multiple frames of captured image data to track the feature as its image moves through the overlapping region of said first and second fields of view, and wherein said processor determines differences in the characteristic of the feature over multiple frames of captured image data.

18. The dynamic image stitching system of claim 16, wherein said processor is operable to adjust said stitching algorithm responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle and in said first and second fields of view.

19. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system of a vehicle, said dynamic image stitching system comprising:

a front camera disposed at a vehicle and having a first field of view forward of the vehicle;

a driver-side camera disposed at a driver side of the vehicle and having a second field of view generally sideward of the vehicle;

a rear camera disposed at a vehicle and having a third field of view rearward of the vehicle;

a passenger-side camera disposed at a passenger side of the vehicle and having a fourth field of view generally sideward of the vehicle;

wherein said first and second fields of view at least partially overlap;

wherein said second and third fields of view at least partially overlap;

wherein said third and fourth fields of view at least partially overlap;

wherein said fourth and first fields of view at least partially overlap;

a processor operable to process image data captured by said front, driver-side, rear and passenger-side cameras;

wherein said processor processes captured image data to determine characteristics of features present in the overlapping regions of said fields of view;

wherein said processor is operable, via a stitching algorithm, to stitch captured image data to form a stitched image;

a display screen operable to display the stitched image for viewing by a driver of the vehicle when normally operating the vehicle;

wherein, with a feature present in an overlapping region of said fields of view of two of said cameras, said processor is operable to adjust said stitching algorithm responsive to a determination of a difference between a characteristic of the feature as captured by one of said two cameras and the characteristic of the feature as captured by the other of said two cameras; and

wherein said processor processes captured image data to determine whether the characteristic of the feature as captured by said one of said two cameras is misaligned with the characteristic of the feature as captured by the other of said two cameras, and wherein said processor adjusts said stitching algorithm responsive to said determination.

20. The dynamic image stitching system of claim 19, wherein said processor is operable to adjust said stitching algorithm responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle and in said first and second fields of view.