MULTI-CAMERA DYNAMIC TOP VIEW VISION SYSTEM

• US 20190143896A1
• Filed: 01/14/2019
• Published: 05/16/2019
• Est. Priority Date: 02/27/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Rathi, Ghanshyam, Faraji, Hilda, Gupta, Nikhil, Traub, Christian, Schaffner, Michael, Pflug, Goerg
• Granted Patent: US 10,486,596 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A dynamic image stitching system for stitching images captured by cameras of a vision system for a vehicle includes a plurality of cameras disposed at the vehicle and having respective fields of view exterior the vehicle. The field of view of each side camera partially overlaps the field of view of a front camera and partially overlaps the field of view of a rear camera. Image data captured by the plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras. The processor stitches, via a stitching algorithm, image data captured by the cameras. The processor adjusts the stitching algorithm responsive to a determination of a difference between a characteristic of the feature as captured by one of the two cameras and the characteristic of the feature as captured by another of the two cameras.

Claims

1. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system for a vehicle, said dynamic image stitching system comprising:

a plurality of cameras disposed at a vehicle and having respective fields of view exterior the vehicle;

wherein said plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of said left-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein the field of view of said right-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein image data captured by said plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras of said plurality of cameras and to determine a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the two cameras;

wherein said processor stitches, via a stitching algorithm, image data captured by said plurality of cameras to form stitched images;

wherein said processor adjusts said stitching algorithm responsive to a determination of a difference between the characteristic of the feature of the detected object as captured by a first camera of the two cameras and the characteristic of the feature of the detected object as captured by a second camera of the two cameras; and

wherein said processor, via adjusting the stitching algorithm, mitigates misalignment of the stitching of image data due to vehicle loading.

2. The dynamic image stitching system of claim 1, wherein said processor stitches, via said stitching algorithm, captured image data to form a seamless stitched image.

3. The dynamic image stitching system of claim 1, wherein the first camera comprises said front camera or said rear camera and the second camera comprises said left-side camera or said right-side camera.

4. The dynamic image stitching system of claim 3, wherein said processor processes captured image data over multiple frames of captured image data to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and wherein said processor determines differences in the characteristic of the feature of the detected object over multiple frames of captured image data.

5. The dynamic image stitching system of claim 1, wherein the characteristic of the feature of the detected object comprises one selected from the group consisting of (i) a vertical characteristic of the detected object and (ii) a horizontal characteristic of the detected object.

6. The dynamic image stitching system of claim 1, wherein the characteristic of the feature of the detected object comprises at least one selected from the group consisting of (i) a curb at a side of a road on which the vehicle is being driven, (ii) a mailbox being approached and passed by the vehicle and (iii) a post at a side of a road on which the vehicle is being driven.

7. The dynamic image stitching system of claim 1, wherein said processor adjusts said stitching algorithm at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

8. The dynamic image stitching system of claim 7, wherein said processor determines a change in distance to the detected object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

9. The dynamic image stitching system of claim 7, wherein the determination of a change in pitch or roll of the vehicle is made at least in part responsive to a suspension system of the vehicle.

10. The dynamic image stitching system of claim 1, wherein said processor adjusts said stitching algorithm at least in part responsive to a non-image-based sensor of the vehicle.

11. The dynamic image stitching system of claim 10, wherein said non-image-based sensor determines a distance to the detected object exterior the vehicle, and wherein said processor, responsive to an output of said non-image-based sensor, determines a change in distance to the detected object that is indicative of a change in pitch or roll of the vehicle.

12. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system for a vehicle, said dynamic image stitching system comprising:

a plurality of cameras disposed at a vehicle and having respective fields of view exterior the vehicle;

wherein said plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of said left-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein the field of view of said right-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein image data captured by said plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras of said plurality of cameras and to determine a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the two cameras;

wherein said processor stitches, via a stitching algorithm, image data captured by said plurality of cameras to form stitched images;

wherein said processor adjusts said stitching algorithm responsive to a determination of a difference between the characteristic of the feature of the detected object as captured by a first camera of the two cameras and the characteristic of the feature of the detected object as captured by a second camera of the two cameras; and

wherein said processor, via adjusting the stitching algorithm, mitigates misalignment of stitching of image data due to due to road surface out-of-flatness.

13. The dynamic image stitching system of claim 12, wherein said processor stitches, via said stitching algorithm, captured image data to form a seamless stitched image.

14. The dynamic image stitching system of claim 12, wherein the first camera comprises said front camera or said rear camera and the second camera comprises said left-side camera or said right-side camera.

15. The dynamic image stitching system of claim 14, wherein said processor processes captured image data over multiple frames of captured image data to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and wherein said processor determines differences in the characteristic of the feature of the detected object over multiple frames of captured image data.

16. The dynamic image stitching system of claim 12, wherein the characteristic of the feature of the detected object comprises one selected from the group consisting of (i) a vertical characteristic of the detected object and (ii) a horizontal characteristic of the detected object.

17. The dynamic image stitching system of claim 12, wherein the characteristic of the feature of the detected object comprises at least one selected from the group consisting of (i) a curb at a side of a road on which the vehicle is being driven, (ii) a mailbox being approached and passed by the vehicle and (iii) a post at a side of a road on which the vehicle is being driven.

18. The dynamic image stitching system of claim 12, wherein said processor adjusts said stitching algorithm at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

19. The dynamic image stitching system of claim 18, wherein said processor determines a change in distance to the detected object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

20. The dynamic image stitching system of claim 18, wherein the determination of a change in pitch or roll of the vehicle is made at least in part responsive to a suspension system of the vehicle.

21. The dynamic image stitching system of claim 12, wherein said processor adjusts said stitching algorithm at least in part responsive to a non-image-based sensor of the vehicle.

22. The dynamic image stitching system of claim 21, wherein said non-image-based sensor determines a distance to the detected object exterior the vehicle, and wherein said processor, responsive to an output of said non-image-based sensor, determines a change in distance to the detected object that is indicative of a change in pitch or roll of the vehicle.

23. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system for a vehicle, said dynamic image stitching system comprising:

a plurality of cameras disposed at a vehicle and having respective fields of view exterior the vehicle;

wherein said plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of said left-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein the field of view of said right-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein image data captured by said plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras of said plurality of cameras and to determine a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the two cameras;

wherein said processor stitches, via a stitching algorithm, image data captured by said plurality of cameras to form seamless stitched images;

wherein said processor adjusts said stitching algorithm responsive to a determination of a difference between the characteristic of the feature of the detected object as captured by a first camera of the two cameras and the characteristic of the feature of the detected object as captured by a second camera of the two cameras;

wherein the first camera comprises said front camera or said rear camera and the second camera comprises said left-side camera or said right-side camera; and

wherein said processor processes captured image data over multiple frames of captured image data to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and wherein said processor determines differences in the characteristic of the feature of the detected object over multiple frames of captured image data.

24. The dynamic image stitching system of claim 23, wherein said processor adjusts said stitching algorithm at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

25. The dynamic image stitching system of claim 23, wherein said processor adjusts said stitching algorithm at least in part responsive to a non-image-based sensor of the vehicle.