METHOD FOR STITCHING IMAGES CAPTURED BY MULTIPLE VEHICULAR CAMERAS

• US 20200101899A1
• Filed: 11/25/2019
• Published: 04/02/2020
• Est. Priority Date: 02/27/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Rathi, Ghanshyam, Faraji, Hilda, Gupta, Nikhil, Traub, Christian, Schaffner, Michael, Pflug, Goerg
• Granted Patent: US 10,780,827 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A method for stitching images captured by multiple vehicular cameras includes disposing a plurality of cameras at the vehicle so as to have respective fields of view exterior the vehicle. Image data captured by first and second cameras of the plurality of cameras is processed to detect an object present in an overlapping portion of the fields of view of the first and second cameras. Image data captured by the first and second cameras is stitched, via processing provided captured image data, to form stitched images. Stitching of captured image data is adjusted responsive to determination of a difference between a characteristic of a feature of a detected object as captured by the first camera and the characteristic of the feature of the detected object as captured by the second camera in order to mitigate misalignment of stitched images.

Claims

1. A method for stitching images captured by multiple vehicular cameras, said method comprising:

disposing a plurality of cameras at a vehicle so as to have respective fields of view exterior the vehicle;

wherein the disposed plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of the left-side camera partially overlaps the field of view of the front camera and partially overlaps the field of view of the rear camera;

wherein the field of view of the right-side camera partially overlaps the field of view of the front camera and partially overlaps the field of view of the rear camera;

capturing image data by the plurality of cameras and providing captured image data to a processor of the vehicle;

processing by the processor the provided captured image data;

detecting, via processing by the processor of the provided captured image data, an object present in an overlapping portion of the fields of view of first and second cameras of the plurality of cameras;

determining, via processing by the processor of the provided captured image data, a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the first and second cameras;

stitching, via processing by the processor of the provided captured image data, image data captured by the first and second cameras to form stitched images; and

adjusting stitching of captured image data responsive to a determination of a difference between the determined characteristic of the feature of the detected object as captured by the first camera and the determined characteristic of the feature of the detected object as captured by the second camera in order to mitigate misalignment of stitched images.

2. The method of claim 1, wherein stitching, via processing by the processor of the provided captured image data, of captured image data forms seamless stitched images.

3. The method of claim 1, wherein the first camera comprises the front camera or the rear camera, and wherein the second camera comprises the left-side camera or the right-side camera.

4. The method of claim 3, wherein processing by the processor of provided captured image data comprises processing by the processor multiple frames of captured image data (i) to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and (ii) to determine differences in the determined characteristic of the feature of the detected object over multiple frames of captured image data.

5. The method of claim 1, wherein the determined characteristic of the feature of the detected object comprises one selected from the group consisting of (i) a vertical characteristic of the detected object and (ii) a horizontal characteristic of the detected object.

6. The method of claim 1, wherein the determined characteristic of the feature of the detected object comprises at least one selected from the group consisting of (i) a curb at a side of a road on which the vehicle is being driven, (ii) a mailbox being approached and passed by the vehicle and (iii) a post at a side of a road on which the vehicle is being driven.

7. The method of claim 1, wherein adjusting stitching of captured image data is at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

8. The method of claim 7, comprising determining via the processor a change in distance to the detected object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

9. The method of claim 7, wherein the determination of a change in pitch or roll of the vehicle is made at least in part responsive to a suspension system of the vehicle.

10. The method of claim 1, wherein adjusting stitching of captured image data is at least in part responsive to a non-image-based sensor of the vehicle.

11. The method of claim 10, comprising determining, via the non-image-based sensor, a distance to the detected object exterior the vehicle, and determining, via the processor, and responsive to an output of the non-image-based sensor, a change in distance to the detected object that is indicative of a change in pitch or roll of the vehicle.

12. A method for stitching images captured by multiple vehicular cameras, said method comprising:

disposing a plurality of cameras at a vehicle so as to have respective fields of view exterior the vehicle;

wherein the disposed plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of the left-side camera partially overlaps the field of view of the front camera and partially overlaps the field of view of the rear camera;

wherein the field of view of the right-side camera partially overlaps the field of view of the front camera and partially overlaps the field of view of the rear camera;

capturing image data by the plurality of cameras and providing captured image data to a processor of the vehicle;

processing by the processor the provided captured image data;

detecting, via processing by the processor of the provided captured image data, an object present in an overlapping portion of the fields of view of first and second cameras of the plurality of cameras;

determining, via processing by the processor of the provided captured image data, a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the first and second cameras;

stitching, via processing by the processor of the provided captured image data, image data captured by the first and second cameras to form stitched images; and

adjusting stitching of captured image data responsive to a determination of a difference between the determined characteristic of the feature of the detected object as captured by the first camera and the determined characteristic of the feature of the detected object as captured by the second camera in order to mitigate misalignment of stitched images due to road surface out-of-flatness.

13. The method of claim 12, wherein stitching, via processing by the processor of the provided captured image data, of captured image data forms seamless stitched images.

14. The method of claim 12, wherein the first camera comprises the front camera or the rear camera, and wherein the second camera comprises the left-side camera or the right-side camera.

15. The method of claim 14, wherein processing by the processor of provided captured image data comprises processing by the processor multiple frames of captured image data (i) to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and (ii) to determine differences in the determined characteristic of the feature of the detected object over multiple frames of captured image data.

16. The method of claim 12, wherein the determined characteristic of the feature of the detected object comprises one selected from the group consisting of (i) a vertical characteristic of the detected object and (ii) a horizontal characteristic of the detected object.

17. The method of claim 12, wherein the determined characteristic of the feature of the detected object comprises at least one selected from the group consisting of (i) a curb at a side of a road on which the vehicle is being driven, (ii) a mailbox being approached and passed by the vehicle and (iii) a post at a side of a road on which the vehicle is being driven.

18. The method of claim 12, wherein adjusting stitching of captured image data is at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

19. The method of claim 18, comprising determining via the processor a change in distance to the detected object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

20. The method of claim 18, wherein the determination of a change in pitch or roll of the vehicle is made at least in part responsive to a suspension system of the vehicle.

21. The method of claim 12, wherein adjusting stitching of captured image data is at least in part responsive to a non-image-based sensor of the vehicle.

22. The method of claim 21, comprising determining, via the non-image-based sensor, a distance to the detected object exterior the vehicle, and determining, via the processor, and responsive to an output of the non-image-based sensor, a change in distance to the detected object that is indicative of a change in pitch or roll of the vehicle.

23. A method for stitching images captured by multiple vehicular cameras, said method comprising:

disposing a plurality of cameras at a vehicle so as to have respective fields of view exterior the vehicle;

wherein the disposed plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of the left-side camera partially overlaps the field of view of the front camera and partially overlaps the field of view of the rear camera;

wherein the field of view of the right-side camera partially overlaps the field of view of the front camera and partially overlaps the field of view of the rear camera;

capturing image data by the plurality of cameras and providing captured image data to a processor of the vehicle;

processing by the processor the provided captured image data;

detecting, via processing by the processor of the provided captured image data, an object present in an overlapping portion of the fields of view of first and second cameras of the plurality of cameras;

determining, via processing by the processor of the provided captured image data, a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the first and second cameras;

stitching, via processing by the processor of the provided captured image data, image data captured by the first and second cameras to form stitched images; and

adjusting stitching of captured image data responsive to a determination of a difference between the determined characteristic of the feature of the detected object as captured by the first camera and the determined characteristic of the feature of the detected object as captured by the second camera in order to mitigate misalignment of stitched images due to vehicle loading.

24. The method of claim 23, wherein stitching, via processing by the processor of the provided captured image data, of captured image data forms seamless stitched images.

25. The method of claim 23, wherein the first camera comprises the front camera or the rear camera, and wherein the second camera comprises the left-side camera or the right-side camera.

26. The method of claim 25, wherein processing by the processor of provided captured image data comprises processing by the processor multiple frames of captured image data (i) to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and (ii) to determine differences in the determined characteristic of the feature of the detected object over multiple frames of captured image data.

27. The method of claim 23, wherein the determined characteristic of the feature of the detected object comprises one selected from the group consisting of (i) a vertical characteristic of the detected object and (ii) a horizontal characteristic of the detected object.

28. The method of claim 23, wherein the determined characteristic of the feature of the detected object comprises at least one selected from the group consisting of (i) a curb at a side of a road on which the vehicle is being driven, (ii) a mailbox being approached and passed by the vehicle and (iii) a post at a side of a road on which the vehicle is being driven.

29. The method of claim 23, wherein adjusting stitching of captured image data is at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

30. The method of claim 29, comprising determining via the processor a change in distance to the detected object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

31. The method of claim 29, wherein the determination of a change in pitch or roll of the vehicle is made at least in part responsive to a suspension system of the vehicle.

32. The method of claim 23, wherein adjusting stitching of captured image data is at least in part responsive to a non-image-based sensor of the vehicle.

33. The method of claim 32, comprising determining, via the non-image-based sensor, a distance to the detected object exterior the vehicle, and determining, via the processor, and responsive to an output of the non-image-based sensor, a change in distance to the detected object that is indicative of a change in pitch or roll of the vehicle.