Multi-camera vehicle vision system with image gap fill

• US 10,187,590 B2
• Filed: 10/26/2016
• Issued: 01/22/2019
• Est. Priority Date: 10/27/2015
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Fluegel, Sebastian
• Primary Examiner(s): Torrente, Richard T
• Application Number:
  US15/334,365
• Publication Number:
  US 20170113614A1
• Time in Patent Office: 818 Days
• Field of Search: 348148
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

A vision system of a vehicle includes an image processor and a plurality of cameras disposed at a vehicle and having respective fields of view exterior of the vehicle. The image processor provides warping or projection of a source image into a target image. The image processor scans through source images in a regular order, wherein each element of the source image is included only once in the scan order. The image processor determines the location of the source image element in the target image, with multiple source image elements existing with the same target image location. The image processor includes a set of accumulator registers, with one accumulator register selected by the location in the target image, and the image processor adds each source image element to the selected accumulator register to obtain an average of all source image elements with the same target image location.

Claims

1. A vision system of a vehicle, said vision system comprising:

a plurality of cameras configured to be disposed at a vehicle so as to have respective fields of view exterior of the vehicle, wherein each camera captures image data, and wherein each camera comprises a two dimensional array of photosensing elements;

an image processor receiving, with the cameras disposed at the vehicle, image data captured by the cameras, the image processor processing image data captured by each of the cameras for warping or projection of a source image into a target image for display at a display device that is viewable by a driver of the vehicle;

wherein the display device comprises a two dimensional array of display pixels, the display device operable to display target images derived from image data captured by the cameras;

wherein said vision system provides images for display by the display device that are derived via backward projection, tracing rays from a display plane pixel grid of the display device backwards to one or more source camera pixel grids of one or more of the cameras using warping and unwarping schemes of the one or more cameras and virtual views that are to be generated for display of images by the display device;

wherein the image processor scans through image data of source images in a scan order, and wherein each source image element of the source image is included only once in the scan order, and wherein the image processor determines the location of a source image element in the target image, and wherein multiple source image elements exist with the same target image location; and

wherein the image processor comprises a set of accumulator registers, and wherein one accumulator register is selected by the location of the source image element in the target image, and wherein the image processor adds each source image element to the selected accumulator register to obtain an average of all source image elements within the same target image location.

2. The vision system of claim 1, wherein one accumulator register exists per horizontal or vertical element of the target image, and wherein the accumulator register is selected by the horizontal or vertical position of the source image element in the target image.

3. The vision system of claim 1, wherein gaps in the target image exist where no source image element is located, and wherein the gaps are between target image elements where source image elements are located, and wherein the image processor detects the gaps while scanning through the source image, and wherein the image processor comprises a memory where a list of detected gaps is maintained, and wherein the image processor uses the list of detected gap in an independent processing step to fill the gaps.

4. The vision system of claim 3, wherein the gap fill values are calculated from one or more values of the surrounding valid image elements.

5. The vision system of claim 4, wherein the gap is expressed as a polygon, where the polygon corners are defined by the location of surrounding valid image elements, and wherein the gap is processed line by line or column by column of target image elements.

6. The vision system of claim 5, wherein in a first step, the image processor calculates the two intersection points between the current line or column and the edges of the gap polygon, and wherein in a second step, the image processor calculates the values of each intersection point from the values of the two corner points of the crossed polygon edge, and wherein in a third step, the image processor calculates the values of the target image elements between the intersections from the values of the intersection points.

7. The vision system of claim 3, wherein multiple source images are warped and transformed into the same target image, where the source images represent the same scene from different views.

8. The vision system of claim 7, wherein in a first step, all source images are processed independently into partial target images, and wherein in a second step, the gaps in the target image are filled, and wherein in a third step, all partial target images are blended into the final target image.

9. The vision system of claim 8, wherein the first step is executed on independent entities of the image processor and the second step is executed on a common entity of the image processor.

10. The vision system of claim 3, wherein a source image is partially or completely visible multiple times in the target image, and wherein in a first step, all views of the source image are processed independently into partial target image, and wherein in a second step, the gaps in the target image are filled, and wherein in a third step, all partial target images are blended into the final target image.

11. The vision system of claim 10, wherein the first step is executed on independent entities of the image processor and the second step is executed on a common entity of the image processor.

12. A vision system of a vehicle, said vision system comprising:

a plurality of cameras configured to be disposed at a vehicle so as to have respective fields of view exterior of the vehicle, wherein each camera captures image data, and wherein each camera comprises a two dimensional array of photosensing elements;

an image processor receiving, with the cameras disposed at the vehicle, image data captured by the cameras, the image processor processing image data captured by each of the cameras for warping or projection of a source image into a target image for display at a display device that is viewable by a driver of the vehicle;

wherein the display device comprises a two dimensional array of display pixels, the display device operable to display target images derived from image data captured by the cameras;

wherein said vision system provides images for display by the display device that are derived via backward projection, tracing rays from a display plane pixel grid of the display device backwards to one or more source camera pixel grids of one or more of the cameras using warping and unwarping schemes of the one or more cameras and virtual views that are to be generated for display of images by the display device;

wherein the image processor scans through image data of source images in a scan order, and wherein each source image element of the source image is included only once in the scan order, and wherein the image processor determines the location of a source image element in the target image, and wherein multiple source image elements exist with the same target image location;

wherein the image processor comprises a set of accumulator registers, and wherein one accumulator register is selected by the location of the source image element in the target image, and wherein the image processor adds each source image element to the selected accumulator register to obtain an average of all source image elements within the same target image location;

wherein one accumulator register exists per horizontal or vertical element of the target image, and wherein the accumulator register is selected by the horizontal or vertical position of the source image element in the target image; and

wherein gaps in the target image exist where no source image element is located, and wherein the gaps are between target image elements where source image elements are located, and wherein the image processor detects the gaps while scanning through the source image, and wherein the image processor comprises a memory where a list of detected gaps is maintained, and wherein the image processor uses the list of detected gap in an independent processing step to fill the gaps.

13. The vision system of claim 12, wherein the gap fill values are calculated from one or more values of the surrounding valid image elements.

14. The vision system of claim 13, wherein the gap is expressed as a polygon, where the polygon corners are defined by the location of surrounding valid image elements, and wherein the gap is processed line by line or column by column of target image elements.

15. The vision system of claim 14, wherein in a first step, the image processor calculates the two intersection points between the current line or column and the edges of the gap polygon, and wherein in a second step, the image processor calculates the values of each intersection point from the values of the two corner points of the crossed polygon edge, and wherein in a third step, the image processor calculates the values of the target image elements between the intersections from the values of the intersection points.

16. The vision system of claim 12, wherein multiple source images are warped and transformed into the same target image, where the source images represent the same scene from different views.

17. The vision system of claim 16, wherein in a first step, all source images are processed independently into partial target images, and wherein in a second step, the gaps in the target image are filled, and wherein in a third step, all partial target images are blended into the final target image.

18. The vision system of claim 17, wherein the first step is executed on independent entities of the image processor and the second step is executed on a common entity of the image processor.

19. A vision system of a vehicle, said vision system comprising:

a plurality of cameras configured to be disposed at a vehicle so as to have respective fields of view exterior of the vehicle, wherein each camera captures image data, and wherein each camera comprises a two dimensional array of photosensing elements;

wherein the plurality of cameras are part of a surround view system of the vehicle and wherein the plurality of cameras at least comprise a forward viewing camera, a rearward viewing camera, a driver side viewing camera and a passenger side viewing camera;

an image processor receiving, with the cameras disposed at the vehicle, image data captured by the cameras, the image processor processing image data captured by each of the cameras for warping or projection of a source image into a target image for display at a display device that is viewable by a driver of the vehicle;

wherein the display device comprises a two dimensional array of display pixels, the display device operable to display target images derived from image data captured by the cameras;

wherein said vision system provides images for display by the display device that are derived via backward projection, tracing rays from a display plane pixel grid of the display device backwards to one or more source camera pixel grids of one or more of the cameras using warping and unwarping schemes of the one or more cameras and virtual views that are to be generated for display of images by the display device;

wherein the image processor scans through image data of source images in a scan order, and wherein each source image element of the source image is included only once in the scan order, and wherein the image processor determines the location of a source image element in the target image, and wherein multiple source image elements exist with the same target image location;

wherein the image processor comprises a set of accumulator registers, and wherein one accumulator register is selected by the location of the source image element in the target image, and wherein the image processor adds each source image element to the selected accumulator register to obtain an average of all source image elements within the same target image location;

wherein gaps in the target image exist where no source image element is located, and wherein the gaps are between target image elements where source image elements are located, and wherein the image processor detects the gaps while scanning through the source image, and wherein the image processor comprises a memory where a list of detected gaps is maintained, and wherein the image processor uses the list of detected gap in an independent processing step to fill the gaps; and

wherein a source image is partially or completely visible multiple times in the target image, and wherein in a first step, all views of the source image are processed independently into partial target image, and wherein in a second step, the gaps in the target image are filled, and wherein in a third step, all partial target images are blended into the final target image.

20. The vision system of claim 19, wherein the first step is executed on independent entities of the image processor and the second step is executed on a common entity of the image processor.