Three-dimensional around view monitoring system of vehicle and method thereof

• US 10,397,524 B1
• Filed: 05/05/2017
• Issued: 08/27/2019
• Est. Priority Date: 05/18/2016
• Status: Expired due to Fees

• Current Assignee: ULSee Inc.
• Original Assignee: ULSee Inc.
• Inventors: Wu, Yi-Tsung
• Primary Examiner(s): Flynn, Nathan J
• Assistant Examiner(s): Somera, Caroline
• Application Number:
  US15/587,885
• Time in Patent Office: 844 Days
• Field of Search: 348148
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2022/3402 Retractor casings; Mounting thereof
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G08B 13/19628 of wide angled cameras and camera groups, e.g. omni-directional cameras, fish eye, single units having multiple cameras achieving a wide angle view
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A three-dimensional around view monitoring system of a vehicle includes a fisheye image correction setting module, configured to receive fisheye image data generated by photographing a correction plate; correct the fisheye image data into corrected image data to generate a fisheye correction lookup table and rotate and translate radiate vertical reference lines presented as straight lines in the corrected image data in a neighborhood into overlaps having a same radiation angle to generate a rotation and translation lookup table; and a static three-dimensional stitched image generation module, configured to generate a stitched image lookup table according to the rotation and translation lookup table and the fisheye correction lookup table, calculate a pixel value of each pixel in the corrected image data by using a pixel value of each pixel in fisheye image data, and perform image stitching on the neighboring corrected image data to generate static stitched image data.

Claims

1. A three-dimensional around view monitoring method of a vehicle, comprising the following steps:

receiving a plurality of pieces of fisheye image data generated by photographing a correction plate, wherein the correction plate comprises a plurality of horizontal reference lines presented as straight lines and a plurality of radiate vertical reference lines presented as straight lines, and the horizontal reference lines and the radiate vertical reference lines are presented as circular arc lines in the pieces of fisheye image data;

correcting the pieces of fisheye image data comprising the horizontal reference lines presented as circular arc lines and the radiate vertical reference lines presented as circular arc lines into a plurality of pieces of corrected image data comprising the horizontal reference lines presented as straight lines and the radiate vertical reference lines presented as straight lines to generate a fisheye correction lookup table, and acquiring a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in the pieces of corrected image data according to the fisheye correction lookup table;

rotating and translating the radiate vertical reference lines presented as straight lines in the pieces of corrected image data in a neighborhood into overlaps having a same radiation angle to generate a rotation and translation lookup table and acquiring, according to the rotation and translation lookup table and the fisheye correction lookup table, a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in the pieces of corrected image data that have been rotated and translated; and

generating a stitched image lookup table according to the rotation and translation lookup table and the fisheye correction lookup table, acquiring, according to the stitched image lookup table, a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in the pieces of corrected image data, calculating a pixel value of each pixel in the pieces of corrected image data by using the pixel value of each pixel in the pieces of fisheye image data, and performing image stitching on the neighboring pieces of corrected image data to generate a piece of static stitched image data.

2. The method according to claim 1, wherein the step of capturing an image of a correction plate comprises the following steps:

adjusting mounting positions of the plurality of fisheye lenses to enable a place at one eighth of an image bottom of the pieces of fisheye image data generated by photographing, by the fisheye lenses, images is an image of a bottom of a vehicle;

adjusting horizontal positions of the fisheye lenses according to heights of two end points of one of the horizontal reference lines presented as circular arc lines in the pieces of fisheye image data;

adjusting each fisheye lens according to the radiate vertical reference lines presented as circular arc lines in the pieces of fisheye image data, so as to enable two ends, namely, left and right ends, of each fisheye lens to photograph the radiate vertical reference lines of the correction plate; and

generating the pieces of fisheye image data from the images of the correction plate that are photographed by the fisheye lenses.

3. The method according to claim 1, wherein a rotation angle by which the radiate vertical reference lines presented as straight lines of the pieces of corrected image data are rotated ranges from 5°

to 45°

.

4. The method according to claim 1, wherein before the step of performing image stitching, the method comprises the following steps:

acquiring, according to the stitched image lookup table, a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in a default image non-overlapping region and a default image overlapping region of each piece of the corrected image data;

calculating, according to the pixel value of each pixel in the pieces of fisheye image data, a pixel value of each pixel of the image non-overlapping region of the corrected image data, so as to obtain a pixel vale of each pixel in the image non-overlapping region of the static stitched image data; and

calculating, according to the pixel value of each pixel in the pieces of fisheye image data, a pixel value of each pixel of the image overlapping region in the pieces of corrected image data and calculating, according to the calculated pixel value, a pixel value of each pixel in the image overlapping region that performs image overlapping in the pieces of corrected image data, so as to obtain a pixel value of each pixel in the image overlapping region of the static stitched image data.

5. The method according to claim 1, after the step of generating a piece of static stitched image data, the method further comprises the following steps:

receiving the pieces of data of fisheye images that are photographed when the vehicle is traveling;

acquiring, according to the stitched image lookup table, a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in a piece of dynamic stitched image data;

calculating a horizontal gradient absolute value, a vertical gradient absolute value, and an edge direction value of each pixel of an image overlapping region of the pieces of fisheye image data and comparing the horizontal gradient absolute value, the vertical gradient absolute value, and the edge direction value of each pixel of the image overlapping region of the pieces of fisheye image data with a plurality of thresholds, so as to classify each pixel of the image overlapping region of the pieces of fisheye image data into one of a plurality of edge styles;

acquiring, according to an edge style of each pixel of the image overlapping region of the pieces of fisheye image data, a size of an object in the image overlapping region of the neighboring images to determine an image compressing size proportion parameter and an image moving degree parameter of the dynamic stitched image data, so as to compress and move an image of image data that is one of the neighboring images in the dynamic stitched image data;

calculating, according to a pixel value of each pixel of an image non-overlapping region of the pieces of fisheye image data, a pixel value of each pixel of the image non-overlapping region of a piece of dynamic stitched image data; and

multiplying the pixel value of each pixel of the image overlapping region of the image data of the neighboring images that is calculated according to the pixel value of each pixel of the image overlapping region of the pieces of fisheye image data by an average gradient absolute value intensity, so as to obtain a pixel value of each pixel of the image overlapping region of the dynamic stitched image data, wherein edge detection is performed on each pixel of the image overlapping region of the pieces of fisheye image data to obtain the average gradient absolute value intensity.

6. A three-dimensional around view monitoring system of a vehicle, comprising:

a fisheye image correction setting module, configured to receive a plurality of pieces of fisheye image data generated by photographing a correction plate, correct the pieces of fisheye image data comprising a plurality of horizontal reference lines presented as circular arc lines and a plurality of radiate vertical reference lines presented as circular arc lines into a plurality of pieces of corrected image data comprising the horizontal reference lines presented as straight lines and the radiate vertical reference lines presented as straight lines to generate a fisheye correction lookup table, and rotate and translate the radiate vertical reference lines presented as straight lines in the pieces of corrected image data in a neighborhood into overlaps having a same radiation angle to generate a rotation and translation lookup table, wherein the correction plate comprises the horizontal reference lines presented as straight lines and the radiate vertical reference lines presented as straight lines, the horizontal reference lines and the radiate vertical reference lines are presented as circular arc lines in the pieces of fisheye image data, a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in the pieces of corrected image data is acquired according to the fisheye correction lookup table and a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in the pieces of corrected image data that have been rotated and translated is acquired according to the rotation and translation lookup table and the fisheye correction lookup table; and

a static three-dimensional stitched image generation module, configured to generate a stitched image lookup table according to the rotation and translation lookup table and the fisheye correction lookup table, calculate a pixel value of each pixel in the pieces of corrected image data by using the pixel value of each pixel in the pieces of fisheye image data, and perform image stitching on the neighboring pieces of corrected image data to generate a piece of static stitched image data, wherein a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in the pieces of corrected image data is acquired according to the stitched image lookup table.

7. The system according to claim 6, further comprising:

a camera mounting correction module, configured to adjust mounting positions of the plurality of fisheye lenses to enable a place at one eighth of an image bottom of the pieces of fisheye image data generated by photographing, by the fisheye lenses, images is an image of a bottom of a vehicle, adjust horizontal positions of the fisheye lenses according to heights of two end points of one of the horizontal reference lines presented as circular arc lines in the pieces of fisheye image data, and adjust each fisheye lens according to the radiate vertical reference lines presented as circular arc lines in the pieces of fisheye image data, so as to enable two ends, namely, left and right ends, of each fisheye lens to photograph the radiate vertical reference lines of the correction plate; and

an image input and output module, wherein the camera mounting correction module and the fisheye image correction setting module receives, by using the image input and output module, the pieces of fisheye image data transmitted by the fisheye lenses, and the static three-dimensional stitched image generation module transmits the static stitched image data to a data display apparatus by using the image input and output module.

8. The system according to claim 7, further comprising:

a characteristic capturing module, configured to receive the pieces of data of fisheye images that are photographed when the vehicle is traveling, calculate a horizontal gradient absolute value, a vertical gradient absolute value, and an edge direction value of each pixel of an image overlapping region of the pieces of fisheye image data and compare the horizontal gradient absolute value, the vertical gradient absolute value, and the edge direction value of each pixel of the image overlapping region of the pieces of fisheye image data with a plurality of thresholds, so as to classify each pixel of the image overlapping region of the pieces of fisheye image data into one of a plurality of edge styles;

a dynamic parameter adjustment module, configured to acquire, according to an edge style of each pixel of the image overlapping region of the pieces of fisheye image data, a size of an object in the image overlapping region of the neighboring images to determine an image compressing size proportion parameter and an image moving degree parameter of the dynamic stitched image data, and perform edge detection on each pixel of the image overlapping region of the pieces of fisheye image data to obtain an average gradient absolute value intensity; and

a dynamic three-dimensional stitched image generation module, configured to compress and move an image of image data that is one of the neighboring images according to the image compressing size proportion parameter and the image moving degree parameter, calculate, according to a pixel value of each pixel of an image non-overlapping region of the pieces of fisheye image data, a pixel value of each pixel of the image non-overlapping region of the dynamic stitched image data; and

multiply the pixel value of each pixel of the image overlapping region of the image data of the neighboring images that is calculated according to the pixel value of each pixel of the image overlapping region of the pieces of fisheye image data by an average gradient absolute value intensity, so as to obtain a pixel value of each pixel of the image overlapping region of the dynamic stitched image data.

9. The system according to claim 6, wherein a rotation angle by which the fisheye image correction setting module rotates the radiate vertical reference lines presented as straight lines of the pieces of corrected image data ranges from 5°

to 45°

.

10. The system according to claim 6, wherein the static three-dimensional stitched image generation module implements the following functions:

acquiring, according to the stitched image lookup table, a coordinate position of each pixel, corresponding to each pixel in the pieces of fisheye image data, in a default image non-overlapping region and a default image overlapping region of each piece of the corrected image data;

calculating a pixel value, corresponding to the pixel value of each pixel of the image non-overlapping region, of each pixel in the pieces of fisheye image data, so as to obtain a pixel vale of each pixel in the image non-overlapping region of the stitched image data; and

calculating a pixel value, corresponding to the pixel value of each pixel in the pieces of fisheye image data, of each pixel of the image overlapping region in the pieces of corrected image data and calculating, according to the calculated pixel value, a pixel value of each pixel in the image overlapping region that performs image overlapping in the pieces of corrected image data, so as to obtain a pixel value of each pixel in the image overlapping region of the stitched image data.