SYSTEM AND METHOD FOR CORRECTING IMAGE THROUGH ESTIMATION OF DISTORTION PARAMETER

0Associated
Cases 
0Associated
Defendants 
0Accused
Products 
0Forward
Citations 
0
Petitions 
1
Assignment
First Claim
1. A system for correcting an image through estimation of a distortion parameter, the system comprising:
 a distorted image receiver configured to receive a distorted image including one or more measurement targets;
a feature point extractor configured to extract a plurality of feature points from each of the measurement targets;
a feature point classifier configured to compare distances between the plurality of extracted feature points and a center point of the received distorted image with each other and classify the one or more measurement targets as a distorted target and an undistorted target;
a distortion parameter estimator configured to estimate a distortion parameter on the basis of standard deviations of a plurality of feature points of the classified distorted target and undistorted target; and
an image corrector configured to correct the received distorted image on the basis of the estimated distortion parameter.
1 Assignment
0 Petitions
Accused Products
Abstract
Provided are a system and method for correcting an image through estimation of a distortion parameter. The method includes receiving a distorted image including one or more measurement targets, extracting a plurality of feature points from each of the measurement targets, classifying the one or more measurement targets as a distorted target and an undistorted target by comparing distances between the plurality of extracted feature points and a center point of the received distorted image with each other, estimating a distortion parameter on the basis of standard deviations of a plurality of feature points of the classified distorted target and undistorted target, and correcting the received distorted image on the basis of the estimated distortion parameter.
0 Citations
No References
No References
14 Claims
 1. A system for correcting an image through estimation of a distortion parameter, the system comprising:
a distorted image receiver configured to receive a distorted image including one or more measurement targets; a feature point extractor configured to extract a plurality of feature points from each of the measurement targets; a feature point classifier configured to compare distances between the plurality of extracted feature points and a center point of the received distorted image with each other and classify the one or more measurement targets as a distorted target and an undistorted target; a distortion parameter estimator configured to estimate a distortion parameter on the basis of standard deviations of a plurality of feature points of the classified distorted target and undistorted target; and an image corrector configured to correct the received distorted image on the basis of the estimated distortion parameter.  View Dependent Claims (2, 3, 4, 5, 6, 7)
 8. A method of correcting an image through estimation of a distortion parameter, the method comprising:
a distorted image receiving operation of receiving, by a distorted image receiver, a distorted image including one or more measurement targets; a feature point extraction operation of extracting, by a feature point extractor, a plurality of feature points from each of the measurement targets; a feature point classification operation of comparing, by a feature point classifier, distances between the plurality of extracted feature points and a center point of the received distorted image with each other and classifying the one or more measurement targets as a distorted target and an undistorted target; a distortion parameter estimation operation of estimating, by a distortion parameter estimator, a distortion parameter on the basis of standard deviations of a plurality of feature points of the classified distorted target and undistorted target; and an image correction operation of correcting, by an image corrector, the received distorted image on the basis of the estimated distortion parameter.  View Dependent Claims (9, 10, 11, 12, 13, 14)
1 Specification
This work was supported by an Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No.2014000077, Development of global multitarget tracking and event prediction techniques based on realtime largescale video analysis). This work was also supported by an Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korean government (MSIT) (2017000250, Intelligent Defense Boundary Surveillance Technology Using Collaborative Reinforced Learning of Embedded Edge Camera and Image Analysis).
This application claims priority to and the benefit of Korean Patent Application No. 1020180132180, filed on Oct. 31, 2018, the disclosure of which is incorporated herein by reference in its entirety.
The present invention relates to image correction, and more particularly, to a system and method for correcting an image through estimation of a distortion parameter in which features are extracted from a distorted image and then the image is optimally corrected by repeatedly estimating a distortion parameter on the basis of optical characteristics.
The wideangle lens, such as a fisheye lens, has a wider field of view (FOV) than a standard lens, and it is possible to obtain information of a wide area. However, a lens designed with a wide FOV excessively refracts light, and thus lens distortion may occur.
Lens distortion occurs in radial directions and is proportional to the distance from the center of an image. Due to lens distortion, distortion of image information occurs such as deformation of an object and bending of a straight line into a curved line. For this reason, performance of a system which identifies a specific object, analyzes circumstances, and determines a situation using image information may be degraded.
There are several methods for correcting such distortion. Representative distortion correction methods are based on a geometric projection model, a pattern, and estimation of a distortion parameter.
According to the method based on a geometric projection model, distortion is corrected by defining a lens distortion model according to geometric projection of a lens. In this case, a model of lens design and a focal length at the time of capturing the image are necessary to apply the method based on a geometric projection model for distortion correction. However, since it is not possible to accurately know an actual focal distance, accurate distortion correction is not possible.
According to the patternbased method, distortion is corrected with a distortion rate which is estimated by using information on a distorted pattern and an ideal pattern. The patternbased method involves photographing a pattern through each lens, and correction performance is affected by an environment in which the pattern is photographed and by accuracy in extracting pattern information.
According to the method based on estimation of a distortion parameter, distortion is corrected by estimating a distortion parameter on the basis of data learned from information extracted from a distorted image and information extracted from an undistorted image or by estimating a distortion parameter on the basis of a characteristic that distortion occurs in proportion to the distance from the origin. When a distortion parameter is estimated on the basis of learned data, performance varies according to the number of images used for prior learning. When a distortion parameter is estimated by using the characteristic that distortion occurs in proportion to the distance from the origin, it is necessary to separately set a threshold value according to an image size, and thus automatic distortion correction is not possible.
Consequently, it is necessary to develop a technology for optimally correcting distortion without optical experts'"'"' knowledge, such as lens design information, prior information which is used for distortion correction, such as a distortion pattern, and preprocessing technology for extracting pattern information.
Korean Patent Registration No. 101172629
Korean Patent Registration No. 101014572
Japanese Patent Publication No. 2005328570
The present invention is directed to selecting an optimal distortioncorrected image through straightline information of corrected images.
The present invention is directed to objectively correcting distortion on the basis of feature information extracted from a distorted image and optical characteristics without using lens information, pattern information, or learned data.
The present invention is directed to optimally correcting distortion through iterative estimation of a distortion parameter and an iterative distortion correction model.
The present invention is directed to maintaining constant performance by correcting distortion through a distortion parameter estimated on the basis of unique feature information of a measurement target.
The present invention is directed to providing a distortion correction system and method which may be applied to a system to which it is not possible to apply existing distortion correction methods.
The present invention is directed to applying a wide image system to extended application fields, such as a widearea surveillance system and a highend driver supporting system including a 360degree augmented reality (AR) or virtual reality (VR) camera.
Objects of the present invention are not limited to those mentioned above, and other objects which have not been mentioned may be clearly understood by those of ordinary skill in the art from the following descriptions.
According to an aspect of the present invention, there is provided a system for correcting an image through estimation of a distortion parameter, the system including a distorted image receiver configured to receive a distorted image including one or more measurement targets, a feature point extractor configured to extract a plurality of feature points from each of the measurement targets, a feature point classifier configured to compare distances between the plurality of extracted feature points and the center point of the received distorted image and classify the one or more measurement targets as a distorted target and an undistorted target, a distortion parameter estimator configured to estimate a distortion parameter on the basis of standard deviations of a plurality of feature points of the classified distorted target and undistorted target, and an image corrector configured to correct the received distorted image on the basis of the estimated distortion parameter.
The one or more measurement targets may be people, and the plurality of feature points may include a center point of an eye corresponding to an iris of a corresponding person'"'"'s left eye, a center point of an eye corresponding to an iris of the corresponding person'"'"'s right eye, an end point of the corresponding person'"'"'s nose, a left end point of the corresponding person'"'"'s mouth, and a right end point of the corresponding person'"'"'s mouth.
The feature point classifier may classify the one or more measurement targets as a distorted target and an undistorted target according to undistortedtarget standard formulae given by Equations 8 and 9 below:
where L_{I }is an undistorted target, L_{i}* represents a measurement target at the shortest distance from the center point of a distorted image, N_{F }represents the number of measurement targets, μ_{I }represents the average point of a plurality of feature points extracted from a measurement target, and C=(x_{c}, y_{c}) is the center point of the received distorted image.
L_{i}={p_{LE}^{i}, p_{RE}^{i}, p_{N}^{i}, p_{LM}^{i}, p_{RM}^{i}}, and
μ_{i}=⅕(p_{LE}^{i}+p_{RE}^{i}+p_{N}^{i}+p_{LM}^{i}+p_{RM}^{i}), [Equation 9]
where L_{i }represent feature points of a measurement target, μ_{i }represents the average point of a plurality of feature points, p_{LE}^{i }represents the center point of a left eye corresponding to an iris of the left eye, p_{RE}^{i }represents the center point of a right eye corresponding to an iris of the right eye, p_{N}^{i }represents the end point of a nose, p_{LM}^{i }represents the left end point of a mouth, p_{RM}^{i }and represents the right end point of the mouth.
The distortion parameter estimator may calculate the distortion parameter to be estimated according to a distortion parameter formula given by Equation 10 below:
where k^{(j) }represents a j^{th }estimated distortion parameter, N_{F }is the number of measurement targets, σ_{i}^{(j) }represents the standard deviation of feature points of a distorted target, and σ_{i}*^{(j) }represents the standard deviation of feature points of an undistorted target.
The system may further include an iterative corrector configured to output one or more repeatedlycorrected images by repeatedly estimating the distortion parameter and repeatedly correcting the corrected image until the corrected image satisfies a preset condition, and a final correctedimage selector configured to detect straightline information including the number and length of straight lines in the one or more output images and select a final corrected image on the basis of the straightline information.
The iterative corrector may repeatedly correct the corrected image according to iterative correction formulae given by Equations 11 and 12 below:
where {circumflex over (f)}_{u}^{(j+1) }represents a j+1^{th }corrected image, k^{(j) }represents a j^{th }estimated distortion parameter, r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, {circumflex over (f)}_{u}^{(j) }represents the j^{th }corrected image, and N_{I }is the number of iterations.
r^{(j)}=√{square root over ((x_{d}^{(j)}−x_{c})^{2}+(y_{d}^{(j)}−y_{c})^{2})}, [Equation 12]
where r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, x_{d}^{(j) }and y_{d}^{(j) }represent horizontal and vertical coordinates of the j^{th }corrected image, and x_{c }and y_{c }represent coordinates of the center point of a distorted image.
The iterative corrector and the final correctedimage selector may operate according to a cost function given by Equation 14 below:
where {circumflex over (f)}_{u}* is a final distortioncorrected image, {circumflex over (f)}_{u}^{(j*) }represents a j*^{th }final distortioncorrected image, D_{j }represents the total length of all straight lines in a j^{th }repeatedlycorrected image, S_{j }represent the set of straight lines extracted from the j^{th }repeatedlycorrected image, σ_{L}^{(j) }represents the standard deviation of feature points of the j^{th }repeatedlycorrected image, and · the number of elements of the set.
According to another aspect of the present invention, there is provided a method of correcting an image through estimation of a distortion parameter, the method including: a distorted image receiving operation of receiving, by a distorted image receiver, a distorted image including one or more measurement targets; a feature point extraction operation of extracting, by a feature point extractor, a plurality of feature points from each of the measurement targets; a feature point classification operation of comparing, by a feature point classifier, distances between a plurality of extracted feature points and the center point of the received distorted image and classifying the one or more measurement targets as a distorted target and an undistorted target; a distortion parameter estimation operation of estimating, by a distortion parameter estimator, a distortion parameter on the basis of standard deviations of a plurality of feature points of the classified distorted target and undistorted target; and an image correction operation of correcting, by an image corrector, the received distorted image including the distorted target on the basis of the estimated distortion parameter.
The one or more measurement targets may be people, and the plurality of feature points may include a center point of an eye corresponding to an iris of a corresponding person'"'"'s left eye, a center point of an eye corresponding to an iris of the corresponding person'"'"'s right eye, an end point of the corresponding person'"'"'s nose, a left end point of the corresponding person'"'"'s mouth, and a right end point of the corresponding person'"'"'s mouth.
The feature point classification operation may include classifying the one or more measurement targets as a distorted target and an undistorted target according to undistortedtarget standard formulae given by Equations 8 and 9 below:
where L_{I }is an undistorted target, L_{i}* represents a measurement target at the shortest distance from the center point of a distorted image, N_{F }represents the number of measurement targets, μ_{i }the average point of a plurality of feature points extracted from a measurement target, and C=(x_{c}, y_{c}) is the center point of the received distorted image.
L_{i}={p_{LE}^{i}, p_{RE}^{i}, p_{N}^{i}, p_{LM}^{i}, p_{RM}^{i}}, and
μ_{i}=⅕(p_{LE}^{i}+p_{RE}^{i}+p_{N}^{i}+p_{LM}^{i}+p_{RM}^{i}), [Equation 9]
where L_{i }represent feature points of a measurement target, μ_{i }represents the average point of a plurality of feature points, p_{LE}^{i }represents the center point of a left eye corresponding to an iris of the left eye, p_{RE}^{i }represents the center point of a right eye corresponding to an iris of the right eye, p_{N}^{i }represents the end point of a nose, p_{LM}^{i }represents the left end point of a mouth, and p_{RM}^{i }represents the right end point of the mouth.
The distortion parameter estimation operation may include calculating the distortion parameter to be estimated according to a distortion parameter formula given by Equation 10 below:
where k^{(j) }represents a j^{th }estimated distortion parameter, N_{F }is the number of measurement targets, σ_{i}^{(j) }represents the standard deviation of feature points of a distorted target, and σ_{i}*^{(j) }represents the standard deviation of feature points of an undistorted target.
The method may further include an iterative correction operation of repeatedly estimating, by an iterative corrector, the distortion parameter and repeatedly correcting the corrected image until the corrected image satisfies a preset condition and outputting one or more repeatedlycorrected images, and a final correctedimage selection operation of detecting, by a final correctedimage selector, straightline information including the number and length of straight lines in the one or more output images and selecting a final corrected image on the basis of the straightline information.
The iterative correction operation may include repeatedly correcting the corrected image according to iterative correction formulae given by Equations 11 and 12 below:
where {circumflex over (f)}_{u}^{(j+1) }represents a j+1^{th }corrected image, k^{(j) }represents a j^{th }estimated distortion parameter, r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, {circumflex over (f)}_{u}^{(j) }represents the j^{th }corrected image, and N_{I }is the number of iterations.
r^{(j)}=√{square root over ((x_{d}^{(j)}−x_{c})^{2}+(y_{d}^{(j)}−y_{c})^{2})}, [Equation 12]
where r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, x_{d}^{(j) }and y_{d}^{(j) }represent horizontal and vertical coordinates of the j^{th }corrected image, and x_{c }and y_{c }represent coordinates of the center point of a distorted image.
The iterative correction operation and the final correctedimage selection operation may be performed according to a cost function given by Equation 14 below:
where {circumflex over (f)}*_{u }is a final distortioncorrected image, {circumflex over (f)}_{u}^{(j*) }represents a j*^{th }final distortioncorrected image, D_{j }represents the total length of all straight lines in a j^{th }repeatedlycorrected image, S_{j }represent the set of straight lines extracted from the j^{th }repeatedlycorrected image, σ_{L}^{(j) }represents the standard deviation of feature points of the j^{th }repeatedlycorrected image, and · the number of elements of the set.
The above and other objects, features and advantages of the present invention will become more apparent to those of ordinary skill in the art by describing exemplary embodiments thereof in detail with reference to the accompanying drawings, in which:
In order to fully understand the present invention, operational advantages of the present invention, and objects achieved by implementing the present invention, the present invention will be described with reference to the accompanying drawings which illustrate preferred embodiments of the present invention and the content illustrated in the accompanying drawings. Features and advantages of the present invention will be more apparent from the following detailed description based on the accompanying drawings. Prior to this, terms and words used in this specification and claims are to be interpreted in relation to the technical idea of the present invention based on the principle that the inventor can properly define the concept of the term to describe his or her own invention in the best way, and the present invention should be interpreted in terms of meaning and concept. Detailed descriptions about related wellknown functions and configurations that may obscure the subject matter of the present invention will be omitted.
As shown in
Two triangles PQC and pqC are similar to each other and thus have the relationship referred to as perspective projection as shown in Equation 1.
where P=(X, Y, Z) is an arbitrary point in a 3D space, and p=(x, y) is an arbitrary point in a 2D image plane.
Perspective projection is the theoretical basis for most camera systems having a thin lens. On the other hand, in a wideangle or fisheye lens camera, projection from an object point to a projected point is a nonlinear function of an incidence angle of the object as shown in
Light refracted by a fisheye lens results in barrel distortion, that is, curves, in an acquired image, and the shape of an object is distorted in radial directions. A projection region may be used to understand a fisheye lens projection model.
Referring to
On the other hand, when the ray is refracted by a fisheye lens, the virtual image is formed at a point p′ in a projection region, and a radial distance r_{d }is determined according to orthographic projection onto an image plane.
The spatial warping relationship of
Each projection model provides the relationship between the radial distance r_{u }from the center to an undistorted point and the radial distance r_{d }from the center to a distorted point. Since the radial distance r_{u }is determined in the projection region by the angle θ of an incident ray, projection mapping functions for determining the radial distance r_{d }may be defined as functions of θ. In
Equidistant projection: r_{d}=2f(θ) [Equation 2]
Equisolid projection: r_{d}=f(sin(θ/2)) [Equation 3]
Orthographic projection: r_{d}=f(sin(θ)) [Equation 4]
Stereoscopic projection: r_{d}=2f(tan(θ/2)) [Equation 5]
In the above equations, f, θ, and r_{d }are the focal length, incidence angle, and distance between the center point in a distorted image and a projected point, respectively.
A system and method for correcting an image through estimation of a distortion parameter according to exemplary embodiments of the present invention will be described below on the basis of the projection models.
As shown in
This is described in further detail below.
Referring to
The distorted image receiver 100 may receive a distorted image including one or more measurement targets. This may be a component for receiving a distorted image including one or more measurement targets to correct distortion using objective feature information. The one or more measurement targets in the received distorted image may be people. Meanwhile, the distorted image receiver 100 may receive a distorted image through a wideangle lens, such as a fisheye lens.
The feature point extractor 200 may extract a plurality of feature points from each measurement target. This may be a component for extracting unique feature information from a measurement target in a distorted image rather than lens information, pattern information, or learned data. When the measurement target is a person, the plurality of extracted feature points may include the center point of an eye corresponding to the iris of his or her left eye, the center point of an eye corresponding to the iris of his or her right eye, the end point of his or her nose, the left end point of his or her mouth, and the right end point of his or her mouth. When several people are measurement targets, a plurality of feature points may be extracted from each person.
In extracting feature points from the measurement target, it is possible to use multitask deep cascaded convolutional neural network.
The feature point classifier 300 may classify the one or more measurement targets as a distorted target and an undistorted target by comparing distances between a plurality of extracted feature points and the center point of the received distorted image with each other. This may be the criterion for classification into a distorted target and an undistorted target based on the characteristic of a wideangle lens that the degree of distortion increases as the distance from the center of an image increases.
The characteristic of a wideangle lens that the degree of distortion increases as the distance from the center of an image increases may be described with reference to
As shown in
To solve this problem, according to a rational functionbased distortion model, it is possible to define the relationship between a distorted point and an undistorted point by using a polynomial function as shown in Equation 6 below.
where (x_{u}, y_{u}), (x_{d}, y_{d}), and (x_{c}, y_{c}) represent coordinates of an undistorted point, a distorted point, and the center point of an image, respectively. The undistorted point has coordinate values obtained by dividing the coordinate values of the distorted point by L(r_{u}) and is given by Equation 7 below.
When distortion parameters {k_{1}, k_{2}, . . . } are estimated, a distorted point may be calculated from an undistorted point by using Equation 6.
Since there is little improvement between distortion correction with a secondary or higher model and distortion correction with a primary model, it is possible to use the distortion model L(r_{u})=1+k_{1}r_{d}^{2 }given by Equation 6 according to the primary model which requires a small amount of calculation. Modeling of a lens other than a fisheye lens is not included in this task, and thus accuracy in another distortion model is not additionally analyzed.
Referring to
Meanwhile, the feature point classifier 300 may classify the one or more measurement targets as a distorted target and an undistorted target according to undistortedtarget standard formulae given by Equations 8 and 9 below.
where L_{I }is an undistorted target, L_{i}* represents a measurement target at the shortest distance from the center point of a distorted image, N_{F }represents the number of measurement targets, μ_{i }the average point of a plurality of feature points extracted from a measurement target, and C=(x_{c}, y_{c}) is the center point of the received distorted image.
L_{i}={p_{LE}^{i}, p_{RE}^{i}, p_{N}^{i}, p_{LM}^{i}, p_{RM}^{i}}, and
μ_{i}=⅕(p_{LE}^{i}+p_{RE}^{i}+p_{N}^{i}+p_{LM}^{i}+p_{RM}^{i}), [Equation 9]
where L_{i }represent feature points of a measurement target, μ_{i }represents the average point of a plurality of feature points, p_{LE}^{i }represents the center point of a left eye corresponding to the iris thereof, p_{RE}^{i }represents the center point of a right eye corresponding to the iris thereof, p_{N}^{i }represents the end point of a nose, p_{LM}^{i }represents the left end point of a mouth, and p_{RM}^{i }represents the right end point of the mouth.
When N_{F }is the number of measurement targets, L_{i }(i=1, . . . , N_{F}) may be a set of a plurality of feature points extracted from a measurement target.
More specifically, according to the undistortedtarget standard formulae given by Equations 8 and 9, a measurement target closest to the center point of a received distorted image among several measurement targets in the received distorted image may be determined to be an undistorted target, and all other measurement targets may be determined to be distorted targets.
Here, L_{i }may include p_{LE}^{i }the center point of a left eye corresponding to the iris thereof, p_{RE}^{i }the center point of a right eye corresponding to the iris thereof, p_{N}^{i }the end point of a nose, p_{LM}^{i }the left end point of a mouth, p_{RM}^{i }an the right end point of the mouth.
μ_{i }may be the average point of the plurality of feature points.
Meanwhile, in equations of the present invention, all specific points (an undistorted point, a distorted point, a center point, etc.) may be coordinate points.
The distortion parameter estimator 400 may estimate a distortion parameter on the basis of standard deviations of a plurality of feature points of a classified distorted target and undistorted target. This may be a component for estimating a distortion parameter using a standard deviation which is an important factor for determining the amount of distortion.
Meanwhile, the distortion parameter estimator 400 may estimate a distortion parameter according to a distortion parameter formula given by Equation 10 below.
where k^{(j) }represents a j^{th }estimated distortion parameter, N_{F }is the number of measurement targets, σ_{i}^{(j) }represents the standard deviation of feature points of a distorted target, and σ_{i}*^{(j) }represents the standard deviation of feature points of an undistorted target.
As shown in
The image corrector 500 may correct the received distorted image on the basis of an estimated distortion parameter. This may be a component for accurately detecting the degree of correction on the basis of a distortion parameter, which has been estimated in consideration of the amount of distortion and the like, and optimally correcting a distorted image.
However, since lens distortion is spatially variable, image information is lost when distortion occurs. Therefore, a process may be additionally required to output a plurality of images through repeated image correction and select an optimally corrected image.
Meanwhile, the system for correcting an image through estimation of a distortion parameter according to an exemplary embodiment of the present invention may further include an iterative corrector 600 and a final correctedimage selector 700.
The iterative corrector 600 may output one or more repeatedlycorrected images by repeatedly estimating a distortion parameter and repeatedly correcting a corrected image until the corrected image satisfies a preset condition. This may be a component for outputting multiple candidate images by repeatedly estimating a distortion parameter and repeatedly using a distortion correction model on the basis of the distortion parameter in order to select an optimally corrected image.
Meanwhile, the iterative corrector 600 may repeatedly correct a corrected image according to iterative correction formulae given by Equations 11 and 12 below.
where {circumflex over (f)}_{u}^{(j+1) }represents a j+1^{th }corrected image, k^{(j) }represents a j^{th }estimated distortion parameter, r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, {circumflex over (f)}_{u}^{(j) }represents the j^{th }corrected image, and N_{I }is the number of iterations.
r^{(j)}=√{square root over ((x_{d}^{(j)}−x_{c})^{2}+(y_{d}^{(j)}−y_{c})^{2})}, [Equation 12]
where r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, x_{d}^{(j) }and y_{d}^{(j) }represent horizontal and vertical coordinates of the j^{th }corrected image, and x_{c }and y_{c }represent coordinates of the center point of a distorted image.
The final correctedimage selector 700 may detect straightline information including the number and length of straight lines in one or more output images and select a final correctedimage on the basis of the detected straightline information. This may be a component for selecting an optimally corrected image by applying an optical characteristic to geometric distortion.
More specifically, the correction algorithm of the present invention may be used in a general image restoration framework. In other words, a distorted image is considered a degraded version of an ideal undistorted scene, and a corrected image is obtained by estimating the ideal scene through an image restoration process, which is a distortion correction operation of this task. The image restoration framework is shown in
In
Ideally, a corrected image satisfies calculation of an ideal image given by Equation 13 below.
∥f_{u}−{circumflex over (f)}_{u}∥=0 [Equation 13]
In Equation 13, f_{u }is an ideal undistorted scene, and {circumflex over (f)}_{u }is an image estimated by correcting the distorted image.
However, lens distortion is spatial warping, and thus it is not possible to accurately model lens distortion through a single image performance degradation task. Therefore, Equation 13 is not satisfied in practice. Accordingly, an optimally corrected image is selected from among one or more repeatedlycorrected images which are output through several iterative calculations of Equation 11 instead of the cost function of Equation 13 for comparing images with each other. In the selection process, an optical characteristic which is robust against geometric distortion is used.
More specifically, in the process of projecting a 3D space into a 2D image, a straight line is refracted into a curve due to radial distortion of a lens. Accordingly, a correction process involves making the curve as straight as possible, and thus a final correctedimage may be selected on the basis of straightline information.
However, when an optimally corrected image is selected with only the straightline information, a line directed toward a distortion center is excessively corrected. Therefore, the final correctedimage selector 700 may select a final correctedimage satisfying the cost function of Equation 14, which includes a total length D_{j }of all straight lines in a j^{th }repeatedlycorrected image and a number S_{j} of straight lines extracted from the j^{th }repeatedlycorrected image. Meanwhile, Equation 14 below may be a preset condition of the iterative corrector 600.
where {circumflex over (f)}*_{u }is a final distortioncorrected image, {circumflex over (f)}_{u}^{(j*) }represents a j*^{th }final distortioncorrected image, D_{j }represents the total length of all straight lines in a j^{th }repeatedlycorrected image, S_{j }represents the set of straight lines extracted from the j^{th }repeatedlycorrected image, σ_{L}^{(j) }represents the standard deviation of feature points of the j^{th }repeatedlycorrected image, and · the number of elements of the set.
Since the final correctedimage is selected by using feature points and straightline information in combination according to Equation 14, it is possible to prevent a straight line from being excessively corrected when the straight line is directed toward a distortion center in the image.
Referring to
In the distorted image receiving operation S100, the distorted image receiver 100 may receive a distorted image including one or more measurement targets. This may be an operation for receiving a distorted image including one or more measurement targets to correct distortion using objective feature information. The one or more measurement targets in the received distorted image may be people. Meanwhile, in the distorted image receiving operation S100, a distorted image may be received through a wideangle lens, such as a fisheye lens.
In the feature point extraction operation S200, the feature point extractor 200 may extract a plurality of feature points from each measurement target. This may be an operation for extracting unique feature information from a measurement target in a distorted image rather than lens information, pattern information, or learned data. When the measurement target is a person, the plurality of extracted feature points may include the center point of an eye corresponding to the iris of his or her left eye, the center point of an eye corresponding to the iris of his or her right eye, the end point of his or her nose, the left end point of his or her mouth, and the right end point of his or her mouth. When several people are measurement targets, a plurality of feature points may be extracted from each person.
In extracting feature points from the measurement target, it is possible to use multitask deep cascaded convolutional neural network.
In the feature point classification operation S300, the feature point classifier 300 may classify the one or more measurement targets as a distorted target and an undistorted target by comparing distance values between a plurality of extracted feature points and the center point of the received distorted image with each other. This may be the criterion for classification into a distorted target and an undistorted target based on the characteristic of a wideangle lens that the degree of distortion increases as the distance from the center of an image increases.
Meanwhile, in the feature point classification operation S300, the one or more measurement targets may be classified as a distorted target and an undistorted target according to undistortedtarget standard formulae given by Equations 8 and 9 below.
where L_{I }is an undistorted target, L_{i}* represents a measurement target at the shortest distance from the center point of a distorted image, N_{F }represents the number of measurement targets, μ_{i }represents the average point of a plurality of feature points extracted from a measurement target, and C=(x_{c}, y_{c}) is the center point of the received distorted image.
L_{i}={p_{LE}^{i}, p_{RE}^{i}, p_{N}^{i}, p_{LM}^{i}, p_{RM}^{i}}, and
μ_{i}=⅕(p_{LE}^{i}+p_{RE}^{i}+p_{N}^{i}+p_{LM}^{i}+p_{RM}^{i}), [Equation 9]
where L_{i }represent feature points of a measurement target, μ_{i }represents the average point of a plurality of feature points, p_{LE}^{i }represents the center point of a left eye corresponding to the iris thereof, p_{RE}^{i }represents the center point of a right eye corresponding to the iris thereof, p_{N}^{i }represents the end point of a nose, p_{LM}^{i }a represents the left end point of a mouth, and p_{RM}^{i }represents the right end point of the mouth.
When N_{F }is the number of measurement targets, L_{i }(i=1, . . . , N_{F}) may be a set of a plurality of feature points extracted from a measurement target.
More specifically, according to the undistortedtarget standard formulae given by Equations 8 and 9, a measurement target closest to the center point of a received distorted image among several measurement targets in the received distorted image may be determined to be an undistorted target, and all other measurement targets may be determined to be distorted targets.
Here, L_{i }may include p_{LE}^{i }the center point of a left eye corresponding to the iris thereof, p_{RE}^{i }the center point of a right eye corresponding to the iris thereof, p_{N}^{i }the end point of a nose, p_{LM}^{i }the left end point of a mouth, and p_{RM}^{i }the right end point of the mouth.
μ_{i }may be the average point of the plurality of feature points.
Meanwhile, in equations described in the present invention, all specific points (an undistorted point, a distorted point, a center point, etc.) may be coordinate points.
In the distortion parameter estimation operation S400, the distortion parameter estimator 400 may estimate a distortion parameter on the basis of standard deviations of a plurality of feature points of a classified distorted target and undistorted target. This may be an operation for estimating a distortion parameter using a standard deviation which is an important factor for determining the amount of distortion.
Meanwhile, in the distortion parameter estimation operation S400, a distortion parameter may be estimated according to a distortion parameter formula given by Equation 10 below.
where k^{(j) }represents a j^{th }estimated distortion parameter, N_{F }is the number of measurement targets, σ_{i}^{(j) }represents the standard deviation of feature points of a distorted target, and σ_{i}*^{(j) }represents the standard deviation of feature points of an undistorted target.
In the image correction operation S500, the image corrector 500 may correct the received distorted image on the basis of the estimated distortion parameter. This may be an operation for accurately detecting the degree of correction on the basis of a distortion parameter, which has been estimated in consideration of the amount of distortion and the like, and optimally correcting a distorted image.
However, since lens distortion is spatially variable, image information is lost when distortion occurs. Therefore, a process may be additionally required to output a plurality of images through repeated image correction and select an optimally corrected image.
Meanwhile, the method of correcting an image through estimation of a distortion parameter according to an exemplary embodiment of the present invention may further include an iterative correction operation S600 and a final correctedimage selection operation S700.
In the iterative correction operation S600, the iterative corrector 600 may output one or more repeatedlycorrected images by repeatedly estimating a distortion parameter and repeatedly correcting a corrected image until the corrected image satisfies a preset condition. This may be an operation for outputting multiple candidate images by repeatedly estimating a distortion parameter and repeatedly using a distortion correction model on the basis of the distortion parameter in order to select an optimally corrected image.
Meanwhile, in the iterative correction operation S600, a corrected image may be repeatedly corrected according to iterative correction formulae given by Equations 11 and 12 below.
where {circumflex over (f)}_{u}^{(j+1) }represents a j+1^{th }corrected image, k^{(j) }represents a j^{th }estimated distortion parameter, r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, {circumflex over (f)}_{u}^{(j) }represents the j^{th }corrected image, and N_{I }is the number of iterations.
r^{(j)}=√{square root over ((x_{d}^{(j)}−x_{x})^{2}+(y_{d}^{(j)}−y_{c})^{2})}, [Equation 12]
where r^{(j) }represents the distance between an arbitrary coordinate point of a j^{th }corrected image and the center point of a received image, x_{d}^{(j) }and y_{d}^{(j) }represent horizontal and vertical coordinates of the j^{th }corrected image, and x_{c }and y_{c }represent coordinates of the center point of a distorted image.
In the final correctedimage selection operation S700, the final correctedimage selector 700 may detect straightline information including the number and length of straight lines in one or more output images and select a final correctedimage on the basis of the detected straightline information. This may be an operation for selecting an optimally corrected image by applying an optical characteristic to geometric distortion.
More specifically, in the process of projecting a 3D space into a 2D image, a straight line is refracted into a curve due to radial distortion of a lens. Accordingly, a correction process involves making the curve as straight as possible, and thus a final correctedimage may be selected on the basis of straightline information.
However, when an optimally corrected image is selected with only the straightline information, a line directed toward a distortion center is excessively corrected. Therefore, in the final correctedimage selection operation S700, a final correctedimage satisfying the cost function of Equation 14, which includes a total length D_{j }of all straight lines in a j^{th }repeatedlycorrected image and a number S_{j} of straight lines extracted from the j^{th }repeatedlycorrected image, may be selected. Meanwhile, Equation 14 below may be a preset condition of the iterative correction operation S600.
where {circumflex over (f)}*_{u }is a final distortioncorrected image, {circumflex over (f)}_{u}^{(j*) }represents a j*^{th }final distortioncorrected image, D_{j }represents the total length of all straight lines in a j^{th }repeatedlycorrected image, S_{j }represents the set of straight lines extracted from the j^{th }repeatedlycorrected image, σ_{L}^{(j) }represents the standard deviation of feature points of the j^{th }repeatedlycorrected image, and · the number of elements of the set.
Since the final correctedimage is selected by using feature points and straightline information in combination according to Equation 14, it is possible to prevent a straight line from being excessively corrected when the straight line is directed toward a distortion center in the image.
Referring to
Referring to
For more objective evaluation, the correction method of the present invention proposes a distance ratio as a new measurement standard for evaluating the performance of correcting geometric distortion. More specifically, according to the proposed correction method of the present invention, distance ratios are defined in three directions including a horizontal direction, a vertical direction, and a diagonal direction as shown in Equation 15 below.
In Equation 15, r_{H }and {circumflex over (r)}_{H }are the distance between two points which are horizontally adjacent in an undistorted image and a corrected image, respectively. ψ_{H}, ψ_{V}, and ψ_{D }are distance ratios in the horizontal direction, the vertical direction, and the diagonal direction, respectively.
The corresponding relationship between vertical and diagonal lines is defined as shown in
In this regard, in Table 1, performance is compared in terms of distance ratio between the calibration patternbased method, which is an existing correction method, and the proposed method of the present invention.
Table 1 shows estimated distance ratios of
According to the proposed method of the present invention, a distortion parameter is estimated by using feature points FLPs of a distorted image, and then distortion is corrected by using the estimated distortion parameter.
Referring to
More specifically, the first row of
Referring to
On the other hand, according to the proposed method of the present invention, it is possible to automatically correct distortion by using an appropriately estimated distortion parameter. Also, an optimally corrected solution is selected by using feature points FLPs and straightline analysis, and it is possible to overcome limitations, such as overcorrection, imposed by correction.
of the cost function calculated in a distortioncorrected image.
In the proposed method of the present invention, the characteristic of a wideangle lens that a straight line is refracted into a curve is used to select an optimally corrected image.
First, as shown in
In comparing
of the cost function calculated in the distortioncorrected image are minimized at the 18^{th }iteration. In other words, straightline information in a distorted image may be a major factor for selecting an optimally corrected image.
Since straightline information is a major factor for selecting an optimally corrected image, it is also important to select an appropriate method for detecting straight lines. In this regard, the Flores method is a recent straightline detection algorithm for correcting geometric distortion, but distortion effects which degrade the accuracy in detecting straight lines are not taken into consideration. This is described in detail below.
Referring to
Referring to
As shown in
Cho'"'"'s method involves correcting distortion until a difference value between an estimated distortion parameter and a previously estimated distortion parameter becomes smaller than a predefined threshold value. Since the threshold value varies according to image size, the accuracy in distortion correction is low.
More specifically, referring to
where H is the distance between the right end and left walls in the ideal undistorted image, L is the maximum distance between the walls in the distorted image, and γ is the minimum distance between the walls in the distorted image. In an undistorted image, 2H equals L+γ. Accordingly, the distortion rate becomes 0.
Table 2 compares the distortion ratio of correction results obtained by using Cho'"'"'s method and the proposed offline method of the present invention (
Referring to Table 2, the distortion rate of the proposed method of the present invention is lower than that of Cho'"'"'s method (see results on left column of
The abovedescribed system and method for correcting an image through estimation of a distortion parameter have the following effects:
First, it is possible to select an optimal distortioncorrected image using straightline information of corrected images.
Second, it is possible to objectively correct distortion on the basis of feature information extracted from a distorted image and optical characteristics without using lens information, pattern information, or learned data.
Third, it is possible to optimally correct distortion through iterative estimation of a distortion parameter and an iterative distortion correction model.
Forth, it is possible to maintain constant performance by correcting distortion through a distortion parameter estimated on the basis of unique feature information of a measurement target.
Fifth, since the system and method can be applied to a system to which it is not possible to apply existing distortion correction methods, the system and method can be applied to various fields.
Sixth, it is possible to apply the system and method to extended application fields, such as a widearea surveillance system and a highend driver supporting system including a 360degree augmented reality (AR) or virtual reality (VR) camera, as a wide image system.
Meanwhile, the term “image” used in this specification, claims, etc. is for the purpose of describing the concepts of terms and drawings.
Although the present invention has been described above with reference to embodiments for exemplifying the technical spirit of the present invention, those of ordinary skill in the art should appreciate that the present invention is not limited to the configurations and effects illustrated and described above and the present invention can be variously modified or altered without departing from the technical spirit of the present invention. Accordingly, all such changes and modifications should be construed as being within the scope of the present invention.