Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination

• US 10,504,239 B2
• Filed: 04/13/2016
• Issued: 12/10/2019
• Est. Priority Date: 04/13/2015
• Status: Active Grant

• Current Assignee: Universidade de Coimbra
• Original Assignee: Universidade de Coimbra
• Inventors: Rodrigues, Pedro Miguel Cerveira, de Almeida Barreto, Joao Pedro
• Primary Examiner(s): Hung, Yubin
• Application Number:
  US15/566,536
• Publication Number:
  US 20180089855A1
• Time in Patent Office: 1,336 Days
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 1/60 Colour correction or control H04N1/54 takes precedence
• CPC Class Codes: H04N 1/6019 using look-up tables H04N1/6025 takes precedence
• CPC Class Codes: H04N 17/02 for colour television signals
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

Methods and systems for describing a camera radiometrically, in terms of camera response function and vignetting, and in terms of color, suitable for non-uniform illumination set-ups. It estimates the camera response function and the camera color mapping from a single image of a generic scene with two albedos. With a second same-pose image with a different intensity of the near-light the vignetting is also estimated. The camera response function calibration involves the segmentation of the two albedos, the definition of the system of equations based on the assumption that can be made about the image formation process, and the actual estimation. For the modelling the vignetting there are three steps: computing the albedo-normalized irradiance, finding points of equal vignetting, when needed, and estimation.

Claims

1. A method for estimating a color model of an imaging device, comprised of the response function and the color mapping, under generic illumination, the method comprising:

i. acquiring at least one calibration image of a scene with at least two regions of constant albedo from an arbitrary pose;

ii. segmenting the calibration image into regions of constant albedo and grouping the pixels in each region into a set , the set of points with albedo ρ

_n;

iii. deriving equations for the estimation using

f<sup>−

1</sup>(d(x))=u(x)ρ

′

(x)

where x are the pixel coordinates that correspond to a scene point, d(x) is the acquired image and is defined as a vector of N_c elements d_c(x) that correspond to values on pixel x and channel c, f<sup>−

1</sup>(d) is the inverse camera response function, a function vector of elements f _c<sup>−

1</sup>(d_c), u(x) is the albedo-normalized irradiance, a scalar function, and ρ

′

is the albedo vector of size N_c in the color space of the camera and is defined by elements ρ

′

_c for each channel c such that

ρ

=Mρ

′

with ρ

being the albedo vector of size N_s in the output color space and is defined by elements ρ

_s for each channel s, and M being a color mapping matrix of size N_s by N_c and composed of rows m_s^T; and

iv. determining the color model of the imaging device by solving the equations derived in step iii using a suitable solver or optimization scheme.

2. The method of claim 1, wherein the scene used in step i is a known planar checkerboard pattern.

3. The method of claim 1, wherein N_c=1 for single-channel imaging devices, N_c=3 or trichromatic imaging devices, N_c>

3 for multispectral and hyperspectral imaging devices, and N_s=N_c for an output image with the same number of channels as the input image.

4. The method of claim 1, wherein the equations derived in step iii are defined directly from the image formation model as

f_c⁻

1(d_c(x))=u(x)ρ

′

_c(x)

or, equivalently, in the logarithmic space using

log f_c<sup>−

1</sup>(d_c(x))=log u(x)+log ρ

′

_c(x)

where log stands for a logarithm of any base and where d(x) is known and f<sup>−

1</sup>, u(x), and ρ

′

(x) are optimized for in step iv.

5. The method of claim 4, wherein the scene contains at least N_c albedos of known values on the output color space, which enables the estimation of M after the estimation of f⁻

1,u(x), and ρ

′

(x) using equations of the form

ρ

_s=m_s^Tρ

′

or together with the estimation of f<sup>−

1</sup> and u(x) using equations of the form

m_s^Tf<sup>−

1</sup>(d(x))=u(x)ρ

_s(x).

6. The method of claim 1, wherein the equations derived in step iii are defined on the isocurves of u(x), in which case the d(x) is known and f⁻

1 is optimized for in step iv.

7. The method of claim 6, wherein to estimate the isocurves of u(x), a parametric model h(x) is fitted to the regions of one albedo or more than one independently or using, for the purpose of incorporating more than one albedo on the same model and estimate only one set of isocurves, the following equations:

8. The method of claim 6, wherein the isocurves are used to establish equations of the form

ρ

′

<sub>c′

,n′</sub>

f_c<sup>−

1</sup>(d_c(x_j))=ρ

′

_c,nf_c′

<sup>−

1</sup>(d_c′

(x_k)), j∈

_i∩

k∈

_i∩

for a pair of albedos ρ

_n and ρ

_n′

, where _i is the set of pixels crossed by isocurve i, and where d(x) is known and f<sup>−

1</sup> and ρ

′

(x) are optimized for in step iv.

9. The method of claim 8, wherein the scene contains at least N _calbedos of known values on the output color space, which enables the estimation of M after the estimation of f⁻

1 and ρ

′

(x) using equations of the form

ρ

s=m_s^Tρ

′

or together with the estimation of f<sup>−

1</sup> using equations of the form

ρ

<sub>s′

,n′</sub>

m_s^Tf<sup>−

1</sup>(d(x_j))=ρ

_s,nm_s′

^Tf<sup>−

1</sup>(d(x_k)),j∈

_i∩

,k∈

_i∩

.

10. The method of claim 1, wherein the equation derivation in step iii uses a particular case matrix for the color mapping matrix comprising a diagonal matrix or a rotation matrix.

11. The method of claim 1, wherein the solution of step iv is found using contributions from multiple calibration images, in which case steps i to iii are executed for each one of these images.

12. The method of claim 1, wherein the determination of the color model in step iv is carried together with one or more of monotonicity constraints on the camera response function and regularization of search parameters.

13. The method of claim 1, wherein for the purpose of having a camera response function defined for the whole range of possible pixel values d, a parametric function, an empiric model of response, or an interpolation/extrapolation method is used together with or as post processing to solving the system of equations in step iv.

14. The method of claim 1, wherein the optimization in step iv is executed with nonlinear optimization algorithms, comprising interior point, gradient descent, and metaheuristics, for the cases of systems of equations that are nonlinear, and it is executed with convex optimization algorithms, comprising quadratic programming, for the cases of system of equations that are linear.

15. The method of claim 1, wherein the color model of the imaging device is used to normalize the response function, by computing a new image where the pixel values d′

(x) are given by

d′

=f<sup>−

1</sup>(d)

for image points x or, in the case of the color mapping matrix being also known, are given by

d′

=Mf¹(d).

16. The method of claim 15, wherein the new image is computed for improved visualization and/or for further processing by other methods or algorithms, comprising shape-from-shading, photometric vision algorithms, computer-aided surgery methods, and transformations of pixel values for enhancing image properties or features, such as contrast or particular colors.

17. The method of claim 1, wherein the output color space is a standard color space, comprising CIEXYZ and sRGB, or other predetermined color space, comprising the color space of another imaging device.

18. The method of claim 1, wherein the algorithms presented may be used as a starting point of, integrated in, or complement other models of imaging devices, comprising multi-dimensional look-up tables.

19. A method for estimating a vignetting model of an imaging device comprising:

i. acquiring at least one set of at least two calibration images with the same relative pose and a different intensity of a near-light source;

ii. segmenting the calibration images into regions of constant albedo;

iii. computing the albedo-normalized irradiance for each pair of same-pose images, u₁(x) and u₂(x);

iv. deriving the equations of the vignetting model using

m(x)=K₁u₁(x)+K₂u₂(x)

 

where K₁ and K₂ are the two model unknowns; and

v. finding a solution for the vignetting model with the equations created in step iv.

20. The method of claim 19, wherein, for the purpose of having additional equations, sets of points with equal vignetting are found in step iv by searching for points where one or more of both u₁(x)/u₂(x) and u₁(x) are constant and both u₁(x)/u₂(x) and u₂(x) are constant, which may be performed either by exhaustive search or by intersecting the isocurves of one or more of u₁(x)/u₂(x) and u₁(x) and u₁(x)/u₂(x) and u₂(x).

21. The method of claim 19, wherein the vignetting estimation equations in step iv, when two or more pairs of same-pose images are available can be defined as

m(x)=k₁u₁(x)+K₂u₂(x)

m(x)=K′

₁u′

₁(x)+k′

₂u′

₂(x)

for every two pairs, which can be rewritten as

k₁u₁(x)+K₂u₂(x)=K′

₁u′

₁(x)+k′

₂u₂(x)

and, in this way, creating a system of equations from which the vignetting model unknowns (K₁, K₂, K′

₁, and k′

₂) can be estimated.

22. The method of claim 19, wherein the vignetting model of the imaging device is used to normalize any acquired images and videos, for improved visualization and/or for further processing by other methods.