Image converter

• US 8,730,323 B2
• Filed: 01/08/2013
• Issued: 05/20/2014
• Est. Priority Date: 09/03/2008
• Status: Active Grant

• Current Assignee: Dai Nippon Printing Company Limited
• Original Assignee: Dai Nippon Printing Company Limited
• Inventors: Inagaki, Koji
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US13/736,552
• Publication Number:
  US 20130177258A1
• Time in Patent Office: 497 Days
• Field of Search: 348143-159
• US Class Current: 348/143
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/006 Geometric correction correction of chromatic aberrations adapted to be used in an image pickup device containing an electronic image sensor H04N23/10; detecting, correcting, reducing or removing artefacts resulting only from the lens unit, e.g. flare, shading, vignetting or "cos4" H04N25/61
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

An image converter for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting the part into a planar regular image. The image converter includes a distorted circular image memory, a planar regular image memory, a parameter input unit, a corresponding coordinate calculating unit, and a planar regular image forming unit.

Claims

1. An image converter for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting said part into a planar regular image, the image converter comprising:

a distorted circular image memory for storing a distorted circular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system and having a radius R taking an origin O of the two-dimensional XY orthogonal coordinate system as a center;

a planar regular image memory for storing a planar regular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system;

a parameter input unit in which in a three-dimensional XYZ orthogonal coordinate system including the two-dimensional XY orthogonal coordinate system, a visual line vector n facing any given direction, with the origin O given as a starting point, is input as a parameter indicating a cut-out position of the planar regular image, a predetermined planar inclination angle N is input as a parameter indicating a cut-out orientation of the planar regular image, and a predetermined magnification m is input as a parameter indicating a cut-out dimension of the planar regular image;

a corresponding coordinate calculating unit calculating corresponding coordinates (x, y) which correspond to any given coordinates (u, v) by using predetermined correspondence relationship equations showing a correspondence relationship between coordinates (u, v) on a two-dimensional UV cylindrical coordinate system and coordinates (x, y) on the two-dimensional XY orthogonal coordinate system, wherein said two-dimensional UV cylindrical coordinate system is defined by curving a two-dimensional UV orthogonal coordinate system which is arranged on a plane passing through a point G given as an origin and orthogonal to the visual line vector n to have an orientation according to the planar inclination angle Φ

, said point G being away from the origin O by “

a product m≅

R of the magnification m and the radius R”

on the visual line vector n, along a side face of a “

virtual cylindrical column in which the point G gives one point on the side face thereof to have a central axis parallel to a V axis of the two-dimensional UV orthogonal coordinate system,”

; and

a planar regular image forming unit giving coordinates (u, v) of a target pixel constituting the planar regular image to the corresponding coordinate calculating unit to obtain corresponding coordinates (x, y), reading out a pixel value of a pixel arranged in the vicinity of the obtained corresponding coordinates (x, y) inside the distorted circular image memory, determining a pixel value of the target pixel on the basis of a read pixel value, thereby forming the planar regular image by determining pixel values of individual pixels, and writing the pixel values into the planar regular image memory;

wherein an orientation of the central axis of the virtual cylindrical column varies depending on parameters input by the parameter input unit so that various UV cylindrical coordinate systems are defined depending on the parameters input.

2. The image converter according to claim 1, wherein when, with respect to a virtual sphere having the radius R taking the origin O as a center, a corresponding point Qi is taken on the sphere which corresponds to a point Si indicated by coordinates (xi, yi) on the two-dimensional XY orthogonal coordinate system depending on a projection method of a fisheye lens used, and coordinates (ui, vi) are taken on the two-dimensional UV cylindrical coordinate system at an intersecting point Ci between a straight line connecting the origin O with the corresponding point Qi on the sphere and a cylindrical column side-face coordinate plane of the two-dimensional UV cylindrical coordinate system, the corresponding coordinate calculating unit uses correspondence relationship equations by which the coordinates (xi, yi) are determined as corresponding coordinates which correspond to the coordinates (ui, vi).

3. The image converter according to claim 2, wherein when a distorted circular image stored in the distorted circular image memory is an orthogonally projected image photographed by use of a fisheye lens based on an orthogonal projection method, the corresponding coordinate calculating unit uses correspondence relationship equations of orthogonally projected images in which, with respect to a point Si indicated by coordinates (xi, yi), a point indicated by coordinates (xi, yi, zi) given as an intersecting point between a straight line passing through the point Si and parallel to a Z axis and the virtual sphere is given as a corresponding point Qi on the sphere, and

when a distorted circular image stored in the distorted circular image memory is a non-orthogonally projected image photographed by use of a fisheye lens based on a non-orthogonal projection method, the corresponding coordinate calculating unit uses correspondence relationship equations of non-orthogonally projected images obtained by using coordinate conversion equations between coordinates on the orthogonally projected image and coordinates on the non-orthogonally projected image so as to correct the correspondence relationship equations of orthogonally projected images.

4. The image converter according to claim 3, wherein the corresponding coordinate calculating unit defines the two-dimensional UV cylindrical coordinate system by using the virtual cylindrical column in which the point G gives one point on the side face thereof to have a central axis parallel to the V axis of the two-dimensional UV orthogonal coordinate system and also passing through the origin O of the three-dimensional XYZ orthogonal coordinate system.

5. The image converter according to claim 4, wherein the corresponding coordinate calculating unit uses the following equations as correspondence relationship equations of orthogonally projected images indicating a correspondence relationship between coordinates (u, v) and coordinates (x, y),

x=G(u′

A+vB+w′

C)

y=G(u′

D+vE+w′

F),

under the following definitions in which an angle formed between an orthogonal projection on the XY plane of a visual line vector n and a Y axis is given as an azimuthal angle α and

an angle formed between the visual line vector n and a positive direction of the Z axis is given as a zenithal angle β

,

A=cos φ

cos α

−

sin φ

sin α

cos β

B=−

sin φ

cos α

−

cos φ

sin α

cos β

C=sin β

sin α

D=cos φ

sin α

+sin φ

cos α

cos β

E=−

sin φ

sin α

+cos φ

cos α

cos β

F=−

sin β

cos α

G=R/√

(w²+v²)

w=mR

u′

=w′

sin(u/w)

w′

=w′

cos(u/w).

6. The image converter according to claim 5, wherein

the corresponding coordinate calculating unit including:

a rotational coefficient operation unit in which, when the visual line vector n and the planar inclination angle φ

are given from the parameter input unit, on the basis of the visual line vector n, the azimuthal angle α and

the zenithal angle β

are determined, and rotational coefficients A, B, C, D, E, F are calculated on the basis of the following operational equations,

A=cos φ

cos α

−

sin φ

sin α

cos β

B=−

sin φ

cos α

−

cos φ

sin α

cos β

C=sin β

sin α

D=cos φ

sin α

+sin φ

cos α

cos β

E=−

sin φ

sin α

+cos φ

cos α

cos β

F=−

sin β

cos α

;

a common coefficient operation unit in which, when the magnification m is given from the parameter input unit and a coordinate v is given from the planar regular image forming unit, the radius R of the distorted circular image is used to calculate a common coefficient G on the basis of the following operational equations,

w=mR

G=R/√

(w²+v²);

a curved coordinate correcting unit in which, when the magnification m is given from the parameter input unit and a coordinate u is given from the planar regular image forming unit, the radius R of the distorted circular image is used to calculate u′ and

w′

on the basis of the following operational equations,

w=mR

u′

=w′

sin(u/w)

w′

=w′

cos(u/w); and

an xy coordinate value calculating unit in which the coordinate v given from the planar regular image forming unit, the rotational coefficients A, B, C, D, E, F calculated by the rotational coefficient operation unit, the common coefficient G calculated by the common coefficient operation unit, the u′ and

w′

calculated by the curved coordinate correcting unit, and the radius R of the distorted circular image are used to calculate x and y on the basis of the following operational equations,

x=G(u′

A+vB+w′

C)

y=G(u′

D+vE+w′

F).

7. The image converter according to claim 6, wherein

the parameter input unit has a function of inputting the radius R as a variable parameter on the basis of instructions from a user or a distorted circular image stored in a distorted circular image memory, and

the common coefficient operation unit and the curved coordinate correcting unit use the radius R input by the parameter input unit to perform an operation.

8. The image converter according to claim 3, wherein

when a distorted circular image stored in the distorted circular image memory is an equidistantly projected image photographed by use of a fisheye lens based on the equidistance projection method, the corresponding coordinate calculating unit uses the following coordinate conversion equations for converting coordinates (xa, ya) on the orthogonally projected image into coordinates (xb, yb) on the equidistantly projected image,

xb=xa(2R/π

r)sin<sup>−

1</sup>(r/R)

yb=ya(2R/π

r)sin<sup>−

1</sup>(r/R)

wherein the condition is r=√

(xa²+ya²), thereby correcting the correspondence relationship equations of orthogonally projected images.

9. The image converter according to claim 1, wherein

the parameter input unit inputs coordinates (x0, y0) of a cut-out center point P on the two-dimensional XY orthogonal coordinate system as a parameter for defining the visual line vector n and defines a virtual sphere having the radius R at the center of an origin O in a three-dimensional XYZ orthogonal coordinate system,

when a distorted circular image stored in the distorted circular image memory is an orthogonally projected image photographed by use of a fisheye lens based on the orthogonal projection method, the parameter input unit defines an intersecting point Q between a straight line passing through the cut-out center point P and parallel to the Z axis and the virtual sphere, thereby giving a vector which starts from the origin O to pass through the intersecting point Q as the visual line vector n, and

when a distorted circular image stored in the distorted circular image memory is a non-orthogonally projected image photographed by use of a fisheye lens based on the non-orthogonal projection method, the parameter input unit uses coordinate conversion equations between coordinates on the orthogonally projected image and coordinates on the non-orthogonally projected image to correct coordinates of the cut-out center point P and defines an intersecting point Q between a straight line passing through a point after correction and parallel to the Z axis and the virtual sphere, thereby giving a vector which starts from the origin O to pass through the intersecting point Q as the visual line vector n.

10. The image converter according to claim 1, wherein

the corresponding coordinate calculating unit arranges the two-dimensional UV orthogonal coordinate system in such an orientation that an angle formed between a rotational reference axis J given as an axis passing through an origin G, parallel to the XY plane and also orthogonal to the visual line vector n and the U axis is equal to the planar inclination angle φ

.

11. The image converter according to claim 1, wherein

the planar regular image stored in the planar regular image memory is rectangular in contour, the two-dimensional UV orthogonal coordinate system having the U axis in a direction parallel to the longer side of the rectangle and the V axis in a direction parallel to the shorter side is curved along the side face of the virtual cylindrical column having a central axis parallel to the V axis, thereby defining the two-dimensional UV cylindrical coordinate system.

12. The image converter according to claim 1, wherein the planar regular image forming unit performs interpolation operation for pixel values of plural reference pixels on a distorted circular image arranged in the vicinity of a position indicated by corresponding coordinates (x, y) upon determination of a pixel value of a target pixel arranged at a position indicated by the coordinates (u, v).

13. A non-transitory computer readable recording medium containing a program for allowing a computer to function as the image converter according to claim 1.

14. A semiconductor integrated circuit into which electronic circuits functioning as a corresponding coordinate calculating unit and a planar regular image forming unit which are constituents of the image converter according to claim 1 are incorporated.

15. A fisheye lens monitoring system including:

the image converter according to claim 1;

a digital camera equipped with a fisheye lens; and

a monitoring device for displaying a planar regular image on a screen;

wherein a distorted circular image photographed by use of the digital camera is stored in the distorted circular image memory, and a planar regular image obtained in the planar regular image memory is displayed by the monitoring device.

16. An image conversion method for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting said part into a planar regular image, the image conversion method including:

an image preparation step for storing in a distorted circular image memory a distorted circular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system and having a radius R taking an origin O of the two-dimensional XY orthogonal coordinate system as a center;

a parameter setting step for setting in a three-dimensional XYZ orthogonal coordinate system including the two-dimensional XY orthogonal coordinate system, a visual line vector n which faces any given direction with the origin O given as a starting point, as a parameter indicating a cut-out position of a planar regular image, a predetermined planar inclination angle φ

as a parameter indicating a cut-out orientation of the planar regular image, and a predetermined magnification m as a parameter indicating a cut-out dimension of the planar regular image;

a corresponding coordinate calculating step in which an operation device calculates corresponding coordinates (x, y) which correspond to coordinates (u, v) of one target pixel for a planar regular image made up of an aggregate of many pixels arranged at a positioned indicated by coordinates (u, v) on the two-dimensional UV orthogonal coordinate system;

a pixel value determining step in which the operation device reads out a pixel value of a pixel arranged in the vicinity of the corresponding coordinates (x, y) inside the distorted circular image memory, determining a pixel value of the target pixel on the basis of a thus read pixel value; and

a pixel value writing step in which the operation device writes a pixel value determined with regard to the target pixel inside a planar regular image memory for storing the planar regular image;

wherein, in the corresponding coordinate calculating step, a point G on the visual line vector n spaced away from the origin O by “

a product m·

R of the magnification m and the radius R”

being given as an origin, a two-dimensional UV orthogonal coordinate system arranged to have an orientation depending on the planar inclination angle φ

on a plane passing through the point G and orthogonal to the visual line vector n is curved along a side face of “

a virtual cylindrical column in which the point G gives one point on the side face to have a central axis parallel to the V axis of the two-dimensional UV orthogonal coordinate system,”

thereby defining a two-dimensional UV cylindrical coordinate system, and predetermined correspondence relationship equations indicating a correspondence relationship between coordinates (u, v) on the two-dimensional UV cylindrical coordinate system and coordinates (x, y) on the two-dimensional XY orthogonal coordinate system are used to calculate corresponding coordinates (x, y) which correspond to any given coordinates (u, v), thereby writing pixel values for all pixels necessary for constituting the planar regular image into the planar regular image memory;

wherein an orientation of the central axis of the virtual cylindrical column varies depending on parameters set at the parameter setting step so that various UV cylindrical coordinate systems are defined depending on the parameters set.