IMAGE CONVERTER

0Associated
Cases 
0Associated
Defendants 
0Accused
Products 
20Forward
Citations 
0
Petitions 
1
Assignment
First Claim
1. An image converter for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting said part into a planar regular image, the image converter comprising:
 a distorted circular image memory for storing a distorted circular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on a twodimensional XY orthogonal coordinate system and having a radius R taking an origin O of the twodimensional XY orthogonal coordinate system as a center;
a planar regular image memory for storing a planar regular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (u, v) on a twodimensional UV orthogonal coordinate system;
a parameter input unit in which in a threedimensional XYZ orthogonal coordinate system including the twodimensional XY orthogonal coordinate system, a visual line vector n facing any given direction, with the origin O given as a starting point, is input as a parameter indicating a cutout position of the planar regular image, a predetermined planar inclination angle φ
is input as a parameter indicating a cutout orientation of the planar regular image, and a predetermined magnification m is input as a parameter indicating a cutout dimension of the planar regular image;
a corresponding coordinate calculating unit calculating corresponding coordinates (x, y) which correspond to any given coordinates (u, v) by using predetermined correspondence relationship equations showing a correspondence relationship between coordinates (u, v) on a twodimensional UV curved coordinate system and coordinates (x, y) on the twodimensional XY orthogonal coordinate system, wherein said twodimensional UV curved coordinate system is defined by curving a twodimensional UV orthogonal coordinate system which is arranged on a plane passing through a point G given as an origin and orthogonal to the visual line vector n to have an orientation according to the planar inclination angle φ
, said point G being away from the origin O by “
a product m·
R of the magnification m and the radius R”
on the visual line vector n, along a side face of a “
virtual cylindrical column in which the point G gives one point on the side face thereof to have a central axis parallel to a V axis of the twodimensional UV orthogonal coordinate system,”
; and
a planar regular image forming unit giving coordinates (u, v) of a target pixel constituting the planar regular image to the corresponding coordinate calculating unit to obtain corresponding coordinates (x, y), reading out a pixel value of a pixel arranged in the vicinity of the obtained corresponding coordinates (x, y) inside the distorted circular image memory, determining a pixel value of the target pixel on the basis of a read pixel value, thereby forming the planar regular image by determining pixel values of individual pixels, and writing the pixel values into the planar regular image memory.
1 Assignment
0 Petitions
Accused Products
Abstract
Any given part is cut out from a distorted circular image photographed by use of a fisheye lens and converted into a planar regular image with less distortion. A virtual sphere H having a radius R on a distorted circular image S on an XY plane is defined, thereby allowing a user to designate a cutout center point P, a magnification m, and a planar inclination angle φ. A visual line vector n passing through an intersecting point Q immediately above the point P is determined to define an UV orthogonal coordinate system having an orientation depending on the angle φ on a plane orthogonal to a visual line vector n at a point G in which a distance between two points OG is given as m·R. The UV orthogonal coordinate system is curved along the side face C of a “virtual cylindrical column in which the point G forms one point on the side face to have a straight line V′ parallel to the V axis and also passing through the point O as a central axis,” thereby defining the UV curved coordinate system. Correspondence relationship equations between a point Ci (ui, vi) on the UV curved coordinate system and a point Si (xi, yi) on the XY coordinate system are used to obtain an image in the vicinity of a point P on the UV curved coordinate system, and the image is expanded on a plane T to obtain a planar regular image.
25 Citations
View as Search Results
Image display apparatus and method  
Patent #
US 7,853,879 B2
Filed 09/06/2006

Current Assignee
Canon Kabushiki Kaisha

Sponsoring Entity
Canon Kabushiki Kaisha

IMAGE CORRECTING DEVICE, METHOD FOR CREATING CORRECTED IMAGE, CORRECTION TABLE CREATING DEVICE, METHOD FOR CREATING CORRECTION TABLE, PROGRAM FOR CREATING CORRECTION TABLE, AND PROGRAM FOR CREATING CORRECTED IMAGE  
Patent #
US 20120051661A1
Filed 02/25/2011

Current Assignee
Hitachi Information Telecommunication Engineering Ltd.

Sponsoring Entity
Hitachi Information Communication Engineering Limited

NAVIGATION DEVICE  
Patent #
US 20130120524A1
Filed 11/14/2012

Current Assignee
NVIDIA Corporation

Sponsoring Entity
NVIDIA Corporation

IMAGE PROCESSOR, IMAGE PROCESSING METHOD AND PROGRAM, AND IMAGING SYSTEM  
Patent #
US 20140071227A1
Filed 09/10/2013

Current Assignee
Ricoh Company Limited

Sponsoring Entity
Ricoh Company Limited

INTERACTIVE SIMULATEDGLOBE DISPLAY SYSTEM  
Patent #
US 20140092017A1
Filed 05/03/2013

Current Assignee
National Taiwan University

Sponsoring Entity
National Taiwan University

Image correcting device, method for creating corrected image, correction table creating device, method for creating correction table, program for creating correction table, and program for creating corrected image  
Patent #
US 8,724,922 B2
Filed 02/25/2011

Current Assignee
Hitachi Information Telecommunication Engineering Ltd.

Sponsoring Entity
Hitachi Information Telecommunication Engineering Ltd.

APPARATUS AND METHOD FOR OUTPUTTING INFORMATION ON FACIAL EXPRESSION  
Patent #
US 20140254939A1
Filed 09/25/2012

Current Assignee
NTT Docomo Incorporated

Sponsoring Entity
NTT Docomo Incorporated

Image processing apparatus, image processing method, storage medium, and image processing system  
Patent #
US 8,854,359 B2
Filed 09/06/2012

Current Assignee
Ricoh Company Limited

Sponsoring Entity
Ricoh Company Limited

IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, PROGRAM FOR IMAGE PROCESSING DEVICE, RECORDING MEDIUM, AND IMAGE DISPLAY DEVICE  
Patent #
US 20140314336A1
Filed 12/18/2012

Current Assignee
Dai Nippon Printing Company Limited

Sponsoring Entity
Dai Nippon Printing Company Limited

Interactive simulatedglobe display system  
Patent #
US 8,982,049 B2
Filed 05/03/2013

Current Assignee
National Taiwan University

Sponsoring Entity
National Taiwan University

Image generation device, camera device, image display device, and image generation method  
Patent #
US 9,251,559 B2
Filed 09/29/2014

Current Assignee
Panasonic Intellectual Property Management Co. Ltd.

Sponsoring Entity
Panasonic Intellectual Property Management Co. Ltd.

Image processing device, image processing method, program for image processing device, recording medium, and image display device  
Patent #
US 9,269,124 B2
Filed 12/18/2012

Current Assignee
Dai Nippon Printing Company Limited

Sponsoring Entity
Dai Nippon Printing Company Limited

Navigation device  
Patent #
US 9,628,705 B2
Filed 11/14/2012

Current Assignee
NVIDIA Corporation

Sponsoring Entity
NVIDIA Corporation

Providing a realtime via a wireless communication channel associated with a panoramic video capture device  
Patent #
US 9,883,101 B1
Filed 07/23/2014

Current Assignee
Hoyos Integrity Corporation

Sponsoring Entity
Hoyos Integrity Corporation

Methods, systems, and computer readable media for unified scene acquisition and pose tracking in a wearable display  
Patent #
US 10,365,711 B2
Filed 05/17/2013

Current Assignee
University of North Carolina At Chapel Hill

Sponsoring Entity
University of North Carolina At Chapel Hill

Methods, systems, and computerreadable media for utilizing radial distortion to estimate a pose configuration  
Patent #
US 10,410,372 B1
Filed 06/14/2018

Current Assignee
University of North Carolina At Chapel Hill

Sponsoring Entity
University of North Carolina At Chapel Hill

Panel transform  
Patent #
US 10,453,173 B2
Filed 07/24/2015

Current Assignee
Robert Bosch LLC, Robert Bosch GmbH

Sponsoring Entity
Robert Bosch GmbH

Processingtarget image creating device, processingtarget image creating method, and operation assisting system  
Patent #
US 10,621,743 B2
Filed 10/08/2015

Current Assignee
Sumitomo Heavy Industries Limited

Sponsoring Entity
Sumitomo Heavy Industries Limited

Image processor, image processing method and program, and imaging system  
Patent #
US 10,666,860 B2
Filed 09/10/2013

Current Assignee
Ricoh Company Limited

Sponsoring Entity
Ricoh Company Limited

Fisheye rendering with lens distortion correction for 360degree video  
Patent #
US 10,699,389 B2
Filed 04/24/2017

Current Assignee
Qualcomm Inc.

Sponsoring Entity
Qualcomm Inc.

Method for providing output image in either cylindrical mode or perspective mode  
Patent #
US 7,961,980 B2
Filed 08/06/2007

Current Assignee
Imay Software Co. Ltd.

Sponsoring Entity
Imay Software Co. Ltd.

Omniview motionless camera orientation system  
Patent #
US 7,714,936 B1
Filed 07/02/1997

Current Assignee
Sony Corporation

Sponsoring Entity
Sony Corporation

PANORAMIC CAMERA  
Patent #
US 20010010555A1
Filed 06/11/1997

Current Assignee
Cedar Lane Technologies Inc.

Sponsoring Entity
Be Here Corporation

Omniview motionless camera surveillance system  
Patent #
US 5,359,363 A
Filed 02/08/1993

Current Assignee
Sony Corporation

Sponsoring Entity
TELEROBOTICS INTERNATIONAL INC.

Omniview motionless camera orientation system  
Patent #
US 5,185,667 A
Filed 05/13/1991

Current Assignee
Sony Corporation

Sponsoring Entity
TELEROBOTICS INTERNATIONAL INC. A CORPORATION OF TN

17 Claims
 1. An image converter for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting said part into a planar regular image, the image converter comprising:
a distorted circular image memory for storing a distorted circular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on a twodimensional XY orthogonal coordinate system and having a radius R taking an origin O of the twodimensional XY orthogonal coordinate system as a center; a planar regular image memory for storing a planar regular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (u, v) on a twodimensional UV orthogonal coordinate system; a parameter input unit in which in a threedimensional XYZ orthogonal coordinate system including the twodimensional XY orthogonal coordinate system, a visual line vector n facing any given direction, with the origin O given as a starting point, is input as a parameter indicating a cutout position of the planar regular image, a predetermined planar inclination angle φ
is input as a parameter indicating a cutout orientation of the planar regular image, and a predetermined magnification m is input as a parameter indicating a cutout dimension of the planar regular image;a corresponding coordinate calculating unit calculating corresponding coordinates (x, y) which correspond to any given coordinates (u, v) by using predetermined correspondence relationship equations showing a correspondence relationship between coordinates (u, v) on a twodimensional UV curved coordinate system and coordinates (x, y) on the twodimensional XY orthogonal coordinate system, wherein said twodimensional UV curved coordinate system is defined by curving a twodimensional UV orthogonal coordinate system which is arranged on a plane passing through a point G given as an origin and orthogonal to the visual line vector n to have an orientation according to the planar inclination angle φ
, said point G being away from the origin O by “
a product m·
R of the magnification m and the radius R”
on the visual line vector n, along a side face of a “
virtual cylindrical column in which the point G gives one point on the side face thereof to have a central axis parallel to a V axis of the twodimensional UV orthogonal coordinate system,”
; anda planar regular image forming unit giving coordinates (u, v) of a target pixel constituting the planar regular image to the corresponding coordinate calculating unit to obtain corresponding coordinates (x, y), reading out a pixel value of a pixel arranged in the vicinity of the obtained corresponding coordinates (x, y) inside the distorted circular image memory, determining a pixel value of the target pixel on the basis of a read pixel value, thereby forming the planar regular image by determining pixel values of individual pixels, and writing the pixel values into the planar regular image memory.  View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
 16. An image conversion method for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting said part into a planar regular image, the image conversion method including:
an image preparation step for storing in a distorted circular image memory a distorted circular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on a twodimensional XY orthogonal coordinate system and having a radius R taking an origin O of the twodimensional XY orthogonal coordinate system as a center; a parameter setting step for setting in a threedimensional XYZ orthogonal coordinate system including the twodimensional XY orthogonal coordinate system, a visual line vector n which faces any given direction with the origin O given as a starting point, as a parameter indicating a cutout position of a planar regular image, a predetermined planar inclination angle φ
as a parameter indicating a cutout orientation of the planar regular image, and a predetermined magnification m as a parameter indicating a cutout dimension of the planar regular image;a corresponding coordinate calculating step in which an operation device calculates corresponding coordinates (x, y) which correspond to coordinates (u, v) of one target pixel for a planar regular image made up of an aggregate of many pixels arranged at a positioned indicated by coordinates (u, v) on the twodimensional UV orthogonal coordinate system; a pixel value determining step in which the operation device reads out a pixel value of a pixel arranged in the vicinity of the corresponding coordinates (x, y) inside the distorted circular image memory, determining a pixel value of the target pixel on the basis of a thus read pixel value; and a pixel value writing step in which the operation device writes a pixel value determined with regard to the target pixel inside a planar regular image memory for storing the planar regular image; wherein, in the corresponding coordinate calculating step, a point G on the visual line vector n spaced away from the origin O by “
a product m·
R of the magnification m and the radius R”
being given as an origin, a twodimensional UV orthogonal coordinate system arranged to have an orientation depending on the planar inclination angle φ
on a plane passing through the point G and orthogonal to the visual line vector n is curved along a side face of “
a virtual cylindrical column in which the point G gives one point on the side face to have a central axis parallel to the V axis of the twodimensional UV orthogonal coordinate system,”
thereby defining a twodimensional UV curved coordinate system, and predetermined correspondence relationship equations indicating a correspondence relationship between coordinates (u, v) on the twodimensional UV curved coordinate system and coordinates (x, y) on the twodimensional XY orthogonal coordinate system are used to calculate corresponding coordinates (x, y) which correspond to any given coordinates (u, v), thereby writing pixel values for all pixels necessary for constituting the planar regular image into the planar regular image memory.
 17. An image conversion method including:
a step for preparing a distorted circular image photographed by use of a fisheye lens as an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on an XY plane in a threedimensional XYZ orthogonal coordinate system; a step for defining, at a desired position inside a space constituting the threedimensional XYZ orthogonal coordinate system, a twodimensional UV curved coordinate system constituted with a V axis arranged on a curved face along a side face of a virtual cylindrical column and parallel to a central axis of the virtual cylindrical column and a circulararc U axis along a circumference of a circle constituting a cross section obtained by cutting the virtual cylindrical column with a face orthogonal to the V axis; a step for obtaining a curved regular image constituted with an aggregate of many pixels on the twodimensional UV curved coordinate system by using a correspondence relationship equation for allowing coordinates (u, v) on the twodimensional UV curved coordinate system to correspond to corresponding coordinates (x, y) on the XY plane one by one to determine a pixel value of a pixel arranged at coordinates (u, v) on the twodimensional UV curved coordinate system on the basis of a pixel value of a pixel arranged in the vicinity of the corresponding coordinates (x, y); and a step for converting the curved regular image into a planar regular image by expanding the twodimensional UV curved coordinate system on a plane.
1 Specification
The present invention relates to an image converter and in particular to an apparatus for conducting processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting it into a planar regular image.
A fisheye lens can be used to obtain hemispherical circular images in all directions without using a mechanical moving mechanism. Therefore, a fisheye lens has been extensively used in photographing scenic pictures and others aiming at eccentric effect. However, images photographed by use of a fisheye lens are distorted in circular images, and the images may be used for artistic pictures, as they are, but not suitable for general photographic uses.
Thus, there has been proposed an apparatus for conducting process of converting a distorted circular image photographed by use of a fisheye lens into a planar regular image with less distortion. For example, Japanese Patents No. 3012142 and No. 3051173 have disclosed technologies by which a computer is used to convert a portion of a distorted circular image to a planar regular image in real time. When this conversion technique is utilized, it is possible to convert a dynamic image made up of distorted circular images photographed by use of a fisheye lens to a dynamic image made up of planar regular images and observe it in real time. Application to a monitoring system and others having a 180degree field angle is expected.
On the other hand, in Japanese Patent No. 3025255, there has been disclosed a technique for providing a panoramic display in the horizontal direction by converting a distorted circular image photographed by use of a fisheye lens into an image developed on a cylindrical face. Further, in Japanese Patent No. 3126955, there has been disclosed a technique for relieving the load of hardware by replacing division and other function operations necessary on conversion of a distorted circular image to a planar regular image with a process to refer to a lookup table.
In recent years, there has been widely used a fisheye lens monitoring system in which a security camera equipped with a fisheye lens is installed on ceilings and walls of a building to display an image photographed by use of the security camera on a monitoring device. In this application, it is required to obtain a planar regular image from which any distortion is removed as much as possible due to a necessity for recognizing the face of a person and features of clothing on a monitoring screen. However, it is difficult to obtain such a distortionfree planar regular image that can satisfy the above demand by using image converters conventionally proposed.
For example, use of the techniques disclosed in the abovedescribed Japanese Patents No. 3012142 and No. 3051173 enables to obtain a rectangular planar regular image by cutting out a part desired by a user from a distorted circular image photographed by use of a fisheye lens. However, the thus obtained planar regular image is distorted and inadequate for an application to confirm detailed features of a person. In particular, since the thus obtained rectangular planar regular image is greatly distorted in particular at an external part (in the vicinity of a contour), it is impossible to obtain a smooth panning image in a case where a cutout portion is gradually moved to conduct a horizontal panning.
On the other hand, the technique disclosed in the abovedescribed Japanese Patent No. 3025255 provides a panoramic image longer in the horizontal direction. Therefore, a part of the image is cut out to be displayed on a monitoring screen, thus making it possible to obtain a smooth panning image even when a horizontal panning is conducted. However, a panoramic image obtained by this technique is distorted greatly in the perpendicular direction although distorted to a small extent in the horizontal direction, and restriction is imposed on a region at which an image can be cut out. More specifically, even if a semisphere region is photographed, a planar regular image which is cut out from the vicinity of the top is greatly distorted. Therefore, it is actually impossible to cut out the vicinity of the top. As described above, if restriction is imposed on a part of an image that can be cut out, it is unfavorable because a blind spot is caused when the technique is applied to a security camera.
Therefore, an object of the present invention is to provide an image converter which is capable of cutting out any desired part from a distorted circular image photographed by use of a fisheye lens to convert it into a planar regular image with less distortion.
According to the image converter of the present invention, such processing is conducted so that individual pixels of a distorted circular image on a twodimensional XY orthogonal coordinate system are allowed to correspond to pixels on a twodimensional UV curved coordinate system which is curved along the side face of a cylindrical column. Therefore, it is possible to obtain a planar regular image by cutting out any desired part from a distorted circular image photographed by use of a fisheye lens and also obtain a planar regular image with less distortion.
(1) The first feature of the present invention resides in an image converter for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting said part into a planar regular image, the image converter comprising:
a distorted circular image memory for storing a distorted circular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on a twodimensional XY orthogonal coordinate system and having a radius R taking an origin O of the twodimensional XY orthogonal coordinate system as a center;
a planar regular image memory for storing a planar regular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (u, v) on a twodimensional UV orthogonal coordinate system;
a parameter input unit in which in a threedimensional XYZ orthogonal coordinate system including the twodimensional XY orthogonal coordinate system, a visual line vector n facing any given direction, with the origin O given as a starting point, is input as a parameter indicating a cutout position of the planar regular image, a predetermined planar inclination angle φ is input as a parameter indicating a cutout orientation of the planar regular image, and a predetermined magnification m is input as a parameter indicating a cutout dimension of the planar regular image;
a corresponding coordinate calculating unit calculating corresponding coordinates (x, y) which correspond to any given coordinates (u, v) by using predetermined correspondence relationship equations showing a correspondence relationship between coordinates (u, v) on a twodimensional UV curved coordinate system and coordinates (x, y) on the twodimensional XY orthogonal coordinate system, wherein said twodimensional UV curved coordinate system is defined by curving a twodimensional UV orthogonal coordinate system which is arranged on a plane passing through a point G given as an origin and orthogonal to the visual line vector n to have an orientation according to the planar inclination angle φ, said point G being away from the origin O by “a product m·R of the magnification m and the radius R” on the visual line vector n, along a side face of a “virtual cylindrical column in which the point G gives one point on the side face thereof to have a central axis parallel to a V axis of the twodimensional UV orthogonal coordinate system,”; and
a planar regular image forming unit giving coordinates (u, v) of a target pixel constituting the planar regular image to the corresponding coordinate calculating unit to obtain corresponding coordinates (x, y), reading out a pixel value of a pixel arranged in the vicinity of the obtained corresponding coordinates (x, y) inside the distorted circular image memory, determining a pixel value of the target pixel on the basis of a read pixel value, thereby forming the planar regular image by determining pixel values of individual pixels, and writing the pixel values into the planar regular image memory.
(2) The second feature of the present invention resides in an image converter according to the first feature, wherein
when, with respect to a virtual sphere having the radius R taking the origin O as a center, a corresponding point Qi is taken on the sphere which corresponds to a point Si indicated by coordinates (xi, yi) on the twodimensional XY orthogonal coordinate system depending on a projection method of a fisheye lens used, and coordinates (ui, vi) are taken on the twodimensional UV curved coordinate system at an intersecting point Ci between a straight line connecting the origin O with the corresponding point Qi on the sphere and a cylindrical column sideface coordinate plane of the twodimensional UV curved coordinate system, the corresponding coordinate calculating unit uses correspondence relationship equations by which the coordinates (xi, yi) are determined as corresponding coordinates which correspond to the coordinates (ui, vi).
(3) The third feature of the present invention resides in an image converter according to the second feature, wherein
when a distorted circular image stored in the distorted circular image memory is an orthogonally projected image photographed by use of a fisheye lens based on an orthogonal projection method, the corresponding coordinate calculating unit uses correspondence relationship equations of orthogonally projected images in which, with respect to a point Si indicated by coordinates (xi, yi), a point indicated by coordinates (xi, yi, zi) given as an intersecting point between a straight line passing through the point Si and parallel to a Z axis and the virtual sphere is given as a corresponding point Qi on the sphere, and
when a distorted circular image stored in the distorted circular image memory is a nonorthogonally projected image photographed by use of a fisheye lens based on a nonorthogonal projection method, the corresponding coordinate calculating unit uses correspondence relationship equations of nonorthogonally projected images obtained by using coordinate conversion equations between coordinates on the orthogonally projected image and coordinates on the nonorthogonally projected image so as to correct the correspondence relationship equations of orthogonally projected images.
(4) The fourth feature of the present invention resides in an image converter according to the third feature, wherein
the corresponding coordinate calculating unit defines the twodimensional UV curved coordinate system by using the virtual cylindrical column in which the point G gives one point on the side face thereof to have a central axis parallel to the V axis of the twodimensional UV orthogonal coordinate system and also passing through the origin O of the threedimensional XYZ orthogonal coordinate system.
(5) The fifth feature of the present invention resides in an image converter according to the fourth feature, wherein
the corresponding coordinate calculating unit uses the following equations as correspondence relationship equations of orthogonally projected images indicating a correspondence relationship between coordinates (u, v) and coordinates (x, y),
x=G(u′A+vB+w′C)
y=G(u′D+vE+w′F),
under the following definitions in which an angle formed between an orthogonal projection on the XY plane of a visual line vector n and a Y axis is given as an azimuthal angle α and an angle formed between the visual line vector n and a positive direction of the Z axis is given as a zenithal angle β,
A=cos φ cos α−sin φ sin α cos β
B=−sin φ cos α−cos φ sin α cos β
C=sin β sin α
D=cos φ sin α+sin φ cos α cos β
E=−sin φ sin α+cos φ cos α cosβ
F=−sin β cos α
G=R/√{square root over ( )}(w^{2}+v^{2})
w=mR
u′=w′ sin(u/w)
w′=w′ cos(u/w).
(6) The sixth feature of the present invention resides in an image converter according to the fifth feature, wherein
the corresponding coordinate calculating unit including:
a rotational coefficient operation unit in which, when the visual line vector n and the planar inclination angle φ are given from the parameter input unit, on the basis of the visual line vector n, the azimuthal angle α and the zenithal angle β are determined, and rotational coefficients A, B, C, D, E, F are calculated on the basis of the following operational equations,
A=cos φ cos α−sin φ sin α cos β
B=−sin φ cos α−cos φ sin α cos β
C=sin β sin α
D=cos φ sin α+sin φ cos α cos β
E=−sin φ sin α+cos φ cos α cos β
F=−sin β cos α;
a common coefficient operation unit in which, when the magnification m is given from the parameter input unit and a coordinate v is given from the planar regular image forming unit, the radius R of the distorted circular image is used to calculate a common coefficient G on the basis of the following operational equations,
w=mR
G=R/√{square root over ( )}(w^{2}+v^{2});
a curved coordinate correcting unit in which, when the magnification m is given from the parameter input unit and a coordinate u is given from the planar regular image forming unit, the radius R of the distorted circular image is used to calculate u′ and w′ on the basis of the following operational equations,
w=mR
u′=w′ sin(u/w)
w′=w′ cos(u/w); and
an xy coordinate value calculating unit in which the coordinate v given from the planar regular image forming unit, the rotational coefficients A, B, C, D, E, F calculated by the rotational coefficient operation unit, the common coefficient G calculated by the common coefficient operation unit, the u′ and w′ calculated by the curved coordinate correcting unit, and the radius R of the distorted circular image are used to calculate x and y on the basis of the following operational equations,
x=G(u′A+vB+w′C)
y=G(u′D+vE+w′F).
(7) The seventh feature of the present invention resides in an image converter according to the sixth feature, wherein
the parameter input unit has a function of inputting the radius R as a variable parameter on the basis of instructions from a user or a distorted circular image stored in a distorted circular image memory, and
the common coefficient operation unit and the curved coordinate correcting unit use the radius R input by the parameter input unit to perform an operation.
(8) The eighth feature of the present invention resides in an image converter according to the third feature, wherein
when a distorted circular image stored in the distorted circular image memory is an equidistantly projected image photographed by use of a fisheye lens based on the equidistance projection method, the corresponding coordinate calculating unit uses the following coordinate conversion equations for converting coordinates (xa, ya) on the orthogonally projected image into coordinates (xb, yb) on the equidistantly projected image,
xb=xa(2R/πr)sin^{−1}(r/R)
yb=ya(2R/πr)sin^{−1}(r/R)
wherein the condition is r=√(xa^{2}+ya^{2}), thereby correcting the correspondence relationship equations of orthogonally projected images.
(9) The ninth feature of the present invention resides in an image converter according to the first feature, wherein
the parameter input unit inputs coordinates (x0, y0) of a cutout center point P on the twodimensional XY orthogonal coordinate system as a parameter for defining the visual line vector n and defines a virtual sphere having the radius R at the center of an origin O in a threedimensional XYZ orthogonal coordinate system,
when a distorted circular image stored in the distorted circular image memory is an orthogonally projected image photographed by use of a fisheye lens based on the orthogonal projection method, the parameter input unit defines an intersecting point Q between a straight line passing through the cutout center point P and parallel to the Z axis and the virtual sphere, thereby giving a vector which starts from the origin O to pass through the intersecting point Q as the visual line vector n, and
when a distorted circular image stored in the distorted circular image memory is a nonorthogonally projected image photographed by use of a fisheye lens based on the nonorthogonal projection method, the parameter input unit uses coordinate conversion equations between coordinates on the orthogonally projected image and coordinates on the nonorthogonally projected image to correct coordinates of the cutout center point P and defines an intersecting point Q between a straight line passing through a point after correction and parallel to the Z axis and the virtual sphere, thereby giving a vector which starts from the origin O to pass through the intersecting point Q as the visual line vector n.
(10) The tenth feature of the present invention resides in an image converter according to the first feature, wherein
the corresponding coordinate calculating unit arranges the twodimensional UV orthogonal coordinate system in such an orientation that an angle formed between a rotational reference axis J given as an axis passing through an origin G, parallel to the XY plane and also orthogonal to the visual line vector n and the U axis is equal to the planar inclination angle φ.
(11) The eleventh feature of the present invention resides in an image converter according to the first feature, wherein
the planar regular image stored in the planar regular image memory is rectangular in contour, the twodimensional UV orthogonal coordinate system having the U axis in a direction parallel to the longer side of the rectangle and the V axis in a direction parallel to the shorter side is curved along the side face of the virtual cylindrical column having a central axis parallel to the V axis, thereby defining the twodimensional UV curved coordinate system.
(12) The twelfth feature of the present invention resides in an image converter according to the first feature, wherein
the planar regular image forming unit performs interpolation operation for pixel values of plural reference pixels on a distorted circular image arranged in the vicinity of a position indicated by corresponding coordinates (x, y) upon determination of a pixel value of a target pixel arranged at a position indicated by the coordinates (u, v).
(13) The thirteenth feature of the present invention resides in a computer readable recording medium containing a program for allowing a computer to function as the image converter according to the first feature.
(14) The fourteenth feature of the present invention resides in a semiconductor integrated circuit into which electronic circuits functioning as a corresponding coordinate calculating unit and a planar regular image forming unit which are constituents of the image converter according to the first feature are incorporated.
(15) The fifteenth feature of the present invention resides in a fisheye lens monitoring system including:
the image converter according to the first feature;
a digital camera equipped with a fisheye lens; and
a monitoring device for displaying a planar regular image on a screen;
wherein a distorted circular image photographed by use of the digital camera is stored in the distorted circular image memory, and a planar regular image obtained in the planar regular image memory is displayed by the monitoring device.
(16) The sixteenth feature of the present invention resides in an image conversion method for performing processes of cutting out a part of a distorted circular image photographed by use of a fisheye lens and converting said part into a planar regular image, the image conversion method including:
an image preparation step for storing in a distorted circular image memory a distorted circular image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on a twodimensional XY orthogonal coordinate system and having a radius R taking an origin O of the twodimensional XY orthogonal coordinate system as a center;
a parameter setting step for setting in a threedimensional XYZ orthogonal coordinate system including the twodimensional XY orthogonal coordinate system, a visual line vector n which faces any given direction with the origin O given as a starting point, as a parameter indicating a cutout position of a planar regular image, a predetermined planar inclination angle φ as a parameter indicating a cutout orientation of the planar regular image, and a predetermined magnification m as a parameter indicating a cutout dimension of the planar regular image;
a corresponding coordinate calculating step in which an operation device calculates corresponding coordinates (x, y) which correspond to coordinates (u, v) of one target pixel for a planar regular image made up of an aggregate of many pixels arranged at a positioned indicated by coordinates (u, v) on the twodimensional UV orthogonal coordinate system;
a pixel value determining step in which the operation device reads out a pixel value of a pixel arranged in the vicinity of the corresponding coordinates (x, y) inside the distorted circular image memory, determining a pixel value of the target pixel on the basis of a thus read pixel value; and
a pixel value writing step in which the operation device writes a pixel value determined with regard to the target pixel inside a planar regular image memory for storing the planar regular image;
wherein, in the corresponding coordinate calculating step, a point G on the visual line vector n spaced away from the origin O by “a product m·R of the magnification m and the radius R” being given as an origin, a twodimensional UV orthogonal coordinate system arranged to have an orientation depending on the planar inclination angle φ on a plane passing through the point G and orthogonal to the visual line vector n is curved along a side face of “a virtual cylindrical column in which the point G gives one point on the side face to have a central axis parallel to the V axis of the twodimensional UV orthogonal coordinate system,” thereby defining a twodimensional UV curved coordinate system, and predetermined correspondence relationship equations indicating a correspondence relationship between coordinates (u, v) on the twodimensional UV curved coordinate system and coordinates (x, y) on the twodimensional XY orthogonal coordinate system are used to calculate corresponding coordinates (x, y) which correspond to any given coordinates (u, v), thereby writing pixel values for all pixels necessary for constituting the planar regular image into the planar regular image memory.
(17) The seventeenth feature of the present invention resides in an image conversion method including:
a step for preparing a distorted circular image photographed by use of a fisheye lens as an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on an XY plane in a threedimensional XYZ orthogonal coordinate system;
a step for defining, at a desired position inside a space constituting the threedimensional XYZ orthogonal coordinate system, a twodimensional UV curved coordinate system constituted with a V axis arranged on a curved face along a side face of a virtual cylindrical column and parallel to a central axis of the virtual cylindrical column and a circulararc U axis along a circumference of a circle constituting a cross section obtained by cutting the virtual cylindrical column with a face orthogonal to the V axis;
a step for obtaining a curved regular image constituted with an aggregate of many pixels on the twodimensional UV curved coordinate system by using a correspondence relationship equation for allowing coordinates (u, v) on the twodimensional UV curved coordinate system to correspond to corresponding coordinates (x, y) on the XY plane one by one to determine a pixel value of a pixel arranged at coordinates (u, v) on the twodimensional UV curved coordinate system on the basis of a pixel value of a pixel arranged in the vicinity of the corresponding coordinates (x, y); and
a step for converting the curved regular image into a planar regular image by expanding the twodimensional UV curved coordinate system on a plane.
Hereinafter, a description will be given for embodiments which illustrate the present invention.
First, a description will be given for general characteristics of a distorted circular image photographed by use of a fisheye lens and a basic principle of cutting out a part thereof and converting it into a planar regular image.
The distorted circular image S formed on the XY plane is an image which forms a circle with a radius R taking an origin O as the center of a coordinate system and corresponds to an image existing in a region having a 180degree field angle on the positive region of the Z axis which has been distorted and recorded.
A fisheye lens is actually constituted with an optical system in combination with a plurality of convex lenses and concave lenses. It is known that the optical characteristics can be modeled by the virtual sphere H as shown in
As a matter of course, in optical phenomena actually found on a fisheye lens, a particular point of an object to be photographed forms an image on the particular point S (x, y) on the XY plane due to the refraction of a plurality of convex lenses and concave lenses. It is quite acceptable to hold a discussion that the optical system is replaced with a model of the virtual sphere H shown in
An object of the image converter related to the present invention is to cut out a portion of a distorted circular image S and convert it to a planar regular image. For example, it is assumed that a user who has seen the distorted circular image S shown in
Here, in order to convert an image inside the cutout region E taking the cutout center point P (x0, y0) as a center to a planar regular image, considered is the following model.
Next, at the intersecting point G (x0, y0, z0), a tangent plane in contact with the virtual sphere H is defined and a twodimensional UV orthogonal coordinate system is defined on the tangent plane. Then, the planar regular image T is to be determined as an image on the twodimensional UV orthogonal coordinate system. In the case of the example shown in
The position of the intersecting point G (x0, y0, z0) which is given as an origin of the UV coordinate system can be identified by an azimuthal angle α and a zenithal angle β, as shown in the figure. In this case, the azimuthal angle α (0≦α<360°) is an angle formed between a straight line connecting a cutout center point P (x0, y0) with an origin O of the XY coordinate system and the Y axis. The zenithal angle β (0≦β<90°) is an angle (acute angle) formed between a straight line connecting a point G (x0, y0, z0) to be given as an origin of the UV coordinate system with an origin O of the XY coordinate system and the Z axis.
As described above, the UV plane can be identified by designating the azimuthal angle α and the zenithal angle β. However, in order to determine the UV coordinate system, it is necessary to designate still another angle φ. This angle φ is a parameter showing an orientation of the UV coordinate system having a straight line OG as a rotational axis, and in the example shown in
In conclusion, the position and orientation of the UV coordinate system for forming the planar regular image T shown in
Incidentally, the image conversion to be performed in the present invention is a coordinate conversion from the XY coordinate system to the UV coordinate system. Therefore, let us take a closer look at the geometric positional relationship between the XY coordinate system and the UV coordinate system. As shown in the perspective view of
The tangent plane S2 is a plane in contact with the virtual sphere H at the point G, and the normal vector n is a vector indicating a normal line direction of the virtual sphere H at the point G. Then, the UV coordinate system is a coordinate system defined on the tangent plane S2 and a twodimensional orthogonal coordinate system defined so that the point G is given as an origin and an angle formed between the U axis and the J axis (which is an axis passing through the point G, parallel to the XY plane, and parallel to the intersecting line between the inclined face S1 and the XY plane in the figure) is given as a planar inclination angle φ.
<<<Section 2. Basic Principle of Image Conversion, with Magnification Taken into Account>>>
In Section 1, a description was given for a basic model for defining the UV coordinate system so that an origin G (x0, y0, z 0) is given as one point on the virtual sphere H. In this case, a distance between an origin O on the XY coordinate system and the origin G on the UV coordinate system is equal to a radius R. However, in most cases, a practical model is utilized in which a scaling factor is introduced into a planar regular image obtained by conversion. More specifically, a practical model is used in which a predetermined magnification m is set, the UV coordinate system is arranged at a position where a distance between two points OG is m times the radius R and a planar regular image T having a size corresponding to the magnification m is defined on the UV coordinate system. In this case, a description will be given for a basic principle of image conversion in this practical model.
Herein, the corresponding point Q (x0, y0, z0) on the sphere is a point on a virtual sphere H corresponding to a cutout center point P (x0, y0). In the case of a fisheye lens based on the orthogonal projection method, the corresponding point Q is a point defined as an intersecting point between a straight line passing through the cutout center point P (x0, y0) and parallel to the Z axis and the virtual sphere H. Since a visual line vector n is defined as a vector extending from the origin O to the corresponding point Q (x0, y0, z0) on the sphere, setting the cutout center point P (x0, y0) is equal in meaning to setting the visual line vector n.
As a matter of course, in the practical model shown in
An object of the image conversion performed here is to cut out a distorted image inside a cutout region taking the cutout center point P (x0, y0) as the center on a distorted circular image S defined on the XY coordinate system and deform the image, thereby obtaining a planar regular image T on the UV coordinate system. More specifically, a pixel value of a pixel positioned at one point Ti (ui, vi) on the planar regular image T on the UV coordinate system is determined on the basis of a pixel value of a pixel positioned in the vicinity of one point Si (xi, yi) on the XY coordinate system corresponding thereto. For this reason, as will be described in Section 3, correspondence relationship equations showing a correspondence relationship between coordinates (ui, vi) and coordinates (xi, yi) are required.
In performing the abovedescribed image conversion, the visual line vector n functions as a parameter indicating a cutout position of a planar regular image. Where the visual line vector n is set in the illustrated direction, the planar regular image is to be cut out from the inside of a cutout region taking the cutout center point P (x0, y0) as the center. Changing the direction of the visual line vector n changes the position of the cutout center point P, by which the position at which the planar regular image is cut out changes. On the other hand, the planar inclination angle φ functions as a parameter indicating a cutout orientation of the planar regular image, and the magnification m (a factor of determining a distance between two points OG) functions as a parameter indicating a cutout size of the planar regular image.
A comparison of
On the other hand, the planar inclination angle φ is a parameter indicating a cutout orientation of a planar regular image. Changing the planar inclination angle φ changes the positional relationship with an image frame of a photographic subject appearing inside the planar regular image (positional relationship with respect to a rotational direction). This will be easily understood from the fact that as shown in
In conclusion, a user is able to obtain a desired planar regular image T on the UV coordinate system by setting three parameters, that is, a visual line vector n (cutout center point P), a magnification m and a planar inclination angle φ. Further, where the thus obtained planar regular image T is not satisfactory, by appropriately adjusting these three parameters, the planar regular image T can be adjusted. More specifically, if the user is not satisfied with an orientation of the thus obtained image, the user may adjust the planar inclination angle φ. If not satisfied with a field angle of the obtained image, the user may adjust the magnification m, and if not satisfied with a cutout position of the obtained image, the user may adjust the visual line vector n (cutout center point P).
In this case, a user may adjust the planar inclination angle φ. For example, when the angle φ is decreased by approximately 90 degrees, the U axis (and the V axis) are rotated clockwise, and the image frame of the planar regular image T is also rotated clockwise, thereby obtaining an upright image of the woman. However, since the illustrated monitor screen is provided with a rectangular frame which is smaller in the vertical dimension b (the number of pixels in the perpendicular direction) than in the transverse dimension a (the number of pixels in the horizontal direction), it is necessary to adjust the magnification m to some extent in order to display an area below the neck of the woman.
It is noted that “the planar regular image” described in the present application does not necessarily mean “a perfect image free of any distortion” but means “a planar image which is closer to an image photographed by use of an ordinary lens than a distorted circular image S photographed by use of a fisheye lens.” Therefore, as compared with the image of the woman on the distorted circular image S shown in
On the other hand,
As described above, in order to obtain the planar regular image T on the twodimensional UV orthogonal coordinate system, it is necessary to determine a pixel value of a pixel positioned at the point Ti (ui, vi) shown in
The abovedescribed correspondence relationship equation can be determined unambiguously by a geometrical method, when the UV coordinate system arranged inside a space of the threedimensional XYZ coordinate system is determined for the position and orientation. For example, in the example shown in
A corresponding point S1 (x1, y1) on the distorted circular image S with regard to any given point T1 (u1, v1) on the planar regular image T is defined as follows. More specifically, a corresponding point Q1 on the sphere is determined at a position of an intersecting point between a straight line connecting the point T1 (u1, v1) with an origin O and a virtual sphere H, and a point on the distorted circular image S at a position immediately below the corresponding point Q1 on the sphere may be given as the corresponding point S1 (x1, y1). Similarly, any given point T2 (u2, v2) on the planar regular image T is defined by processes in which a corresponding point Q2 on the sphere at a position of an intersecting point between a straight line connecting a point T2 (u2, v2) with the origin O and the virtual sphere H is determined to give a point on the distorted circular image S immediately below the corresponding point Q2 on the sphere as a corresponding point S2 (x2, y2).
A correspondence relationship between these two points can be described by geometrical correspondence relationship equations. More specifically, it is known that the relationship can be described by equations (1) to (9) shown in
The correspondence relationship equations shown in
More specifically, the following equation:
x=R(uA+vB+wC)/√{square root over ( )}(u^{2}+v^{2}+w^{2}) Equation (1)
is to determine an xcoordinate value of the corresponding point S (x, y) on the XY coordinate system by using coordinate values u and v at one point T(u, v) on the UV coordinate system. A, B and C are values respectively determined by the following mathematical equations:
A=cos φ cos α−sin φ sin α cos β Equation (3)
B=−sin φ cos α−cos φ sin α cos β Equation (4)
C=sin β sin α Equation (5).
They are determined by operation by use of trigonometric functions of Euler angles α, β, φ.
In a similar manner, the following equation:
y=R(uD+vE+wF)/√{square root over ( )}(u^{2}+v^{2}+w^{2}) Equation (2)
is to determine a ycoordinate value of the corresponding point S (x, y) on the XY coordinate system by use of coordinate values u, v at one point T(u, v) on the UV coordinate system. D, E and F are values respectively determined by the following mathematical equations:
D=cos φ sin α+sin φ cos α cos β Equation (6)
E=−sin φ sin α+cos φ cos α cos β Equation (7)
F=−sin β cos α Equation (8).
They are determined by operation by use of trigonometric functions of Euler angles α, β, φ.
It is noted that w in the equations (1) and (2) is a value given by the following equation:
w=mR Equation (9)
wherein R is a radius of the distorted circular image S as described above, and m is a magnification. The magnification m indicates a relationship between the scaling of coordinate values u, v and the scaling of coordinate values x, y. The greater the setting of the magnification m, the greater an enlarged image is required for a planar regular image T. However, the cutout region E of the distorted circular image S is made smaller.
In conclusion, in the equation shown in
Under the abovedescribed special condition, a positional relationship between any given point T (u, v) on the planar regular image T and the corresponding point S (x, y) on the distorted circular image S will be as illustrated. More specifically, a point Q on the virtual sphere H immediately above the point S (x, y) is determined and an intersecting point between a straight line connecting two points OQ and the coordinate plane of the UV coordinate system is determined, thereby providing a relationship in which the intersecting point concerned is given as a point T (u, v). Therefore, the following simple correspondence relationship equations are obtained between the coordinates (x, y) and coordinates (u, v).
x=(R/L)·u Equation (10)
y=(R/L)·v Equation (11)
The equations (1) and (2) shown in
The image conversion method disclosed in the abovedescribed Patent Documents 1, 2 and others is to convert a part of the distorted circular image S into the planar regular image T by use of the correspondence relationship equations shown in
The present invention is to present a new conversion method for solving the above problems. Here, a description will be given for the basic concept with reference to
The planar regular image T indicated by the broken line in
In other words, the twodimensional UV curved coordinate system defined by the model shown in
Since the twodimensional UV curved coordinate system is also a twodimensional coordinate system having the U axis and the V axis, a point at which any given one point inside the curved regular image C is indicated by coordinates (u, v) is the same as in the case of an ordinary twodimensional coordinate system on a plane. As described above, the twodimensional UV curved coordinate system is a coordinate system defined by giving a point G as an origin and curving the twodimensional UV orthogonal coordinate system arranged on a plane orthogonal to a visual line vector n along the side face of the virtual cylindrical column. Some conditions are, however, imposed on this curving process.
First, the virtual cylindrical column used in the curving process is to be a cylindrical column having a central axis parallel to the V axis of the twodimensional UV orthogonal coordinate system. In particular, in the present embodiment, used is a virtual cylindrical column, the radius of which is equal to mR (a distance between points OG). Then, the virtual cylindrical column is arranged so that the point G is positioned on the side face thereof. In conclusion, when a virtual cylindrical column that can satisfy these conditions is arranged on the threedimensional XYZ orthogonal coordinate system, the central axis of the virtual cylindrical column is, as shown by the single dotted and dashed line in
This curving process will be described by using an actual object in a metaphorical manner as follows. First, as shown in
Next, a cylindrical column having a radius mR is prepared and the cylindrical column is arranged so that the central axis is positioned on the axis V′. In this case, the axis V′ is a straight line parallel to the V axis depicted on the slip of paper and passing through the origin O. When the cylindrical column is arranged as positioned above, the V axis depicted on the slip of paper is kept in contact with the side face of the cylindrical column. As a matter of course, the center point T (0,0) of the slip of paper, that is, a point G (xg, yg, zg), is also a point in contact with the side face of the cylindrical column. In this case, this slip of paper is curved so as to roll around the side face of the cylindrical column and pasted on the side face of the cylindrical column, as it is. Thereby, the coordinate system depicted on the curved slip of paper is to be a twodimensional UV curved coordinate system.
In conclusion, the V axis of the twodimensional UV curved coordinate system is exactly the same as the V axis of the twodimensional UV orthogonal coordinate system and given as a straightline coordinate axis. However, the U axis of the twodimensional UV curved coordinate system is a circulararc axis along the side face of the cylindrical column. In other words, the twodimensional UV curved coordinate system is constituted with the V axis parallel to the central axis V′ of the virtual cylindrical column and the circulararc U axis along the circumference of a circle constituting a cross section when the virtual cylindrical column is cut by a face orthogonal to the V axis. As a matter of course, a coordinate graduation engraved on the U axis is to be a graduation defined along the circular arc.
Now, a principle of obtaining the curved regular image C on the twodimensional UV curved coordinate system is completely identical with a principle of obtaining the planar regular image T on the twodimensional UV orthogonal coordinate system. More specifically, one point Ci (ui, vi) on the curved regular image C is allowed to correspond to one point Si (xi, yi) on the distorted circular image S, by which a pixel value of a pixel positioned at the point Ci (ui, vi) may be determined on the basis of a pixel value of a pixel positioned in the vicinity of the point Si (xi, yi). For this reason, a oneforone correspondence relationship equation is required which allows any given coordinates (u, v) on the twodimensional UV curved coordinate system to correspond to corresponding coordinates (x, y) on an XY plane. As will be described in detail in Section 5, the correspondence relationship equation is obtainable by correcting the correspondence relationship equations shown in
As described above, after the curved regular image C constituted with an aggregate of many pixels is obtained on the twodimensional UV curved coordinate system, the twodimensional UV curved coordinate system is expanded on a plane, by which the curved regular image C may be converted into the planar regular image T. The expansion on the plane may be taken as a work for peeling off a slip of paper pasted on the side face of a cylindrical column and pasting it again on the plane in the abovedescribed example.
However, the expansion on the plane will not need any specific work in terms of operation. Pixels (pixels constituting the curved regular image C) positioned at coordinates (u, v) on the twodimensional UV curved coordinate system are, pixels located at the same coordinates (u, v), even if they are given as pixels (pixels constituting the planar regular image T) on the twodimensional UV orthogonal coordinate system by the expansion on the plane. Therefore, where pixel array data showing pixel values of many pixels located at predetermined coordinates (u, v) on the twodimensional UV curved coordinate system is obtained, the pixel array data is output, as it is, as pixel array data on the twodimensional UV orthogonal coordinate system and an image is displayed on an ordinary monitor screen (a planar screen). Thereby, the planar regular image T after the expansion on the plane is to be displayed on a monitor screen.
As described above, the pixel array data obtained for displaying the curved regular image C on the twodimensional UV curved coordinate system which is originally to give a side wall of the cylindrical column is used to display an image on an ordinary monitor screen (a screen on the plane) having the twodimensional UV orthogonal coordinate system. This processing corresponds to the work in which, in the abovedescribed example, a slip of paper pasted on the side wall of the cylindrical column is peeled off and again pasted on the plane. The display screen will be changed from the side face of the cylindrical column to the plane, but no change is needed for the pixel array data itself.
In conclusion, in the present invention, adopted is a method in which, in place of obtaining the planar regular image T on the twodimensional UV orthogonal coordinate system, the curved regular image C is obtained on the twodimensional UV curved coordinate system defined on the side face of the cylindrical column, and pixel array data showing the curved regular image C is used to display an image on an ordinary monitor screen (a planner screen). Thereby, the planar regular image T is finally obtained on the monitor screen.
On the basis of the abovedescribed basic principle, a part of a distorted circular image photographed by use of a fisheye lens is cut out and converted into a planar regular image. This conversion may be made by conducting the following processes.
First, performed is an image preparation step in which the distorted circular image S constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on the twodimensional XY orthogonal coordinate system and having a radius R at the center of an origin O of the twodimensional XY orthogonal coordinate system is incorporated into a distorted circular image memory as pixel array data. The thus incorporated distorted circular image S is an image obtained by photography by use of a fisheye lens and also an image to be converted. A planar regular image memory for storing the planar regular image T obtained by conversion is also provided.
Next, a setting is performed, such as, which part of the distorted circular image S is to be put at the center, in which orientation and to what extent the size the image is to be cut out. As described above, this setting is performed by setting the following three parameters. First, in a threedimensional XYZ orthogonal coordinate system including the above twodimensional XY orthogonal coordinate system, a visual line vector n starting from the origin O to face any given direction is set as a parameter indicating a cutout position of a planar regular image. This visual line vector n is geometrically a vector which is regulated by an azimuthal angle α and a zenithal angle β. However, in practice, as described above, the vector can be defined by designating a cutout center point P (x0, y0). Next, a predetermined planar inclination angle φ is set as a parameter indicating the cutout orientation of the planar regular image, and a predetermined magnification m is set as a parameter indicating the cutout size of the planar regular image.
When the abovedescribed settings are made, the preparation for conversion has been completed. Thereafter, an operation device (which may be constituted with hardware using an LSI and others or may be constituted by installing dedicated software for a general computer) having functions to gain access to the abovedescribed individual memories is used to perform each of the following processings.
First, a corresponding coordinate calculating step is performed in which coordinates (x, y) which correspond to coordinates (u, v) of one target pixel, with respect to a planar regular image (an image made up of an aggregate of many pixels arranged at a position indicated by coordinates (u, v) on the twodimensional UV orthogonal coordinate system) to be stored inside a planar regular image memory, are calculated. However, in reality, in place of calculating the corresponding coordinates (x, y) which correspond to coordinates (u, v) on the twodimensional UV orthogonal coordinate system, calculation is made for the corresponding coordinates (x, y) which correspond to the coordinates (u, v) on the twodimensional UV curved coordinate system. More specifically, as described in the model of
Then, a pixel value of a pixel arranged in the vicinity of the thus calculated corresponding coordinates (x, y) is read out from the distorted circular image memory, and a pixel value of the target pixel is determined on the basis of the readout pixel value. Further, a target pixel inside the planar regular image memory for storing a planar regular image is subjected to processing for writing the thus determined pixel value. Therefore, pixel values of all pixels necessary for constituting the planar regular image may be written into the planar regular image memory.
As described above, the twodimensional UV orthogonal coordinate system and the twodimensional UV curved coordinate system are common in that they are both twodimensional coordinate systems having the U axis and the V axis. Accordingly, a planar regular image T defined on the twodimensional UV orthogonal coordinate system and a curved regular image C defined on the twodimensional UV curved coordinate system are common in that they are constituted with an aggregate of many pixels arranged at a position indicated by coordinate value (u, v). Therefore, on the basis of pixel array data stored inside the planar regular image memory, an image is displayed on a plane, thereby obtaining the planar regular image T. However, when an image is displayed on the side face of a cylindrical column, the curved regular image C is to be obtained. In this case, for the sake of convenience in explanation, the memory for storing the pixel array data after conversion is referred to as “a planar regular image memory.” The pixel array data to be stored into the memory itself may be referred to as data for the planar regular image T or also referred to as data for the curved regular image C.
A difference of characteristics between a conventional conversion method shown in the model of
As described above, a difference between these is only in that the correspondence relationship equations to be used are different. This difference will greatly improve a distortion on the planar regular image T. In particular, the vicinity of the right contour and the vicinity of the left contour on an image in the example shown in
The twodimensional UV curved coordinate system is a coordinate system on the side face of a cylindrical column. Thus, even when the U axis is a circulararc axis, the V axis is a straight line axis. For this reason, the vicinity of the upper contour and the vicinity of the lower contour on an image in the example shown in
In the basic embodiment described here, “a virtual cylindrical column in which a point G gives one point on the side face thereof to have a central axis parallel to the V axis of the twodimensional UV orthogonal coordinate system and passing through the origin O of the threedimensional XYZ orthogonal coordinate system” is used to define the twodimensional UV curved coordinate system. Therefore, as shown by the single dotted and dashed line in
However, in practical use, as described in the basic embodiment, it is preferable that “a cylindrical column having a radius mR, with a straight line V′ passing through the origin O given as a central axis” is used as the virtual cylindrical column. This is because a curvature radius of the virtual cylindrical column is made equal to the mR, thereby, at least at a part along the circulararc U axis, a distance from the origin O is a constant value, mR, and an optimal distortion correction effect may be obtained. In reality, experiments conducted by the present inventor have provided an ideal planar regular image which is least distorted, where “a cylindrical column having a radius mR, with a straight line V′ passing through the origin O given as a central axis” is used as the virtual cylindrical column.
Further, since a display screen of a monitoring device for general use is rectangular in contour, it is preferable to obtain a planar regular image having a rectangular contour for adapting to the display screen concerned in view of carrying out the present invention. As described above, where the planar regular image stored inside a planar regular image memory is rectangular in contour, it is preferable that a twodimensional UV orthogonal coordinate system having the U axis in a direction parallel to the longer side of the rectangle and the V axis in a direction parallel to the shorter side is curved along the side face of the virtual cylindrical column having a central axis parallel to the V axis, thereby defining the twodimensional UV curved coordinate system.
This is because, in the present invention, a coordinate axis which is curved in a circulararc shape is referred to as the U axis, while a coordinate axis which is formed in a straight line is referred to as the V axis, and when the U axis is given in a direction parallel to the longer side, a greater effect is obtained in curving the ends. An extent that both right and left ends come out from a plane in a case where a cylindrical column is arranged on a planar regular image T having a rectangular contour as shown in
As described above, a substantial difference between a conventional conversion method shown by the model of
As described above in Section 3, the equations (1) to (9) in
On the other hand, a UV curved face, which is a coordinate plane of the twodimensional UV curved coordinate system, is a face obtained by curving the UV plane along the side face of the virtual cylindrical column. In this case, the virtual cylindrical column is a cylindrical column in which the Y axis (an axis passing the origin O in a perpendicular direction with respect to the drawing sheet) is given as a central axis to have a radius w=mR and in contact with the UV plane at a position of the V axis. The UV curved face is in alignment with the side face of the virtual cylindrical column, the U axis is defined in a direction along a circular arc of a circle having a radius w=mR, and the V axis is defined at the point G in a perpendicular direction with respect to the drawing sheet. Next, the curved regular image C is defined on the UV curved face.
As described above, the twodimensional UV orthogonal coordinate system and the twodimensional UV curved coordinate system are common in that they both have the V axis which is a first coordinate axis but different in that spatial position of the U axis which is a second coordinate axis. However, their coordinate graduations are all engraved in the same width. They are also common in that the position of any given one point on the coordinate plane is indicated by coordinates (u, v). Then, consideration will be given to a relationship of twopoints on each coordinate plane indicated by the same coordinates (u, v).
First, as shown at the above left in the figure, attention is paid to a point T (u, v) on the twodimensional UV orthogonal coordinate system. This point T (u, v) is a point on the planar regular image T, spaced away from the origin G by a coordinate value u along the U axis (the point is plotted at a position spaced away by the length of a segment D depicted by the heavy line in the figure), and a point spaced away from the origin G in the V axis direction by a coordinate value v (a distance in the V axis direction is a distance in a perpendicular direction with respect to the drawing sheet, and therefore not shown in the figure).
Next, attention is paid to a point C (u, v) on the twodimensional UV curved coordinate system. This point C (u, v) is a point on the curved regular image C, spaced away from the origin G along a circular arc A by a coordinate value u, and a point spaced away from the origin G in the V axis direction by a coordinate value v (a distance in the V axis direction is a distance in a perpendicular direction with respect to the drawing sheet, and therefore not shown in the figure). Here, the length of the circular arc A depicted by the heavy line is equal to the length of the segment D depicted by the heavy line.
As described above, the point T (u, v) and the point C (u, v) are both points expressed by coordinates (u, v). However, since these are points defined by different coordinate systems, these are different in spatial position indicated by an XYZ threedimensional orthogonal coordinate system. More specifically, in view of a coordinate value of the XYZ threedimensional orthogonal coordinate system, coordinates of the point T (u, v) are given as T (u, v, w), while coordinates of the point C (u, v) are given as C (u′, v, w′). Here, as shown in the figure, the following equations are obtained.
u′=w·sin θ Equation (12)
w′=w·cos θ Equation (13)
Further, since the length of the circular arc A is equal to an absolute value of the coordinate value u and the radius of the circular arc A is w, the angle θ shown in
u′=w·sin(u/w) Equation (14)
w′=w·cos(u/w) Equation (15)
It is noted that both of these UV coordinate systems are common in that the V axis is a coordinate axis and therefore, Y coordinate values both at the points T and C have the same coordinate value v.
Accordingly, in place of a variable u in the equations (1), (2) shown in
x=R(u′A+vB+w′C)/√{square root over ( )}(u′^{2}+v^{2}+w′^{2}) Equation (1′)
y=R(u′D+vE+w′F)/√{square root over ( )}(u′^{2}+v^{2}+w′^{2}) Equation (2′)
In this case, there is no change in correction of rotation factors based on Euler angle α, β, φ. Thus, rotational coefficients A to F can be calculated by using the equations (3) to (8) shown in
Thus, correspondence relationship equations used in the present invention, that is, correspondence relationship equations which show a correspondence relationship between coordinates (u, v) on the twodimensional UV curved coordinate system and coordinates (x, y) on the twodimensional XY orthogonal coordinate system will be given as the equations (1′), (2′) shown above in
With convenience of actual operation taken into account, a common coefficient G is defined as
G=R/√{square root over ( )}(u′^{2}+v^{2}+w′^{2}),
and the equations (14), (15) are substituted for u′ and w′ in the above equation to obtain the following as shown in the middle of
G=R/√{square root over ( )}(w^{2}+v^{2}).
In conclusion, the correspondence relationship equations used in the present invention will be defined by the following equation as shown below in
x=G(u′A+vB+w′C) Equation (16)
y=G(u′D+vE+w′F) Equation (17)
G=R/√{square root over ( )}(w^{2}+v^{2}) Equation (18)
and the equations (3) to (9) in
First, the block B1 indicated by the broken line above in the figure includes parameters determined by instructions input from a user. More specifically, as a parameter indicating a cutout position of a planar regular image, a visual line vector n is given, as a parameter indicating a cutout orientation, a planar inclination angle φ is given, and as a parameter indicating a cutout dimension, a magnification m is given. In this case, the visual line vector n is actually given as a position of a cutout center point P (x0, y0) on a distorted circular image S and an azimuthal angle α and a zenithal angle β can be determined on the basis of the information thereof.
Further, the block B2 indicated by the broken line in the above right in the figure includes a radius R of the distorted circular image S. The radius R is a parameter inherent to a fisheye lens to be used. Where the present invention is applied to a fisheye lens monitoring system and others, a fisheye lens usually mounted on a monitoring camera is a commercially available one. Therefore, a value of the radius R inherent to the fisheye lens concerned may be set in advance. As a matter of course, in a system in which a fisheye lens can be selected from various types thereof and used, it is necessary to set a radius R inherent to the selected fisheye lens.
The block B9 shown by the broken line below in the figure includes coordinates (u, v) on the twodimensional UV curved coordinate system, and the block B8 shown by the broken line left below in the figure includes coordinates (x, y) on the twodimensional XY orthogonal coordinate system. In conclusion, the process shown in
The block B3 includes the equations (3) to (8) in
The block B4 includes the equation (9) in
Then, the block B7 includes the equations (16), (17) shown in
As described above, on the basis of any given coordinates (u, v) on the twodimensional UV curved coordinate system, calculation is to be made for corresponding coordinates (x, y) on the twodimensional XY orthogonal coordinate system corresponding thereto.
Herein, a description will be given for an image converter of one embodiment of the present invention with reference to the block diagram in
Of these individual constituents, a part constituted with the parameter input unit 100, the corresponding coordinate calculating unit 200, the planar regular image forming unit 300, the planar regular image memory 420 and the distorted circular image memory 440 is the image converter of the present invention, having a function of cutting out a part of a distorted circular image photographed by use of a fisheye lens to convert it into a planar regular image. A fisheye lens monitoring system is constituted by adding, to the image converter, the fisheye lensmounted digital camera 410 for photographing a distorted circular image S and the monitoring device 430 for displaying a planar regular image after conversion.
The fisheye lens mounted digital camera 410 is a digital video camera generally used in a monitoring camera and others, having a function of photographing a subject in the vicinity at a 180degree field angle and generating the distorted circular image S with a radius R as digital data. The distorted circular image S generated as the digital data is stored into the distorted circular image memory 440. The image converter converts it into a planar regular image T and stores the image into the planar regular image memory 420. Therefore, the planar regular image T obtained at the planar regular image memory is displayed on a screen by the monitoring device 430.
The distorted circular image memory 440 is constituted with an image data storing buffer memory for general use, thereby storing the distorted circular image S as pixel array data constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on the twodimensional XY orthogonal coordinate system. This distorted circular image S is, as described already, generated as an image having a radius R, with an origin O of the twodimensional XY orthogonal coordinate system given at the center.
On the other hand, the planar regular image memory 420 is also constituted with an image data storing buffer memory for general use, thereby storing the planar regular image T as pixel array data constituted with an aggregate of many pixels arranged at a position indicated by coordinates (u, v) on the twodimensional UV orthogonal coordinate system. In most cases, the planar regular image T is generated as a rectangular image appropriate for displaying on a display screen of the monitoring device 430. As a matter of course, the planar regular image T may be generated in any contour shape.
In this case, for the sake of conveniences in explanation, a memory for storing the pixel array data after conversion is referred to as “the planar regular image memory 420.” However, as described in Section 4, the pixel array data stored inside the memory 420 itself may be also referred to as data of planar regular image T or may be referred to as data of curved regular image C. In reality, where the distorted circular image S stored inside the distorted circular image memory 440 is converted into a regular image on the UV coordinate system, as described by using the model of
The distorted circular image S stored inside the distorted circular image memory 440 is converted into the planar regular image T and written directly into the planar regular image memory 420. This direct processing is executed by the planar regular image forming unit 300. However, in carrying out this processing, it is necessary to obtain corresponding coordinates (x, y) on the XY coordinate system which corresponds to coordinates (u, v) on the UV coordinate system. The corresponding coordinate calculating unit 200 is to deal with the processing of calculating the corresponding coordinates (x, y). More specifically, the corresponding coordinate calculating unit 200 has a function of carrying out the operation using correspondence relationship equations described in Section 5 where any given coordinates (u, v) are given from the planar regular image forming unit 300, thereby calculating the corresponding coordinates (x, y) and returning these to the planar regular image forming unit 300.
In order to execute the operation using the correspondence relationship equations described in Section 5, there are needed three parameters, that is, a visual line vector n, a planar inclination angle φ and a magnification m. The parameter input unit 100 is a constituent which inputs these three parameters on the basis of instructions input from a user. More specifically, the parameter input unit 100 has functions of inputting the visual line vector n facing any given direction starting from an origin O in the threedimensional XYZ orthogonal coordinate system as a parameter indicating a cutout position of a planar regular image, inputting the predetermined planar inclination angle φ as a parameter indicating a cutout orientation of the planar regular image and also inputting the predetermined magnification m as a parameter indicating a cutout dimension of the planar regular image.
It is noted that in the present embodiment, the monitoring device 430 is designed to display the distored circular image S stored inside the distorted circular image memory 440, when necessary. Then, the parameter input unit 100 has a function of receiving instructions input from a user who designates one point on the distorted circular image S displayed on the monitoring device 430, thereby recognizing a position of the one point concerned as a cutout center point P(x0, y0) and taking it as a parameter indicating the visual line vector n.
For example, in a state that the distorted circular image S shown in
The corresponding coordinate calculating unit 200 uses the thus set individual parameters and the previously set radius R to perform the operation of calculating corresponding coordinates (x, y) which correspond to any given coordinates (u, v) given from the planar regular image forming unit 300. Correspondence relationship equations used in the operation are, as shown in Section 5. The gist thereof will be given as follows.
More specifically, as shown in the model of
Next, a definition will be made for “a virtual cylindrical column in which the point G gives one point on the side face to have a central axis parallel to the V axis of he twodimensional UV orthogonal coordinate system.” As described above, in practical use, it is preferable that the radius of the virtual cylindrical column is set to be mR and the central axis gives a straight line V′ passing through the origin O and parallel to the V axis. Then, the twodimensional UV orthogonal coordinate system is curved along the side face of the virtual cylindrical column, thereby defining the twodimensional UV curved coordinate system.
Correspondence relationship equations used in an operation by the corresponding coordinate calculating unit 200 may be any equation which shows a correspondence relationship between coordinates (u, v) on the thus defined twodimensional UV curved coordinate system and coordinates (x, y) on the twodimensional XY orthogonal coordinate system. More specifically, such a correspondence relationship equation may be geometrically determined by which, when on a virtual sphere having a radius R at the center of an origin O, a corresponding point Qi on the sphere corresponding to a point Si indicated by coordinates (xi, yi) on the twodimensional XY orthogonal coordinate system is taken depending on a projection method of the fisheye lens used and given as (ui, vi) are coordinates on the twodimensional UV curved coordinate system of an intersecting point Ci between a straight line connecting the origin O with the corresponding point Qi on the sphere and a cylindrical column sideface coordinate plane of the twodimensional UV curved coordinate system, the coordinates (xi, yi) are determined as corresponding coordinates which correspond to the coordinates (ui, vi).
In particular, as in the examples described above, where the distorted circular image S stored in the distorted circular image memory 440 is an orthogonally projected image photographed by a fisheye lens based on an orthogonal projection method, as shown in
Then, in the orthogonal projection method, as shown in
x=G(u′A+vB+w′C) Equation (16)
y=G(u′D+vE+w′F) Equation (17)
G=R/√{square root over ( )}(w^{2}+v^{2}) Equation (18).
In the above equations, rotational coefficients A to E will be given as follows:
A=cos φ cos α−sin φ sin α cos β Equation (3)
B=−sin φ cos α−cos φ sin α cos β Equation (4)
C=sin β sin αEquation (5)
D=cos φ sin α+sin φ cos α cos β Equation (6)
E=−sin φ sin α+cos φ cos α cos β Equation (7)
F=−sin β cos α Equation (8).
In this case, the azimuthal angle α is an angle formed between an orthogonal projection on an XY plane of the visual line vector n and the Y axis, the zenithal angle β is an angle formed by the visual line vector n and a positive direction of the Z axis, and the planar inclination angle φ is an angle set as a parameter.
The corresponding coordinate calculating unit 200 shown in
The common coefficient operation unit 210 performs such processing that when a magnification m is given from the parameter input unit 100 and a coordinate v is given from the planar regular image forming unit 300, a previously set radius R of a distorted circular image is used to calculate a common coefficient G on the basis of the following operational equations:
w=mR Equation (9)
G=R/√{square root over ( )}(w^{2}+v^{2}) Equation (18).
On the other hand, the curved coordinate correcting unit 220 has a function of performing an operation necessary for correcting coordinates (u, v) on the twodimensional UV orthogonal coordinate system defined on a plane to give coordinates (u, v) on the twodimensional UV curved coordinate system defined on the side face of a cylindrical column. Thus, the curved coordinate correcting unit 220 performs such processing that when a magnification m is given from the parameter input unit 100 and a coordinate u is given from the planar regular image forming unit 300, a previously set radius R of a distorted circular image is used to calculate u′ and w′ on the basis of the following operational equations:
w=mR Equation (9)
u′=w′ sin(u/w) Equation (14)
w=w′ cos(u/w) Equation (15).
If sin (u/w) of the equation (14) is rewritten to be cos (u/w+π/2), it is possible to calculate both the equation (14) and the equation (15) by an operation unit of cos function. Therefore, it is possible to reduce hardware to be mounted. (As a matter of course, similar results may be obtained by rewriting the equation (15) into an operational equation of sin function and both the equations can be calculated by an operation unit of sin function.)
Further, the rotational coefficient operation unit 230 performs such processing that when a visual line vector n and a planar inclination angle φ are given from the parameter input unit 100, the azimuthal angle α and the zenithal angle β are determined on the basis of the visual line vector n to calculate rotational coefficients A, B, C, D, E, and F on the basis of the following operational equations:
A=cos φ cos α−sin φ sin α cos β Equation (3)
B=−sin φ cos α−cos φ sin α cos β Equation (4)
C=sin β sin α Equation (5)
D=cos φ sin α+sin φ cos α cos β Equation (6)
E=−sin φ sin α+cos φ cos α cos β Equation (7)
F=−sin β cos α Equation (8).
Then, the xy coordinate value calculating unit 240 performs such processing that a coordinate v given from the planar regular image forming unit 300, rotational coefficients A, B, C, D, E, F calculated by the rotational coefficient operation unit 230, a common coefficient G calculated by the common coefficient operation unit 210, u′ and w′ calculated by the curved coordinate correcting unit 220 and a previously set radius R of a distorted circular image are used to calculate x and y on the basis of the following operational equations:
x=G(u′A+vB+w′C) Equation (16)
y=G(u′D+vE+w′F) Equation (17)
thereby giving the result to the planar regular image forming unit 300. It is noted that since the operation of the above equation (16) is different from that of the equation (17) only in a combination of rotational coefficients to be used, the same operational circuit is in practice used to calculate the equation (16) and the equation (17) by time division, thus making it possible to reduce hardware to be mounted.
Further, in
A specific description has been so far given for operation processing performed by the corresponding coordinate calculating unit 200 shown in
More specifically the planar regular image forming unit 300 first gives coordinates (u, v) of one target pixel constituting a planar regular image to the corresponding coordinate calculating unit 200 so that the corresponding coordinates (x, y) can be calculated. Next, it reads out a pixel value of a pixel arranged in the vicinity of the corresponding coordinates (x, y) inside the distorted circular image memory 440, performing processing for determining a pixel value of a target pixel on the basis of the thus read pixel value with regard to individual pixels constituting the planar regular image, and writing pixel values of the individual pixels into the planar regular image memory 420, thereby forming the planar regular image.
In the example shown in
In this case, the planar regular image memory controller 320 is a control device for writing data into and reading it from the planar regular image memory 420. When a pixel value of a specific pixel is determined by the pixel value determining unit 330, the planar regular image memory controller 320 writes the determined pixel value into the specific pixel concerned inside the planar regular image memory 420. Thereby, upon completion of processing for writing the pixel values for all pixels, a planar regular image T is to be formed inside the planar regular image memory 420. Therefore, this time, the planar regular image memory controller 320 reads out data of the planar regular image T, outputs it on the monitoring device 430, thereby performing processing for displaying the planar regular image T on a monitor screen.
On the other hand, the distorted circular image memory controller 340 is a control device for writing into and reading out data from the distorted circular image memory 440. Data of a distorted circular image S photographed by use of a fisheye lensmounted digital camera 410 is to be written into the distorted circular image memory 440 by the distorted circular image memory controller 340. Further, whenever necessary, the distorted circular image memory controller 340 is also able to read out the data of the distorted circular image S inside the distorted circular image memory 440 and output it on the monitoring device 430, thereby displaying the distorted circular image S on a monitor screen. Still further, when coordinates (x, y) are given from the corresponding coordinate calculating unit 200, the distorted circular image memory controller 340 has a function of reading out a pixel value of a pixel positioned in the vicinity of the coordinates (x, y) from the data of the distorted circular image S inside the distorted circular image memory 440, thereby giving it to the pixel value determining unit 330.
Processes of forming the planar regular image T by the planar regular image forming unit 300 as shown in the figure will be performed as follows. Here, a description will be given for a case where the planar regular image T is constituted with pixel array data in which the U axis is disposed in a line direction and the V axis is disposed in a column direction. First, the uv coordinate value producing unit 310 produces a pair of coordinates (u, v) indicating one target pixel on a pixel array constituting the planar regular image T. The thus produced coordinates (u, v) are given from the uv coordinate value producing unit 310 to the corresponding coordinate calculating unit 200. Thereby, corresponding coordinates (x, y) which correspond to the coordinates (u, v) are calculated and these corresponding coordinates concerned (x, y) are given to the distorted circular image memory controller 340. As described above, the distorted circular image memory controller 340 reads out a pixel value of a pixel positioned in the vicinity of the coordinates (x, y) from data of the distorted circular image S inside the distorted circular image memory 440 and gives it to the pixel value determining unit 330.
The distorted circular image S is an image constituted with an aggregate of many pixels arranged at a position indicated by coordinates (x, y) on the twodimensional XY orthogonal coordinate system. In reality, it is constituted with digital data which defines individually inherent pixel values at positions of many lattice points arrayed vertically and transversely at a predetermined pitch. Therefore, a position of the corresponding coordinates (x, y) calculated by the corresponding coordinate calculating unit 200 is usually positioned between a plurality of lattice points. For example, where the distorted circular image S is constituted with digital data which defines pixel values of many lattice points arrayed vertically and transversely at one pitch, any of the lattice points will be an integer value in terms of the coordinate value thereof. Therefore, if values of the corresponding coordinates x, y calculated by the corresponding coordinate calculating unit 200 are values including a decimal fraction (this will be expected in most cases), the position of the corresponding coordinates (x, y) is positioned between a plurality of lattice points. It is, therefore, impossible to determine only one corresponding pixel value.
Thus, in reality, when the pixel value determining unit 330 determines a pixel value of a target pixel on the planar regular image T arranged at a position indicated by coordinates (u, v), it is necessary to read out pixel values of a plurality of reference pixels on the distorted circular image S arranged in the vicinity of a position indicated by the corresponding coordinates (x, y), thereby performing interpolation operation for pixel values of the plurality of reference pixels. Since many methods to perform this interpolation operation are known, such as the bilinear interpolation method and the bicubic/spline interpolation method, a detailed description thereof will be omitted here. As a matter of course, there may be also adopted a method in which in place of performing the interpolation, a pixel value of the closest pixel at a position indicated by the corresponding coordinates (x, y) is determined as a pixel value of the target pixel, as it is.
Therefore, a pixel value of a target pixel determined by the pixel value determining unit 330 is given to the planar regular image memory controller 320. On the other hand, the thus produced coordinates (u, v) are given from the uv coordinate value producing unit 310 to the planar regular image memory controller 320. Then, the planar regular image memory controller 320 performs processing for writing a pixel value determined by the pixel value determining unit 330 as the pixel value of the target pixel positioned at coordinates (u, v) inside the planar regular image memory 420.
A description has been so far given for processing in which a pixel value is determined for one target pixel and written. The uv coordinate value producing unit 310 sequentially produces coordinates (u, v) indicating all pixels on a pixel array constituting the planar regular image T. Thereby, individual pixels are determined for their individual pixel values and written into the planar regular image memory 420.
Individual operation units of the corresponding coordinate calculating unit 200 are not necessarily subjected to recalculation every time the uv coordinate value producing unit 310 produces new coordinates (u, v). For example, a common coefficient G calculated by the common coefficient operation unit 210 is a value which is not dependent on u. Therefore, where only a coordinate value u of the coordinates (u, v) given from the uv coordinate value producing unit 310 has been changed, no recalculation is needed. As a result, where pixel array data is operated in which the U axis is disposed in a line direction and the V axis is disposed in a column direction, an operation newly executed for each line will suffice. Of operations performed by the xy coordinate value calculating unit 240, a new operation for each line will suffice in terms of values of “vB” and “vE” as well. As a matter of course, the rotational coefficient operation unit 230 is not required to perform a new operation unless there is some change in the visual line vector n or the planar inclination angle φ.
In the embodiment shown in
As a matter of course, in the fisheye lens monitoring system, a part functioning as an image converter, that is, a part made up of the parameter input unit 100, the corresponding coordinate calculating unit 200, the planar regular image forming unit 300, the planar regular image memory 420 and the distorted circular image memory 440 may be constituted by installing dedicated programs into a generalpurpose computer. Further, it is also acceptable that the parameter input unit 100 and the rotational coefficient operation unit 230 shown in
In the examples described above, a center point of the thus obtained planar regular image is in alignment with an origin G of the UV coordinate system. However, a planar regular image is not necessarily an image cut out at the center of the origin G. Further, the image converter of the present invention is a device having a function of converting a part of a distorted circular image photographed by use of a fisheye lens into a planar regular image. However, an image to be converted by the device is not limited only to an image photographed by use of a fisheye lens. Any image, for example, an image photographed by use of a convex mirror to which a semisphere projection model similar to that photographed by use of a fisheye lens is applied can be subjected to image conversion by using the image converter of the present invention.
All the embodiments described above are examples in which a fisheye lens based on the orthogonal projection method is used. Correspondence relationship equations used in image conversion are equations based on the distorted circular image S photographed by use of a fisheye lens according to the orthogonal projection method. However, commercially available fisheye lenses are in reality not necessarily lenses based on the orthogonal projection method. There are actually known various projection methods for fisheye lenses such as the equidistance projection method, stereographic projection method and equisolid angle projection method. Thus, fisheye lenses based on these various projection methods are used depending on applications. Herein, a description will be given for a method for applying the present invention to a nonorthogonally projected image photographed by use of a fisheye lens based on a nonorthogonal projection method.
As described above, the optical characteristics of a fisheye lens based on the orthogonal projection method may be explained with reference to a model shown in the
In this case, a relationship between the zenithal angle β of the incident point H (x, y, z) and a distance r from an origin O of a reaching point S (x, y) at which the incident light L2, which has passed through the incident point H (x, y, z), reached the XY plane is expressed by the equation of r=f·sin β in the case of an orthogonally projected image photographed by use of a fisheye lens based on the orthogonal projection method. In this case, f is a constant inherent to the fisheye lens. On the other hand, for example, in the case of an equidistantly projected image photographed by use of a fisheye lens based on the equidistance projection method, the relationship between these will be expressed by the equation of r=f·β.
As described above, the orthogonally projected image and the equidistantly projected image are common in that these are both distorted circular images. However, since there is a difference in the distorted state between these, transformation equations used for converting these into a planar regular image are also different accordingly. Thus, correspondence relationship equations for an orthogonally projected image shown in
However, coordinates on the orthogonally projected image and coordinates on the nonorthogonally projected image can be converted to each other by using predetermined coordinate conversion equations. Therefore, in carrying out the present invention, where the distorted circular image S stored in the distorted circular image memory 440 is a nonorthogonally projected image photographed by use of a fisheye lens based on the nonorthogonal projection method, correspondence relationship equations for nonorthogonally projected images may be used which are obtained by using coordinate conversion equations between coordinates on the orthogonally projected image and coordinates on the nonorthogonally projected image to correct correspondence relationship equations for orthogonally projected images. Hereinafter, a description will be given for this method in a case where an equidistantly projected image is used as a nonorthogonally projected image.
As shown in
xa=xb(R/r)sin(πr/2R) Equation (19)
ya=yb(R/r)sin(πr/2R) Equation (20)
wherein the condition is r=√{square root over ( )}(xb^{2}+yb^{2}).
On the other hand, if coordinates on any give one point on the orthogonally projected image are given as (xa, ya) and coordinates corresponding thereto on a specific one point on the equidistantly projected image are given as (xb, yb), the following equations are satisfied between these as shown below in
xb=xa(2R/πr)sin^{−1}(r/R) Equation (21)
yb=ya(2R/πr)sin^{−1}(r/R) Equation (22)
wherein the condition is r=√{square root over ( )}(xa^{2}+ya^{2}).
In conclusion, the above equations (19), (20) are equations for converting the coordinates (xb, yb) on the equidistantly projected image into the coordinates (xa, ya) on the orthogonally projected image (hereinafter, referred to as a first coordinate conversion equation). The above equations (21), (22) are equations for converting the coordinates (xa, ya) on the orthogonally projected image into the coordinates (xb, yb) on the equidistantly projected image (hereinafter, referred to as a second coordinate conversion equation).
Thus, where an image stored in a distorted circular image memory 440 shown in
Further, attention is needed also where the parameter input unit 100 receives such input instructions that one point on a distorted circular image S is designated as a cutout center point P (x0, y0) and inputs a position of the cutout center point P (x0, y0) as a parameter indicating a visual line vector n. As shown in the embodiments described above, where the distorted circular image S stored in the distorted circular image memory 440 is an orthogonally projected image photographed by use of a fisheye lens based on the orthogonal projection method, as in the model of
However, where the distorted circular image S stored in the distorted circular image memory 440 is a nonorthogonally projected image photographed by use of a fisheye lens based on the nonorthogonal projection method, the corresponding point Q on the sphere is not positioned immediately above the cutout center point P. Thus, in this case, it is necessary that coordinate conversion equations between coordinates on the orthogonally projected image and coordinates on the nonorthogonally projected image are used to correct coordinates of the cutout center point P, define an intersecting point Q between a straight line passing through a point after conversion and parallel to the Z axis and the virtual sphere H, thereby giving a vector which starts from an origin O to pass through the intersecting point Q as a visual line vector n.
Where the distorted circular image S is an equidistantly projected image, coordinates on the cutout center point P designated as one point on the distorted circular image S correspond to coordinates (xb, yb) on the equidistantly projected image. Therefore, the abovedescribed first coordinate conversion equation may be used to convert these into coordinates (xa, ya) on the orthogonally projected image, defining an intersecting point Q between a straight line passing through a point indicated by coordinates (xa, ya) after conversion and parallel to the Z axis and the virtual sphere H, thereby giving a vector which starts from an origin O to pass through the intersecting point Q as a visual line vector n.
Use of the above method makes it possible to apply, in principle, the conversion processing similar to that described above in the embodiments, even where the image stored in the distorted circular image memory 440 is an equidistantly projected image photographed by use of a fisheye lens based on the equidistance projection method. As a matter of course, this method shall not be limited to an application to equidistantly projected images but also widely applicable to nonorthogonally projected images in general.