Method and device for processing an image in order to construct from a source image a target image with charge of perspective

• US 5,586,231 A
• Filed: 12/20/1994
• Issued: 12/17/1996
• Est. Priority Date: 12/29/1993
• Status: Expired due to Fees

• Current Assignee: SiCrystal GmbH (ROHM Co., Ltd.)
• Original Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Inventors: Lelong, Pierre, Florent, Raoul
• Primary Examiner(s): Herndon, Heather R.
• Assistant Examiner(s): Vo, Cliff N.
• Application Number:
  US08/359,364
• Time in Patent Office: 728 Days
• Field of Search: 395/119, 395/125, 395/127, 348/580, 348/581, 348/583, 348/584
• US Class Current: 345/472
• CPC Class Codes: G06T 15/20 Perspective computation

Abstract

A method of processing a digital image in order to construct a calculated image, termed the target image (I*), which represents a so-called source image (I) reconstructed with modifications to the tilting angle, panning angle and scale factor in relation to this source image. This method includes determining a common view point (P), for the source image (I) and the target image (I*), as the origin of an orthonormal reference frame; and calculating, for every pixel (M*) with address (X*, Y*) in the target image (I*), spherical coordinates (α, β) by a linear function G^-1 such that (α, β)=G^-1 (X*Y*), the coefficients being functions of the modified tilting and panning angles and scale factor. Finally, the method includes reading a pair (X, Y) which is stored in a table and which constitutes the address of a point (M), in the source image (I), corresponding to the pixel (M*) in the target image (I*), by the matching, to the resolution of the table, the spherical coordinates (α, β) with one of the addresses (α_o, β_o) of the table.

Claims

1. A method of processing images for constructing a digitized target image from a digitized source image having a fixed frame line, and including pixels each having a digital signal value, said source and target images having substantially common view points, said method comprising the steps:

(a) determining a fixed set of orthonormal reference axes originating at said common view point;

(b) generating a fictive spherical surface centered at the common view point, provided with pixels at a predetermined resolution;

(c) generating an address for each pixel on said spherical surface, associated with a couple of spherical coordinates, constituted of orientation angles, with respect to said set of orthonormal reference axes, of a fictive straight light ray passing through said common view point and through said pixel of the spherical surface;

(d) calculating, for each spherical coordinates couple, a corresponding address of a point situated at the intersection of the source image and of the corresponding fictive straight light ray, by using a direct perspective trigonometrical transform;

(e) storing the address of said corresponding point of the source image at the address of said pixel of the fictive spherical surface associated with said spherical coordinates couple;

(f) generating an address for each pixel of the target image;

(g) calculating, on the basis of the address of a pixel, called the initial pixel, in the target image, approximate orientation angles, with respect to said fixed set of orthonormal reference axes, of an actual straight light ray passing through said initial pixel and said common view point, by using an inverse perspective linear transform;

(h) determining an address of a point in the source image, by matching at best the approximate orientation angles related to said initial pixel with a stored spherical coordinates couple;

(I) calculating a function characteristic of the digital signal value at said point in said source image and assigning said value to said initial pixel in said target image; and

(j) repeating steps (g) to (I) for each pixel of the target image.

2. A method as claimed in claim 1, wherein said method further comprises the calibration steps:

storing parameters of the source and target images, respectively, including a scale factor, and orientation angles constituted by an azimuth angle and a sight angle defined in said fixed set of orthonormal reference axes, and dimensions of the frame of the target image; and

constructing respective real and virtual cameral models for providing said source and target images with said fixed view point which is substantially common with the origin of said fixed set of orthonormal reference axes, and with said respective parameters.

3. A method as claimed in claim 2, wherein said step of determining the fixed set of reference axes comprises defining a horizontal plane and a vertical axis;

said step of calculating approximate angles of the actual straight light ray passing through the initial pixel and through the common view point comprises;

respectively defining a target image local set of orthonormal reference axes, and a source image local set of orthonormal reference axes,

calculating the address of the initial pixel, as a couple of coordinates in said target image local set of reference axes, and

calculating said couple of approximate angles by using a couple of linear relations constituting said inverse linear transform which, respectively, makes to coordinates of the initial pixel linearly correspond to the orientation angles of said actual straight light ray defined, respectively, in the horizontal plane and in the vertical plane of the fixed set of reference axes, said linear relations having the parameters and the framing of said target image as coefficients, and said step of calculating the address of a corresponding point in the source image comprises;

calculating said address in the source image by using a couple of trigonometrical relations constituting said direct trigonometrical transform which transforms, respectively, the spherical coordinates in the spherical surface into a couple of coordinates in the source image local set of reference axes, said trigonometrical relations having the parameters of said source image as coefficients, and

defining said couple of coordinates in said source image local set of reference axes as the address of the corresponding point in the source image.

4. A method as claimed in claim 3, wherein the step of calculating said couple of approximate orientation angles comprises utilizing, as inverse linear transform, the following couple of linear relations joining, respectively, an orientation angle defined in said horizontal plane to one of the initial pixel coordinates and an orientation angle defined in a vertical plane to another of the initial pixel coordinates:

space="preserve" listing-type="equation">α

=(1/D.sub.x *) X*+Θ

*

space="preserve" listing-type="equation">β

=(1/D.sub.y *) Y*+φ

*

where X* and Y* are the coordinates of the initial pixel in the local set of reference axes of the target image, Θ

* is the panning angle, φ

* is the tilting angle, in the fixed set of reference axes, and D_x * and D_y * are, respectively, the resolutions of the target image parallel to said local reference axes of said target image, which depend on the scale factor, and on the frame dimensions of the target image; and

the step of calculating said couple of coordinates of the corresponding point in the source image comprises utilizing, as direct trigonometrical transform, the following relations joining, respectively, said coordinates to said spherical coordinates in the spherical surface;

##EQU3## and Y=d.tan B/cos A with

B=arcsin(cosβ

_o. cos(α

_o -θ

)sinφ

+sinβ

_o cosφ

).

5. A method of processing images for constructing a digitized target image from n adjacent digitized images having a fixed frame line and referred to as source images, the pixels of the source images having digital signal values, and said source and target images having substantially common view points, comprising:

(a'"'"') determining a fixed set of orthonormal reference axes originated at said common view point;

(b'"'"') generating a fictive spherical surface centered at the common view point, provided with pixels at a predetermined resolution;

(c'"'"') generating an address for each pixel on said spherical surface, associated with a couple of spherical coordinates, constituted of orientation angles, with respect to said set of orthonormal reference axes, of a fictive straight light ray passing through said common view point and through said pixel of the spherical surface;

(d'"'"') calculating, for each spherical coordinates couple, a corresponding address of a point situated at the intersection of one of the source images and of the corresponding fictive straight light ray, by using a direct perspective trigonometrical transform;

(e'"'"') storing the address of a corresponding point of said particular source image, associated with an index indicating the particular source image out of the n source images that the fictive straight light ray intersects, at the address of said pixel of the fictive spherical surface associated with said spherical coordinates couples;

(f'"'"') generating an address for each pixel of the target image;

(g'"'"') calculating, on the basis of the address of a pixel called initial pixel in the target image, approximate orientation angles, with respect to said fixed set of orthonormal reference axes, of an actual straight light ray passing through said initial pixel and said common view point, by using an inverse perspective linear transform;

(h'"'"') determining an address of a point in a source image, and an index indicating the particular source image where said point lies, by matching at best the approximate orientation angles related to said initial pixel with a store spherical coordinates couple;

(I'"'"') calculating a function characteristic of the digital signal value at said point in said particular source image and assigning said value to said initial pixel in said target image; and

(j'"'"') repeating steps (g'"'"') to (j'"'"') for each pixel of the target image.

6. A method as claimed in claim 5, comprising calibration steps of:

storing parameters of the n source images including scale factors and orientation angles constituted by azimuth angles and angles of sight, said angles being defined in said fixed set of orthonormal reference axes, and dimensions of the frame of the target image;

storing parameters of the target image including scale factor and orientation angles constituted by an azimuth angle and a sight angle, said angles being defined in said fixed set of orthonormal reference axes; and

constructing respective real and virtual camera models for providing said source and target images with said respective parameters and with said fixed view point which is substantially common with the origin of said fixed set of orthonormal reference axes.

7. A method as claimed in claim 6, for constructing the target image representing sub-portions of the fixed frame constituted by the n adjacent source images, comprising varying the target image parameters and the dimension of the frame of the virtual camera, the source image parameters of the real camera being fixed.

8. A method a claimed in claim 7, wherein the step of determining the fixed set of reference axes comprises defining a horizontal plane and a vertical axis;

the step of calculating approximate angles of the actual straight light ray passing through the initial pixel and through the common view point comprises;

respectively defining a target image local set of orthonormal reference axes, and local sets of orthonormal reference axes for the source image,

calculating the address of the initial pixel, as a couple of coordinates in said target image local set of reference axes,

calculating said couple of approximate angles by using a couple of linear relations constituting said inverse linear transform, which makes, respectively, the coordinates of the initial pixel in the local set of reference axes, respectively, linearly correspond to the orientation angles of said actual straight light ray, respectively defined in the horizontal plane and in the vertical plane of the fixe set of reverence axes, said linear relations having the variable parameters and the framing of said target image as coefficients, an the step of calculating the address of a corresponding point in the source image comprises;

calculating said address, in the local set of reference axes of the source image, by using a couple of trigonometrical relations constituting said direct trigonometrical transform which, respectively, transforms the spherical coordinates in the spherical surface into a couple of coordinates in the source image local set of reference axes, said trigonometrical relations having the fixed parameters of the particular source image as coefficients, and

defining said couple of coordinates in said particular source image local set of reference axes as the address of the corresponding point in the particular source image.

9. A method as claimed in claim 8, wherein the step of calculating said couple of approximate orientation angles comprises utilizing, as inverse liner transform, the following couple of linear relations joining, respectively, an orientation angle defined in said horizontal plane to one of the initial pixel coordinates, and an orientation angle defined in a vertical plane to another of the initial pixel coordinates:

space="preserve" listing-type="equation">α

=(1/D.sub.x *) X*+Θ

*

space="preserve" listing-type="equation">β

=(1/D.sub.y *) Y*+φ

*

where X* and Y* are the coordinates of the initial pixel in the local set of reference axes of the target image, Θ

* is the panning angle, φ

* is the tilting angle, in the fixed set of reference axes, and D_x * and D_y * are, respectively, the resolutions of the target image parallel to said local reference axes of said target image, which depend on the scale factor, and on the frame dimensions of the target image; and

the step of calculating said couple of coordinates of the corresponding point in the source image comprises utilizing, as direct trigonometrical transform, the following relations joining, respectively, said coordinates to said spherical coordinates in the spherical surface;

X=d.tan A with ##EQU4## and Y=d.tan B/cos A with

B=arcsin(cosβ

_o. cos(α

_o -θ

)sinφ

+sinβ

_o cosφ

).

10. An image processing device comprising:

a system of n fixed real cameras, n being an integer greater than 0, said real cameras being arranged such that their individual fields of view merge so as to form a single wide angle field of view for observation of a panoramic scene, said real cameras providing adjacent images referred to as source images;

an image construction system simulating a mobile virtual camera continuously scanning the panoramic scene so as to form a sub-image referred to as a target image corresponding to a selected section of a wide-angle field of view, and constructed from said source images;

means for digitizing said source and target images such that each pixel corresponds to a digital signal value;

calibration means for determining a substantially common view point to said source and target images, and a fixed set of orthonormal reference axes originated at said common view point;

an address generator for generating, pixel by pixel, respective addresses for the pixels of the target image;

an address computer including;

tabulation means for storing addresses of pixels provided on a fictive spherical surface, centered at the common view point, said pixels being disposed at a predetermined resolution, each of said addresses being associated with a couple of spherical coordinates constituted by orientation angles, defined with respect to said fixed set of orthonormal reference axes, of a fictive straight light ray passing through said common view point and through said pixel at the corresponding address of the fictive spherical surface, and each couple of spherical coordinates being associated with an index indicating a particular source image out of the n source images that the fictive straight light ray intersects, and being associated with an address of a corresponding point of said particular source image;

calculation means for calculating, on the basis of the address of a pixel called initial pixel in the target image, approximate orientation angles, with respect to said fixed set of orthonormal reference axes, of an actual straight light ray passing through said initial pixel and said common view point, by using an inverse perspective linear transform system;

addressing means for determining, from the tabulation means, an address of a point in a source image, and an index indicating the particular source image where said point lies, by matching at best the approximate orientation angles related to said initial pixel with a stored spherical coordinates couple;

calculation means for calculating a function characteristic of the digital signal value at said point in said particular source image; and

means for assigning said value to said initial pixel in said target image.

11. A device as claimed in claim 10, wherein the calibration means further comprises:

virtual camera storage means for storing parameters relating to said virtual camera for supplying said address computer with scale factor and with orientation angles of an optical axis of the virtual camera, said orientation angle being constituted by the azimuth and the sight angle defined in said fixed set of orthonormal reference axes;

frame dimensions storage means for storing the frame dimensions of the target image, for supplying the address computer with the resolution of said target image; and

real camera storage means for storing parameters relating to said real cameras for supplying said address computer with scale factor and with orientation angles of an optical axis of each real camera, said orientation angles being constituted by azimuth angles and angles of sight, defined in said fixed set of orthonormal reference axes.

12. A device as claimed in claim 11, wherein the address computer comprises:

first construction means for constructing a model of the virtual camera with the parameters stored in the virtual camera storage means and in the frame dimensions storage means; and

second construction means for constructing models of the real cameras with the parameters stored in the real camera storage means.

13. A device as claimed in claim 10, wherein the calculating means of the address computer comprises:

two numerical ramp calculation modules for computing a geometrical transform, referred to as an inverse linear transform, one of said modules for calculating the first approximate spherical coordinate by a first linear relation having, as a variable, the first coordinate of the starting pixel, in a local set of reference axes of the target image, and the other of said modules for calculating the second approximate spherical coordinate by a second linear relation having, as a variable, the second coordinate o the pixel in said local frame, the coefficients of these linear relations being functions of the azimuth and sight angles, of the scale factor and of the framing of the target image.

14. A device as claimed in claim 10, wherein the address computer further comprises:

computing means for generating a fictive spherical surface centered at the common view point, provided with pixels at a predetermined resolution, for generating an address for each pixel on said fictive spherical surface, associated with a coupled of spherical coordinates, constituted of orientation angles, with respect to said fixed set of orthonormal reference axes, of a fictive straight light ray passing through said common view point and through said pixel of the fictive spherical surface;

computing means, referred to as direct trigonometrical transform, for calculating, for each spherical coordinates couple, a corresponding address of a point situated at the intersection of one of the source images and of the corresponding fictive straight light ray, and an index indicating the particular source image where said point lies, taking into account the models of the real cameras provided by the second construction means, the parameters constituted by the azimuth angles, the angles of sight, and the scale factors of the respective real cameras corresponding to said particular source image; and

addressing means for storing the address of the corresponding point of said particular source image, associated with an index indicating the particular source image out of the n source images that the fictive spherical surface associated with said spherical coordinates couples in the tabulation means.

15. A device as claimed in claim 10, wherein the means for determining the function characteristic of the digital value comprises:

an interpolator for computing a most probable value of the digital signal at the address found by the address computer in said particular source image; and

storage means for assigning said computed digital signal value corresponding to the address found in said particular source image to the initial pixel in the target image at the address furnished by said address generator.

16. A device as claimed in claim 10, further comprising a display system with a screen for displaying the target image in real time on said screen.

17. A device as claimed in claim 10, further comprising a recording system for recording the target image.

18. A device as claimed in claim 12, wherein the system for constructing the target image also comprises:

an interface module for enabling a user to define or to vary, in the virtual camera storage means, said parameters including the scale factor and the orientation of the optical axes, and, in the frame dimensions storage means, said framing for defining the target image as a sub-portion of the fixed frame constituted by the n adjacent source images.

19. A device as claimed in claim 18, wherein the interface module is controlled automatically.

20. A device as claimed in claim 18, wherein the interface module is controlled manually.