Method and device for processing an image in order to construct a target image from a plurality of contiguous source images

• US 5,594,845 A
• Filed: 12/20/1994
• Issued: 01/14/1997
• Est. Priority Date: 12/29/1993
• Status: Expired due to Fees

• Current Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Original Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Inventors: Lelong, Pierre, Florent, Raoul
• Primary Examiner(s): Herndon, Heather R.
• Assistant Examiner(s): BUCHEL JR, RUDOLPH JOHN
• Application Number:
  US08/359,378
• Time in Patent Office: 756 Days
• Field of Search: 395/127, 345/9
• US Class Current: 345/427
• CPC Class Codes: G06T 15/20 Perspective computation

Abstract

Method of processing a digital image in order to construct a calculated image, termed the target image (I*), which represents a so-called source image (I) reconstructed with modifications to the tilting angle, panning angle and scale factor in relation to this source image, including the determination of a common view point (P) for the source image (I) and the target image (I*), and of an orthonormal reference frame; the definition of an image (H*) termed the projection of the target image (I*), with geometrical center (Ω_H *) aligned with the optical axis (PZ*) of the target image, with zero tilting angle, with the same panning angle (Θ*) and the same framing (2V_x*, 2V_y *) as the target image (I*) so that every pixel (M*) with address (X*, Y*) in the target image (I*) has a counterpart under projection (M_H *) with the same address (X*, Y*) in the projection image (H*); the calculation by a trigonometric function (G^-1), for every pixel (M*) with address (X*, Y*) in the target image, of the spherical coordinates (α, β) defining, in the reference frame, the ray linking the origin (P) to the corresponding pixel (M_H *) of (H*).

Claims

1. A method of processing images for constructing a digitized target image (I*) from a digitized source image (I) having a fixed frame line, and whose pixels have a digital signal value, said source and target images having a substantially common view point (P) and respective optical axes (PZ, PZ*), comprising:

(A) calibration steps of;

determining a fixed set of orthonormal reference axes (PXa, PYa, PZa) originated at said common view point (P), defining a horizontal plane (PXa, PZa) and a vertical axis (PYa); and

storing addresses (X, Y) of points (M) situated in the source image (I), associated with respective corresponding couples of spherical coordinates (α

₀, β

₀), constituted of orientation angle values with respect to said set of orthonormal reference axes;

(B) initialization steps of;

storing parameters of the target image (I*), including a scale factor (d*), dimensions of the frame (2V_x *, 2V_y *), and orientation angles of the optical axis constituted by a panning angle (Θ

*) and a tilting angle (ρ

*) defined in said fixed set of orthonormal reference axes;

(C) calculation steps of;

generating an address for each pixel of the target image (I*);

generating, from said target image (I*), an image called transformed target image (H*) by using an inverse perspective transform (G^-1) consisting in two successive sub-transforms;

a rotation sub-transform applied to the target image (I*) operating with a rotation angle (ρ

*) having the opposite value to the target image tilting angle (ρ

*), for generating a first vertical sub-image (R*);

a vertical translation sub-transform applied to said first vertical sub-image (R*), operating with a vertical translation vector (Y₀) for generating a second vertical sub-image (H*) constituting said transformed target image (H*), having the same frame dimensions (2V_x *, 2V_y *) and the same azimuth angle as the target image (I*), and a null sight angle, and having a geometrical center (Ω

_H *) situated on the optical axis (PZ*) of the target image (I*);

localizing, on the basis of the address (X*, Y*) of a pixel called initial pixel (M*) in the target image (I*), a pixel (M_H *) called transformed initial pixel which corresponds to said initial pixel (M*) by said inverse perspective transform (G^-1) having the same address in the transformed target image (H*), and calculating orientation angles (α

, β

) called approximate angles, with respect to said fixed set of orthonormal reference axes, of a straight line (PM_H *) passing through said common view point (P) and through said transformed initial pixel (M_H *);

determining an address (X, Y) of a point (M) in the source image (I), by matching, at best, said approximate orientation angles (α

, β

) with a stored spherical coordinates couple (α

₀, β

₀); and

calculating a function characteristic of the digital signal value (Q) at said point (M) in said source image (I) and assigning said value to said initial pixel (M*) in said target image (I*);

(D) repeating calculation steps (C) for each pixel of the target image (I*).

2. A method as claimed in claim 1, further comprising, for carrying out the calibration steps (A):

storing parameters of the source image (I), including a scale factor (d) and orientation angles of the optical axis constituted by an azimuth angle (Θ

) and a sight angle (ρ

) defined in said fixed set of orthonormal reference axes;

constructing respective real and virtual camera models for providing said source and target images with said respective parameters and with said fixed view point (P) which is substantially common with the origin of said fixed set of orthonormal reference axes; and

storing addresses of points (M) of the source image (I), including the sub-steps of;

generating a fictive spherical surface (K*) centered at the common view point (P), provided with pixels (K₀) at a predetermined resolution;

generating an address for each pixel (K₀) on said spherical surface, associated with a couple of spherical coordinates (α

₀, β

₀), constituted or orientation angles, with respect to said set of orthonormal reference axes, of a fictive straight light ray (PW) passing through said common view point (P) and through said pixel (K₀) of the spherical surface;

calculating, for each spherical coordinates couple (α

₀, β

₀), a corresponding address (X, Y) of a point (M) situated at the intersection of the source image (I) and of the corresponding fictive straight light ray (PW), by using a direct perspective trigonometrical transform (F); and

storing the address (X, Y) of said corresponding point (M) of the source image (I) at the address of said pixel (K₀) of the fictive spherical surface associated with said spherical coordinates couple (α

₀, β

₀).

3. A method as claimed in claim 2, comprising:

in the step of calculating approximate angles (α

, β

) of the straight line passing through the transformed initial pixel (M_H *) and through the common view point (P);

respective definitions of a target image (I*) and of a transformed target image local sets of orthonormal reference axes, and of a source image local set of orthonormal reference axes;

the calculation of the address of the initial pixel (M*), and of the transformed initial pixel (M_H *) as a couple of coordinates (X, Y) in said target image (I*) in said transformed target image local sets of reference axes, respectively;

the calculation of said couple of approximate angles (α

, β

) by using a couple of relations constituting said inverse perspective transform (G^-1), which respectively makes the coordinates of the transformed initial pixel (M_H *) in the local set of reference axes correspond to the orientation angles (α

, β

) of said straight line, said orientation angles being respectively defined in the horizontal plane and in the vertical plane of the fixed set of reference axes, and said relations having the parameters (d*, Θ

*, ρ

*) of the target image (I*) as coefficients;

and in the step of calculating the address of a corresponding point (M) in the source image (I);

the calculation of said address (X, Y) in the source image (I) by using a couple of trigonometrical relations constituting said direct trigonometrical transform (F) which respectively transforms the spherical coordinates (α

₀, β

₀) in the spherical surface into a couple of coordinates (X, Y) in the source image local set of reference axes, said trigonometrical relations having the parameters (dj, Θ

j, ρ

j) of said source image (I) as coefficients;

the definition of said couple of coordinates (X, Y) in said source image local set of reference axes as the address of the corresponding point (M) in the source image (I).

4. A method as claimed in claim 3, comprising:

in the step of calculation of said couple of approximate orientation angles (α

, β

), the utilization, as inverse perspective transform (G^-1) of the following couple of relations joining, respectively, an orientation angle (α

) defined in said horizontal plane to one of the transformed initial pixel coordinates (X*) and an orientation angle (β

) defined in a vertical plane to another of the transformed initial pixel coordinates (Y*);

space="preserve" listing-type="equation">α

=A*+Θ

*

space="preserve" listing-type="equation">β

=arctan {(Y*-Yo) cos A*)/d*}

where X* and Y* are the coordinates of the initial pixel (M*) in the local sets of reference axes of the target image (I*), where Θ

* is the panning angle, ρ

* is the tilting angle, in the fixed set of reference axes, and where

space="preserve" listing-type="equation">A*=arctan (X*;

d*)

space="preserve" listing-type="equation">Yo=d*×

tan ρ

*2

and in the step of calculation of said couple of coordinates (X, Y) of the corresponding point (M) in the source image (I), the utilization as direct trigonometrical transform (F) of the following relations joining, respectively, said coordinates to said spherical coordinates (α

₀, β

₀) in the fictive spherical surface (K*);

space="preserve" listing-type="equation">X=d×

tan A, with ##EQU3##

space="preserve" listing-type="equation">and Y=d×

tan B/cos A, with

space="preserve" listing-type="equation">B=arcsin(cos β

.sub.o ·

cos (α

.sub.o -Θ

) sin ρ

+sin β

.sub.o cos ρ

).

5. A method of processing images for constructing a digitized target image (I*) from a number n of digitized source images (I1 . . . In) having a fixed frame line, and whose pixels have a digital signal value, said source and target images having a substantially common view point (P) and respective optical axes (PZ1 . . . PZn;

PZ*), comprising;

(A) calibration steps of;

determining a fixed set of orthonormal reference axes (PXa, PYa, PZa) originated at said common view point (P), defining a horizontal plane (PXa, PZa) and a vertical axis (PYa); and

storing addresses (X, Y) of points (M) situated in the source images (I1 . . . In), associated with an index indicating a particular source image out of the n source images where a particular point (M) lies, and associated with respective corresponding couples of spherical coordinates (α

₀, β

₀), constituted of orientation angle values with respect to said set of orthonormal reference axes (P);

(B) initialization steps of;

storing parameters of the target image (I*), including a scale factor (d*), dimensions of the frame (2V_x *, 2V_y *), and orientation angles of the optical axis constituted by a panning angle (Θ

*) and a tilting angle (ρ

*) defined in said fixed set of orthonormal reference axes;

(C) calculation steps of;

generating an address for each pixel of the target image (I*);

generating, from said target image (I*), an image called transformed target image (H*) by using an inverse perspective transform (G^-1) consisting in two successive sub-transforms;

a rotation sub-transform applied to the target image (I*) operating with a rotation angle (-ρ

*) having the opposite value to the target image tilting angle (ρ

*), for generating a first vertical sub-image; and

a vertical translation sub-transform applied to said first vertical sub-image (R*), operating with a vertical translation vector (Y₀) for generating a second vertical sub-image (H*) constituting said transformed target image (H*), having the same frame dimensions (2V_x *, 2V_y *) and the same azimuth angle as the target image (I*), and a null sight angle, and having a geometrical center (Ω

_H *) situated on the optical axis of the target image (I*);

localizing, on the basis of the address (X*, Y*) of a pixel called initial pixel (M*) in the target image (I*), a pixel called transformed initial pixel (M_H *) which corresponds to said initial pixel (M*) by said inverse perspective transform (G^-1) having the same address in the transformed target image (H*), calculating orientation angles called approximate angles (α

, β

), with respect to said fixed set of orthonormal reference axes, of a straight line (PM_H *) passing through said common view point (P) and through said transformed initial pixel (M_H *);

determining an address (X, Y) of a point (M) and an index indicating the particular source image (Ij) where said point lies, by matching, at best, said approximate orientation angles (α

, β

) with a stored spherical coordinates couple (α

₀, β

₀); and

calculating a function characteristic of the digital signal value (Q) at said point (M) in said particular source image (Ij) and assigning said value to said initial pixel (M*) in said target image (I*);

(D) repeating steps (C) for each pixel of the target image (I*).

6. A method as claimed in claim 5, further comprising, for carrying out the calibration steps (A):

storing parameters of the source images (I1 . . . In), including a scale factor (d1 . . . dn) and orientation angles constituted by respective panning angles (Θ

1 . . . Θ

n) and tilting angles (ρ

1 . . . ρ

n) defined in said fixed set of orthonormal reference axes;

constructing respective real and virtual camera models for providing said source and target images with said fixed view point (P) which is substantially common with the origin of said fixed set of orthonormal reference axes, and with said respective parameters; and

storing addresses of points (M) of the source images (I1 . . . In) including sub-steps of;

generating a fictive spherical surface (K*) centered at the common view point (P), provided with pixels (K₀) at a predetermined resolution;

generating an address for each pixel (K₀) on said spherical surface, associated with a couple of spherical coordinates (α

₀, β

₀), constituted of orientation angles, with respect to said set of orthonormal reference axes, of a fictive straight light ray (PW) passing through said common view point (P) and through said pixel (K₀) of the spherical surface;

calculating, for each spherical coordinates couple (β

₀, β

₀), a corresponding address (X, Y) of a point (M) situated at the intersection of one of the source images (I1 . . . In), called particular source image (Ij), and of the corresponding fictive straight light ray (PW), by using a direct perspective trigonometrical transform (F); and

storing the address (X, Y) of said corresponding point (M) of said particular source image (Ij), associated with an index indicating said particular source image (Ij) out of the n source images (I1 . . . In) where said point (M) lies at the address of said pixel (K₀) of the fictive spherical surface associated with said spherical coordinates couple (α

₀, β

₀).

7. A method as claimed in claim 6, further comprising, for carrying out the initialization steps (B):

constructing the target image (I*) representing sub-portions of the fixed frame constituted by the n adjacent source images (I1 . . . In), by varying the target image parameters (d*, Θ

*, ρ

*) and the dimension of the frame (2V_x *, 2V_y *) of the virtual camera, the source image parameters of the real cameras being fixed.

8. A method as claimed in claim 7, comprising:

in the step of calculating approximate angles (α

, β

) of the straight line passing through the transformed initial pixel (M_H *) and through the common view point (P);

respective definitions of target image (I*) and transformed target image local sets of orthonormal reference axes, and of local sets of orthonormal reference axes for the source image (I);

the calculation of the address of the initial pixel (M*) and of the transformed initial pixel (M_H *), as a couple of coordinates (X*, Y*) in said target image and in said transformed target image local sets of reference axes, respectively;

the calculation of said couple of approximate angles (α

, β

) by using a couple of relations constituting said inverse perspective transform (G^-1), which, respectively, makes the coordinates of the transformed initial pixel (M_H *) in the local set of reference axes correspond to the orientation angles (α

, β

) of said straight line, said orientation angles being respectively defined in the horizontal plane and in the vertical plane of the fixed set of reference axes, said relations having the variable parameters (d*, Θ

*, ρ

*) of said target image (I*) as coefficients;

and in the step of calculating the address of a corresponding point (M) in the source image (I);

the calculation of said address (X, Y), in the local set of reference axes of the source image (I), by using a couple of trigonometrical relations constituting said direct trigonometrical transform (F) which respectively transforms the spherical coordinates (α

₀, β

₀) in the spherical surface into a couple of coordinates (X, Y) in the source image local set of reference axes, said trigonometrical relations having the fixed parameters of the particular source image as coefficients;

the definition of said couple of coordinates (X, Y) in said particular source image local set of reference axes as the address of the corresponding point (M) in the particular source image.

9. A method as claimed in claim 8, comprising:

in the step of calculation of said couple of approximate orientation angles (α

, β

), the utilization, as inverse perspective transform (G^-1), of the following couple of relations joining, respectively, an orientation angle (α

) defined in said horizontal plane to one of the transformed initial pixel coordinates (X*) and an orientation angle (β

) defined in a vertical plane to another of the transformed initial pixel coordinates (Y*);

space="preserve" listing-type="equation">α

=A*+Θ

*

space="preserve" listing-type="equation">β

=arctan{(Y*-Yo) cos A*)/d*}

where X* and Y* are the coordinates of the initial pixel (M*) in the local set of reference axes of the target image (I*), where Θ

* is the panning angle, ρ

* is the tilting angle, in the fixed set of reference axes, and where

space="preserve" listing-type="equation">A*=arctan(X*;

d*)

space="preserve" listing-type="equation">Yo=d*×

tan ρ

*

and in the step of calculation of said couple of coordinates (X, Y) of the corresponding point (M) in the source image (I), the utilization as direct trigonometrical transform (F) of the following relations joining, respectively, said coordinates to said spherical coordinates (α

₀, β

₀) in the spherical surface;

space="preserve" listing-type="equation">X=d×

tan A, with ##EQU4##

space="preserve" listing-type="equation">and Y=d×

tan B/cos A, with

space="preserve" listing-type="equation">B=arcsin(cos β

.sub.o ·

cos (α

.sub.o -Θ

) sin ρ

+sin β

.sub.o cos ρ

).

10. Am image processing device comprising:

a system including a number n of fixed real cameras (C1 . . . Cn), n being an integer number≠

0, said cameras being arranged in such a way that their individual fields of view merge so as to form a single wide-angle field of view for observation of a panoramic scene, said real cameras providing adjacent images referred to as source images (I1 . . . In); and

an image construction system simulating a mobile virtual camera (C*) continuously scanning the panoramic scene so as to form an image referred to as target image (I*) corresponding to a selected section of said wide-angle field of view, and constructed from said source images (I1 . . . In);

said systems further comprising;

(A'"'"') calibration means including;

means for digitizing said source and target images, in a manner that each pixel corresponds to a digital signal value;

means for determining a substantially common view point (P) to said source and target images, and for determining a fixed set of orthonormal reference axes (PXa, PYa, PZa) originated at said common view point (P), defining a horizontal plane (PXa, PZa) and a vertical axis (PYa); and

tabulation means for storing addresses (X, Y) of points (M) situated in the source images (I1 . . . In), associated with an index indicating a particular source image (Ij) out of the n source images (I1 . . . In) where a particular point (M) lies, and associated with respective corresponding couples of spherical coordinates (α

₀, β

₀), constituted of orientation angle values with respect to said set of orthonormal reference axes (P);

(B'"'"') initialization means including storing means for storing parameters relating to the virtual camera (C*) for supplying parameters of the target image (I*), including a scale factor (d*), dimensions of the frame (2V_x *, 2V_y *), and orientation angles constituted by a panning angle (Θ

*) and a tilting angle (ρ

*) defined in said fixed set of orthonormal reference axes;

(C'"'"') calculations means comprising an address computer including;

an address generator for generating, pixel by pixel, respective addresses for the pixels of the target image (I*);

an inverse perspective transform (G^-1) system operating two successive sub-transforms including a rotation sub-transform and a vertical translation sub-transform for generating, from said target image (I*), an image called transformed target image (H*) having the same frame dimensions (2V_x *, 2V_y *) and the same azimuth angle as the target image (I*), and a null sight angle, and having a geometrical center (Ω

_H *) situated on the optical axis of the target image (I*), said inverse perspective transform system having means for localizing, on the basis of the address (X*, Y*) of a pixel called initial pixel (M*) in the target image (I*), a pixel called transformed initial pixel (M_H *) which corresponds to said initial pixel (M*) by said inverse perspective transform (G^-1) having the same address in the transformed target image (H*), and said inverse perspective transform system having means for calculating orientation angles called approximate angles (α

, β

), with respect to said fixed set of orthonormal reference axes, of a straight line (PM_H *) passing through said common view point (P) and through said transformed initial pixel (M_H *);

addressing means for determining, from the tabulation means, an address (X, Y) of a point (M) in a source image (I), and an index indicating the particular source image (Ij) where said point (M) lies, by matching, at best, the approximate orientation angles (α

, β

) related to said initial pixel (M*) with a stored spherical coordinates couple (α

₀, β

₀); and

calculation means for calculating a function characteristic of the digital signal value (Q) at said point (M) in said particular source image (Ij), and means for assigning said value to said initial pixel (M*) in said target image (I*).

11. A device as claimed in claim 10, wherein:

the calibration means (A'"'"') further comprises;

storage means for storing parameters relating to said real cameras (C1 . . . Cn) for supplying said address computer with scale factor (d1 . . . dn) and with orientation angles of an optical axis of each real camera (C1 . . . Cn), said orientation angles being constituted by panning angles (Θ

1-Θ

n) and tilting angles (ρ

1-ρ

n), defined in said fixed set of orthonormal reference axes;

tabulation means for storing addresses of pixels (K₀) provided on a fictive spherical surface (K*), centered at the common view point (P), said pixels (K₀) being disposed at a predetermined resolution, each of said addresses being associated with a couple of spherical coordinates constituted by orientation angles (α

₀, β

₀), defined with respect to said fixed set of orthonormal reference axes, of a fictive straight light ray passing through said common view point (P) and through said pixel (K₀) at the corresponding address of the fictive spherical surface (K*), and each couple of spherical coordinates (α

₀, β

₀) being associated with an index indicating a particular source image (Ij) out of the n source images (I1 . . . In) that the fictive straight light ray intersects, and being associated with an address (X, Y) of a corresponding point (M) of said particular source image (Ij).

12. A device as claimed in claim 11, wherein the address computer comprises:

first construction means for constructing a model of the virtual camera with the parameters stored in said virtual camera storage means; and

second construction means for constructing models of the real cameras with the parameters stored in said real camera storage means.

13. A device as claimed in claim 10, wherein the inverse perspective transform system of the address computer comprises:

modules for calculating trigonometrical functions, associated with an adder, a multiplier, and inverter modules, for implementing the calculation of a couple of relations forming said inverse perspective transform (G^-1);

space="preserve" listing-type="equation">α

=A*+Θ

*

space="preserve" listing-type="equation">β

=arctan{(Y*-Yo) cos A*)/d*}

where X* and Y* are the coordinates of the initial pixel (M*) in the local set of reference axes of the target image (I*), where Θ

* is the panning angle, ρ

* is the tilting angle, in the fixed set of reference axes, and where

space="preserve" listing-type="equation">A*=arctan (X*;

d*)

space="preserve" listing-type="equation">Yo=d*×

tan ρ

*.

14. A device as claimed in claim 12, wherein the address computer further comprises:

computing means for generating a fictive spherical surface (K*) centered at the common view point (P), provided with pixels (K₀) at a predetermined resolution, for generating an address for each pixel (K₀) on said fictive spherical surface, associated with a couple of spherical coordinates (α

₀, β

₀), constituted of orientation angles, with respect to said fixed set of orthonormal reference axes, of a fictive straight light ray (PW) passing through said common view point (P) and trough said pixel (K₀) of the fictive spherical surface (K*);

computing means, referred to as direct trigonometrical transform (F), for calculating, for each spherical coordinates couple (α

₀, β

₀), a corresponding address (X, Y) of a point (M) situated at the intersection of one of the source images (I1 . . . In) and of the corresponding fictive straight light ray (PW), and an index indicating the particular source image (Ij) where said point (M) lies, taking into account the models of the real cameras provided by the second construction means, the parameters constituted by the panning angles (Θ

1-Θ

n), the tilting angles (ρ

1-ρ

n), and the scale factors (d1-dn) of the respective real cameras (C1-Cn) corresponding to said particular source image (Ij); and

addressing means for storing the address (X, Y) of the corresponding point (M) of said particular source image (Ij), associated with an index indicating the particular source image (Ij) out of the n source images (I1 . . . In) that the fictive straight light ray intersects, at the address of said pixel (K₀) of the fictive spherical surface associated with said spherical coordinates couples (α

₀, β

₀) in the tabulation means.

15. A device as claimed in claim 10, wherein the means for determining the function characteristic of the digital value (Q) comprises:

an interpolator for computing a most probable value (Q) of the digital signal at the address (X, Y) found by the address computer in said particular source image (Ij); and

storage means for assigning said computed digital signal value (Q) corresponding to the address (X, Y) found in said particular source image (Ij) to the initial pixel (M*) in the target image (I*) at the address (X*, Y*) furnished by said address generator.

16. A device as claimed in claim 10, further comprising a display system with a screen for displaying the target image (I*) in real time on said screen.

17. A device as claimed in claim 10, further comprising a recording system for recording the target image (I*).

18. A device as claimed in claim 12, wherein the system for constructing the target image (I*) also comprises:

an interface module for enabling a user to define or to vary, in the storage means of the virtual camera (C*), said parameters including the scale factor (d8) and the orientation of the optical axes (Θ

*, ρ

*), and in the storage means of said virtual camera, said framing (2V_x *, 2V_y *), for defining the target image (I*) as a sub-portion of the fixed frame constituted by the n adjacent source images (I1 . . . In).

19. a device as claimed in claim 18, wherein the user interface is controlled automatically or manually.