Image processing method and device for constructing an image from adjacent images

• US 5,444,478 A
• Filed: 12/28/1993
• Issued: 08/22/1995
• Est. Priority Date: 12/29/1992
• Status: Expired due to Fees

• Current Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Original Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Inventors: Lelong, Pierre, Dalm, Govert, Klijn, Jan
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): AU, AMELIA MEGAN
• Time in patent office: 602 Days
• Field of Search: 348/36, 348/39, 348/580, 348/383, 348/37, 348/38, 382/41
• US Class Current: 348/39
• CPC Class Codes: H04N 5/23238 Control of image capture or reproduction to achieve a very large field of view, e.g. panorama panoramic or widescreen photography G03B27/00
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method of processing images for constructing a target image (Io) from adjacent images having a fixed frame line and referred to as source images (I1, . . . , Ii, Ij, . . . , In), the source and target images having substantially common view points. This method includes the steps of: digitizing the images, determining, for one of the pixels of the target image (Io), the address (Aq) of a corresponding point in one of all source images (Ij), determining the luminance value (F) at this corresponding point, assigning the luminance value (F) of this corresponding pixel to the initial pixel in the target image (Io), and repeating these steps for each pixel of the target image (Io). A device for performing this method includes a system of n fixed real cameras (C1, . . . , Cn) which provide n adjacent source images (I1, . . . , In) covering a wide-angle field of view and which have common view points (P), and an image reconstruction system (100) simulating a mobile camera referred to as target image (Co) for providing a sub-image referred to as target image (Io) of the wide-angle field of view, and constructed on the basis of source images having the same view point (P).

Claims

1. A method of processing images for constructing a target image (Io) from adjacent images having a fixed frame line and referred to as source images (I1, . . . , Ii, Ij, . . . , In), said source and target images having substantially common view points, characterized in that the method comprises the steps of:

(a) digitizing said source and target images;

(b) determining a substantially common view point (P) to said source and target images, and a fixed orthonormal landmark (Px, Py, Pz) originated at said common view point (P);

(c) generating, pixel by pixel, an address, for each pixel, in the target image (Io), so as to entirely cover said target image (Io);

(d) calculating, on the basis of an address (Ao) of a current pixel referred to as an initial pixel (m'"'"') in the target image (Io), an orientation, in said fixed landmark, of a straight light ray (PM) passing through said initial pixel (m'"'"') and through said common view point;

(e) selecting a source image (Ij) traversed by said straight light ray (PM);

(f) calculating, from said orientation of the straight light ray (PM), an address (Aq) of a distortion corrected point (m), in said selected source image (Ij), said distortion corrected point (m) corresponding to said initial pixel (m'"'"');

(g) calculating a luminance value (F) at said distortion correct point m in said source image Ij;

(h) assigning the luminance value (F) of said distortion corrected point (m) to the initial pixel (m'"'"') in the target image (Io);

(i) repeating steps (d)-(h) for each pixel of the target image (Io), until all pixels of the target image (Io) have been processed.

2. A method as claimed in claim 1, characterized in that said method comprises calibration steps including:

storing parameters of the source images, including scale factors (z1, . . . , zn) and orientation angles constituted by azimuth angles (Θ

l, . . . , Θ

n), angles of sight (φ

1, . . . , φ

n) and angles of rotation (ψ

1, . . . , ψ

n) of source images optical axis, said angles being defined in said fixed orthonormal landmark (Px, Py, Pz);

constructing real camera models for providing said source images with said view point which is substantially common with the origin of said landmark, and for correcting aberrations in said source images;

storing target image parameters, including scale factor (zo) and orientation angles constituted by an azimuth angle (Θ

o), an angle of sight (φ

o) and an angle of rotation (ψ

o) of the target image optical axis, said angles being defined in said fixed orthonormal landmark (Px, Py, Pz); and

constructing a virtual camera model for providing said target image with said view point which is substantially common with that of the source images.

3. A method as claimed in claim 2, characterized in that

the step of determining the position of the straight light ray (PM) in said landmark comprises a geometrical transform referred to as "inverse perspective transform" (Ho^-1) which takes the virtual camera model into account;

the step of determining the address (Aq) of said distortion corrected point (m) corresponding to said initial pixel (m'"'"') comprises perspective geometrical transforms (H1-Hn) referred to as "direct perspective transforms", which, in accordance with said selected source image Ij, take into account the respective real camera model corresponding to said selected source image.

4. A method as claimed in claim 3, characterized in that the step of

calculating a luminance value (F) at said distortion corrected point (m) in said selected source image (Ij) includes performing an interpolation for computing a most probable value of a luminance function (F) at the address (Aq) in the source image (Ij).

5. A method as claimed in claim 1, characterized in that the step of

calculating a luminance value (F) at said distortion corrected point (m) in said selected source image (Ij) includes performing an interpolation for computing a most probable value of a luminance function (F) at the address (Aq) in the source image (Ij).

6. An image processing device comprising:

2

a system of n fixed real cameras (C1, . . . , Ci, Cj, . . . , Cn) arranged in such a way that their individual fields of view merge so as to form a single wide-angle field of view for observation of a panoramic scene, said real cameras providing adjacent images referred to as source images;

an image construction system simulating a mobile, virtual camera (Co) continuously scanning the panoramic scene so as to form a sub-image referred to as target image (Io) corresponding to a selected section of the wide-angle field of view and constructed from said source images (I1, . . . , Ii, Ij, . . . , In) furnished by the n real cameras, characterized in that said image processing device comprises;

means for digitizing said source and target images;

calibration means for determining a substantially common view point (P) to said images, and a fixed orthonormal landmark (Px, Py, Pz) originated at said common view point;

an address generator for generating, pixel by pixel, respective addresses for the pixels of said target image (Io) so as to cover the entire target image (Io);

an address computer for calculating, on the basis of an address (Ao) of a current pixel referred to as initial pixel (m'"'"') in the target image (Io), an orientation, in said fixed landmark, of a straight light ray (PM) passing through said initial pixel (m'"'"') and through said common view point (P) , selecting a source image (Ij) traversed by said straight light ray (PM), calculating, from said orientation of said straight light ray (PM), an address (Aq) of a distortion corrected point (m), in said selected source image (Ij), said distortion corrected point (m) corresponding to said initial point (m'"'"'); and

means for determining a luminance value (F) at said distortion corrected point (m), and assigning said luminance value to said initial point (m'"'"') .

7. A device as claimed in claim 6, characterized in that the calibration means further comprises:

first storage means for storing the parameters relating to said virtual camera for supplying the address computer with a scale factor (zo) and orientation angles of the optical axis of said virtual camera (Co) in said fixed orthonormal landmark (Px, Py, Pz) which is independent of the cameras, said orientation angles being constituted by an azimuth angle (Θ

o), an angle of sight (φ

o) and an angle of rotation (ψ

o);

second storage means for storing parameters relating to said real cameras (C1-Cn) for supplying said address computer with scale factor (z1-zn) and with orientation angles of an optical axis of each real camera (C1, . . . , Ci, Cj, Cn), said orientation angles being constituted by azimuth angles (Θ

1-Θ

n), angles of sight (φ

1-φ

n) and angles of rotation (ψ

1-ψ

n) defined in said fixed landmark.

8. A device as claimed in claim 7, characterized in that the address computer comprises:

first construction means for constructing a model (MCo) of the virtual camera with a projection via the view point P; and

second construction means for constructing models (MC1-MCn) of the real cameras with a projection via the view point P and with corrections of distortions.

9. A device as claimed in claim 8, characterized in that the address computer comprises:

first means for computing a geometrical transform, referred to as "inverse perspective transform" (H₀^-1), to said initial pixel (m'"'"') at an address (Ao) of the image (Io) of the virtual camera (Co), in said inverse perspective transform, the model of the virtual camera (MCo) provided by said first construction means and the parameters constituted by the azimuth angle (Θ

o), the angle of sight (φ

o), the angle of rotation (ψ

o) and the scale factor (zo) of said virtual camera provided by said first storage means being taken into account for determining, on the basis of said inverse perspective transform (H₀^-1), the positioning, in said landmark, of said straight light ray passing through said initial pixel (m'"'"') and through the view point (P);

means for storing the position of said straight light ray obtained by the inverse perspective transform (H_o^-1);

selection means for selecting a source image (I1-In) traversed by said straight light ray;

second means for computing a geometrical transform, referred to as "direct perspective transform" (H1-Hn), to said straight light ray in said landmark, said direct perspective transform, the models of the real cameras provided by the second construction means, the parameters constituted by the azimuth angles (Θ

1-Θ

n), the angles of sight (φ

1-φ

n), the angles of rotation (Θ

1-Θ

n) and the scale factors (z1-zn) of the respective real camera (C1-Cn) corresponding to said selected source image provided by said second storage means being taken into account; and

storage means for supplying, on the basis of said direct perspective transform (H1-Hn), an address (Aq) in said selected source image (I1-In) which corresponds to said straight light ray and thus to said initial pixel at the address (Ao) in the target image (Io).

10. A device as claimed in claim 9, characterized in that the means for determining the luminance comprise:

an interpolator for computing a most probable value (F) of a luminance function at the address (Aq) found by the address computer in said selected source image furnished by the selection means; and

third storage means for assigning said computed luminance value (F) corresponding to the address (Aq) found in said selected source image to the initial pixel in the target image (Io) at the address (Ao) furnished by said address generator.

11. A device as claimed in claim 10, characterized in that said device further comprises a display system with a screen for displaying the target image (Io) in real time on said screen.

12. A device as claimed in claim 10, characterized in that the device further comprises a recording system for recording the target image (Io).

13. A device as claimed in claim 10, characterized in that the system for constructing the target image (Io) also comprises:

an interface for enabling a user to define said parameters of the virtual camera (Co), said parameters including the scale factor (zo) and the orientation of the optical axis (Θ

o, φ

o, ψ

o).

14. A device as claimed in claim 13, characterized in that the user interface is controlled automatically or manually.

15. A device as claimed in claim 6, characterized in that the system for constructing the target image (Io) also comprises:

an interface for enabling a user to define parameters for the virtual camera (Co), said parameters including a scale factor (zo) and orientation angles (Θ

o, φ

o, ψ

o) for the optical axis defined in said fixed landmark of said virtual camera.

16. A device as claimed in claim 6, characterized in that said device further comprises a display system with a screen for displaying the target image (Io) in real time on said screen.

17. A device as claimed in claim 6, characterized in that the device further comprises a recording system for recording the target image (Io).