Image processing system comprising fixed cameras and a system simulating a mobile camera

• US 5,650,814 A
• Filed: 10/19/1994
• Issued: 07/22/1997
• Est. Priority Date: 10/20/1993
• Status: Expired due to Fees

• Current Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Original Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Inventors: Lelong, Pierre, Florent, Raoul
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US08/326,629
• Time in Patent Office: 1,007 Days
• Field of Search: 348/39, 348/36, 348/587, 348/586, 348/592, 348/38, 348/37
• US Class Current: 348/39
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

System for processing digital images includes a system of n fixed real cameras whose individual fields of view merge for recording a panoramic scene. An image construction system simulates a mobile, virtual camera continuously scanning the panoramic scene (Iv) to furnish an arbitrary sub-image, referred to as a digital target image (Iv) constructed from adjacent source images of the real cameras. A luminance equalizing system includes modules which apply correction laws (Gi, Gj), respectively, to the sets (R and F(R)) of the corresponding luminance levels, in portions (Ivi, Ivj) of a digital target image (Iv) constructed from two adjacent digital source images (Ii, Ij) to equalize these corresponding levels in accordance with the relation Gi(R)=Gj[F(R)].

Claims

1. An image processing system comprising:

a system of n fixed real cameras arranged in such a way that their individual fields of view merge so as to form a single wide-angle field of view for recording a panoramic scene,

an image construction system simulating a mobile, virtual camera (Cv) continuously scanning the panoramic scene to furnish a target sub-image (Iv) corresponding to an arbitrary section of the wide-angle field of view and constructed from adjacent source images (I1, . . . , Ii, Ij, . . . , In) furnished by the n real cameras,

wherein said processing system is a digital system that further comprises;

a luminance equalizing system for overall equalizing of corresponding luminance levels of first and second portions of a digital target image derived from two adjacent source images (Ii, I_j), said luminance equalizing System including first and second luminance correction modules which apply a first and a second correction law (Gi, Gj), respectively, to first and second sets (R,S) of the corresponding luminance levels of the first and the second portion (Ivi, Ivj) of the digital target image derived from said two adjacent digital source images (Ii, Ij), to equalize the corresponding luminance levels to the best possible extent, in accordance with a relation Gi(R)=Gj (S).

2. An image processing system as claimed in claim 1, wherein the luminance equalizing system also includes a computation module:

which computes a law F establishing a correspondence between the first set (R) of luminance levels of the first target image portion (Ivi) and the second set (S) of luminance levels of the second target image portion (Ivj), so that S=F(R), and which thereafter computes the first and second correction laws (Gi,Gj) to be applied in the first and second luminance correction modules to equalize to the best possible extent the corresponding luminance levels in accordance with a relation Gi(R)=Gj[F(R)].

3. A system as claimed in claim 1, characterized in which further comprises, for constructing the target image (Iv) in the image construction system (GVVB):

a control module controlling the mobile virtual camera (Cv) using 3 parameters;

1) varying the panoramic orientation (PAN), designated the azimuth angle, of its optical axis, corresponding to a variation in the orientation parallel to a horizontal plane of the optical axis passing through a fixed view point common to all of the fixed real cameras and the mobile virtual camera;

2) varying the orientation (TILT), designated the angle of sight, of its optical axis, always passing through the fixed view point, in a vertical plane; and

3) varying the focal length of the virtual camera to furnish a more or less enlarged target image.

4. A system as claimed in claim 1, which further comprises modules for storing processed data of the first and second target image portions (Ivi, Ivj), and a display or recording module.

5. A system as claimed in claim 1, which further comprises:

a multiplexing module for selecting from all of the real cameras C the two cameras (Ci, Cj) which correspond to the source images (Ii, Ij) required for providing the data of the target image portions (Ivi, Ivj) used in the construction of the target image (Iv); and

two analog-to-digital converter modules for converting respective analog data supplied by the two selected real cameras (Ci, Cj) into digital data to form the two corresponding digital source images (Ii,Ij).

6. An image processing system as claimed in claim 2, wherein the computation module determines the law F for the correspondence S=F(R) as the correspondence which causes the set (r) of the luminance levels of the first pixels of each pair, in the first subset (Ai), to correspond to the best possible extent to the set(s) of the corresponding luminance levels of the second pixels of each pair in the second subset (Aj) to realise to the best possible extent the equation:

s=F(r), and wherein the correspondence function F is given by F(r)=a+br^c, where a, b and c are constants determined by iteration.

7. An image processing system as claimed in claim 2, wherein the computation module computes the first luminance correction law (Gi) and the second law (Gj) as linear functions of the law of correspondence F(R).

8. An image processing system as claimed in claim 7, the first correction law is chosen to be Gi(R)=kF(R)+(1-k)R, and the second correction law is chosen to be Gj(R)=k.R+(1-k)F^-1 (R), wherein k is a constant which takes into account the surface proportion of one of the target image portions (Ivi, Ivj) in the overall image.

9. An image processing system as claimed in claim 2, wherein the luminance equalizing system also comprises:

a first memory module for selecting and storing a first sub-set (Ai) of pixels of the first target image portion (Ivi), comprising a first sub-set (r) of luminance levels;

a second memory module for selecting and storing a second sub-set (Aj) of pixels of the second target image portion (Ivj), comprising a second sub-set (s) of luminance levels;

and wherein the computation module computes the law F of correspondence S=F(R) by establishing the correspondence between the luminance levels of the two sub-sets (Ai, Aj) of target image portions selected for realising the equation s=F(r) to the best possible extent.

10. An image processing system as claimed in claim 9, characterized in that in the first and second memory modules, the first and second target image portions (Ivi, Ivj) furnish the first and second sub-sets (Ai, Aj):

by determining a position of a straight boundary line which, in the target image (Iv) corresponds to a border-to-border coincidence of the two target image portions (Ivi, Ivj) derived from the two adjacent source images (Ii and Ij),

by selecting, as the first and second subsets (Ai, Aj) of pixels, the pixels aligned in the rows and columns which are parallel to the straight boundary line and symmetrical with respect to said straight boundary line, at either side of the straight boundary line, in the first and second target image portions (Ivi, Ivj), respectively,

and in that in the computation module for the law of correspondence F;

the pixels of the first and second subsets (Ai, Aj), respectively, are transferred and classified by pixel pairs which are considered to correspond if they have as their coordinates the same coordinate on an axis parallel to the straight boundary line, and a coordinate which is symmetrical relative to the coordinate of the straight boundary line on an axis perpendicular to the straight boundary line, in a system of axes in the target image (Iv).

11. An image processing system as claimed in claim 9, characterized in that in the first and second memory modules, the first and second target image portions (Ivi, Ivj) are selected:

by determining an overlapping band (LiLj) which, in the target image (Iv), corresponds to an overlap of the two target image portions (Ivi, Ivj) one over the other, the limit of one band (Li) defining the parallel limit (Lj) of the other for forming the overlapping band derived from a zone in which the fields of view of the two adjacent real cameras (Ci, Cj) which furnish the source images (Ii, Ij) from which the target image portions (Ivi, Ivj) are constructed, overlap,

by selecting as the first and second subsets (Ai, Aj) of pixels, the pixels of the overlapping band located in the first and second target image portions (Ivi, Ivj), respectively,

and in that in the computation module for the law of correspondence F;

the pixels of the first and second subsets (Ai, Aj), respectively, are transferred and classified by pixel pairs which are considered to correspond if they have the same coordinates in a system of axes in the target image (Iv).

12. An image processing system as claimed in claim 10, wherein the computation module determines the law F for the correspondence S=F(R) as the correspondence which causes the set (r) of the luminance levels of the first pixels of each pair, in the first subset (Ai), to correspond to the best possible extent to the set (s) of the corresponding luminance levels of the second pixels of each pair in the second subset (Aj) to realise to the best possible extent the equation:

space="preserve" listing-type="equation">s=F(r).

13. An image processing system as claimed in claim 12, wherein the computation module computes the law F by means of an iterative thinning-out method by computing in a first step the function s=F(r) which approximately passes through a cluster of points N1 constituted by the luminance levels (r, s) of corresponding pairs of pixels, thereafter in a subsequent step, by weighting the points (r, s) according to their distance from a curve which is representative of the function s=F(r), the smallest weights being associated with the points most remote from said representative curve, for forming a second cluster of points N2 and by recomputing the function s=F(r) with the weighted points of said second cluster (N2), and by reducing at each step the weights of the points most remote from the curve which is representative of the function s=F(r) until the curve which is representative of the function s=F(r) passes to the best possible extent through the remaining points to which the largest weights were assigned.

14. An image processing system as claimed in claim 12, wherein the computation module computes the law F by means of an iterative thinning-out method by first computing the function s=F(r) which approximately passes through a cluster of points N1 constituted by the luminance levels (r, s) of a pair of corresponding pixels, thereafter in a subsequent step, by eliminating the points (r, s) located at a distance larger than a predetermined permissible distance from a curve which is representative of the approximative function s=F(r), to preserve a second cluster of points N1 and by recomputing the function s=F(r) using the remaining points (r, s) in the thinned-out second cluster N2, thereafter, then in each subsequent step, by reducing the value of the predetermined permissible distance between the points (r, s) and the curve which is representative of the function s=F(r), by eliminating the points (r, s) located at a distance larger than the permissible distance, and by recomputing the function s=F(r) using the remaining points (r, s), until the curve which is representative of the function s=F(r) passes to the best possible extent through the remaining points (r, s).

15. An image processing system as claimed in claim 12, wherein the correspondence function F is given by the following:

space="preserve" listing-type="equation">F(r)=a+br.sup.c

wherein a, b and c are constants determined by iteration, such that the computation module determines the law F by double iteration, in which each step comprises two phases in which a first phase for determining the parameters a, b, c includes a first stage in which a first value of the parameter c of the exponent is fixed and the parameters a and b are computed for a first cluster N1 of points (r, s) of subsequent stages in which the second, third . . . , etc. values of the parameter c of the exponent are fixed and the parameters a and b are computed for the same cluster N1 of points (r, s), whereafter the values of a group of parameters a, b, c are selected as furnishing the best function F(r)=a+br^c passing to the best possible extent through the first cluster N1 of points (r, s), while subsequently a second thinning-out stage is carried out in which the points of the first cluster N1 most remote from the function F(r) determined in the first phase are eliminated to preserve a second cluster N2 of thinned-out points, in the subsequent step the first phase is carried out by fixing the trial values of the parameter c as the value determined in the preceding step and the values around this value, by selecting the parameters a, b, c to ensure that the function F(r)=a+br^c passes to the best possible extent through the second cluster N2 of thinned-out points (r, s), and by continuing this double iteration until the function F(r)=a+br^c passes to the best possible extent through the remaining points (r, s) whose distance to the curve which is representative of the function F(r) is less than a distance determined as being permissible.

16. An image processing system as claimed in claim 11, wherein the computation module determines the law F for the correspondence S=F(R) as the correspondence which causes the set (r) of the luminance levels of the first pixels of each pair, in the first subset (Ai), to correspond to the best possible extent to the set(s) of the corresponding luminance levels of the second pixels of each pair in the second subset (Aj) to realise to the best possible extent the equation:

space="preserve" listing-type="equation">s=F(r)

17.

17. An image processing system as claimed in claim 16, wherein the correspondence function F is given by the following:

space="preserve" listing-type="equation">F(r)=a+br.sup.c

wherein a, b and c are constants determined by iteration, such that the computation module determines the law F by double iteration, in which each step comprises two phases in which a first phase for determining the parameters a, b, c includes a first stage in which a first value of the parameter c of the exponent is fixed and the parameters a and b are computed for a first cluster N1 of points (r, s) of subsequent stages in which the second, third . . . , etc. values of the parameter c of the exponent are fixed and the parameters a and b are computed for the same cluster N1 of points (r, s), whereafter the values of a group of parameters a, b, c are selected as furnishing the best function F(r)=a+br^c passing to the best possible extend through the first cluster N1 of points (r, s), while subsequently a second thinning-out stage is carried out in which the points of the first cluster N1 most remote from the function F(r) determined in the first phase are eliminated to preserve a second cluster N2 of thinned-out points, in the subsequent step the first phase is carried out by fixing the trial values of the parameter c as the value determined in the preceding step and the values around this value, by selecting the parameters a, b, c to ensure that the function F(r)=a+br^c passes to the best possible extent through the second cluster N2 of thinned-out points (r, s), and by continuing this double iteration until the function F(r)=a+br^c passes to the best possible extent through the remaining points (r s) whose distance to the curve which is representative of the function F(r) is less than a distance determined as being permissible.

18. An image processing system comprising:

a system of n fixed real cameras arranged so that their individual fields of view merge so as to form a single wide-angle field of view for recording a panoramic scene,

an image construction system simulating a mobile, virtual camera (Cv) continuously scanning the panoramic scene to construct a target sub-image (Iv) corresponding to an arbitrary section of the wide-angle field of view and derived from adjacent source images (I1, . . . , Ii, Ij . . . , In) furnished by the system of n fixed real cameras,

wherein said processing system is a digital system and further comprises;

a luminance equalizing system for overall equalizing of corresponding luminance levels of first and second portions (Ivi, Ivj) of a digital target image, where said first and second portions (Ivi, Ivj) are derived from two adjacent source images (Ii, Ij ),

said luminance equalizing system including first and second luminance correction modules for overall applying to said corresponding luminance levels of the first and second portions first and second luminance correction laws (Gi, Gj), said correction laws being evaluated, respectively, from first and second sets (R,S) of corresponding luminance levels relating to first and second sub-sets of pixels selected respectively in said first and second portions for equalizing said corresponding luminance levels of said first and second sets (R,S) to the best possible extent in accordance with the relation Gi(R)=Gj(S).

19. A system as claimed in claim 18, which further comprises, for constructing the target image (Iv) in the image construction system (GVVB):

a control module controlling the mobile virtual camera (Cv) using 3 parameters;

1) varying the panoramic orientation (PAN), designated the azimuth angle, of its optical axis, corresponding to a variation in the orientation parallel to a horizontal plane of the optical axis passing through a fixed view point common to all of the fixed real cameras and the mobile virtual camera;

2) varying the orientation (TILT), designated the angle of sight, of its optical axis, always passing through the fixed view point, in a vertical plane; and

3) varying the focal length of the virtual camera to furnish a more or less enlarged target image.