Computational methods and electronic camera apparatus for determining distance of objects, rapid autofocusing, and obtaining improved focus images

• US 5,193,124 A
• Filed: 06/15/1990
• Issued: 03/09/1993
• Est. Priority Date: 06/29/1989
• Status: Expired due to Term

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Subbarao, Muralidhara
• Primary Examiner(s): COUSO, JOSE L
• Application Number:
  US07/537,117
• Time in Patent Office: 998 Days
• Field of Search: 382/1, 382/7, 382/41, 382/43, 382/49, 358/88, 358/218, 358/227, 356/3, 356/4, 356/8, 356/12, 354/400, 354/403, 354/404, 350/130
• US Class Current: 382/255
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: H04N 13/236 using varifocal lenses or mirrors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

The present invention is a method and apparatus for determining the distance of a surface patch of an object from a camera system and also for focusing a surface patch of such object as well as obtaining an improved focus image of the surface patch. The present invention also includes a method of determining a set of unknown parameters of a linear shift-invariant system. The camera system of the present invention has an aperture through which light enters, an image detector, an image forming optical system having first and second principal planes and a focal length, the second principal plane arranged closer to the image detector than the first principal plane, a light filter, a camera controller, and an image processor operatively connected to the image detector and to the camera controller. The camera is set to a first set of camera parameters which include the distance(s) between the second principal plane and the image detector, the diameter (D) of the camera aperture, the focal length (f) of the camera system and the spectral characteristic (λ) of light transmitted by the light filter. The apparatus and the method of the present invention are widely applicable and they significantly enhance the efficiency of image processing to provide the distance of an object and required changes in the camera parameters to focus the object.

Claims

1. A method of determining the distance of a surface patch of an object from a camera system having an aperture through which light enters, an image detector, an image forming optical system having first and second principal planes and a focal length, said second principal plane arranged closer to said image detector than said first principal plane, a light filter, a camera controller, and an image processor operatively connected to said image detector and to said camera controller, said method comprising:

a) setting said camera to a first set of camera parameters including at least one of (i) distance (s) between said second principal plane and said image detector, (ii) diameter (D) of said camera aperture, (iii) focal length (l) of said camera system, and (iv) spectral characteristic (λ

) of light transmitted by said light filter;

b) forming a first image of said surface patch on said image detector;

c) recording said first image in said image processor as a first digital image;

d) preprocessing said first digital image to provide a first normalized image which is related to the point spread function of said camera system by a convolution operation;

e) calculating a first set of Fourier coefficients of said first normalized image at a set of discrete spatial frequencies;

f) changing at least one of said camera parameters;

g) forming a second image of said surface patch on said image detector;

h) recording said second image in said image processor as a second digital image;

i) preprocessing said second digital image to provide a second normalized image which is related to the point spread function of said camera by a convolution operation;

j) calculating a second set of Fourier coefficients of said second normalized image at said set of discrete spatial frequencies;

k) dividing said first set of Fourier coefficients by said second set of Fourier coefficients to provide a set of ratio values;

l) providing a signal mathematical relationship corresponding to each member of said set of ratios, said mathematical relationship provided without any intermediate parameter and representing the ratio of the optical transfer function of said first normalized image to the optical transfer function of said second normalized image, each said mathematical relationship dependent only on said camera parameters and having one unknown equal to said distance of said surface patch from said camera system;

m) substituting each ratio value resulting from step (k) in the signal corresponding mathematical relationship provided in step (l) to obtain a set of equations; and

n) solving said set of equations resulting from step (m) to determine said distance of said surface patch from said camera system.

2. The method of claim 1 which further includes selecting said surface patch of said object in a square field-of-view centered at selection coordinates θ

₀, φ

₀ and having corner points at θ

_i, φ

₀, where i=1,2,3,4, said selecting performed after forming of said first and second images in steps (b) and (g) and said recording of said images in steps (c) and (h).

3. The method of claim 1 wherein said preprocessing set forth in steps (d) and (i) include at least one of the following operations:

i) correcting for sensor-response characteristic of said image detector,

ii) correcting for non-uniform light transmission of said image forming optical system,

iii) normalization of the brightness of said first and second images, and

iv) magnification normalization.

4. The method of claim 1 wherein said first normalized image resulting from step (d) is stored in said image processor and is used after said second normalized image resulting from step (i) has been calculated in order to find the ratio of values of step (k).

5. The method of claim 1 wherein the region surrounding the border of said surface patch is kept dark whereby image-overlap is avoided and accuracy of said distance of said surface patch determination is enhanced.

6. The method of claim 1 wherein said surface patch is illuminated by projecting a light pattern onto it whereby a brightness variation is provided for determination of said distance of said surface patch.

7. The method of claim 1 which further comprises dividing the complete field-of-view of said camera system into many subfields-of-view each containing a surface patch, and performing steps (a) through (n) for each said surface patch whereby said distances of all surface patches are determined.

8. A method of determining the distance of a surface patch of an object from a camera system having an aperture through which light enters, an image detector, an image forming optical system having first and second principal planes and a focal length, said second principal plane arranged closer to said image detector than said first principal plane, a light filter, a camera controller, and an image processor operatively connected to said image detector and to said camera controller, said method comprising:

a) setting said camera to a first set of camera parameters including at least one of (i) distance (s) between said second principal plane and said image detector, (ii) diameter (D) of said camera aperture, (iii) focal length (f) of said camera system, and (iv) spectral characteristic (λ

) of light transmitted by said light filter;

b) forming a first image of said surface patch on said image detector;

c) recording said first image in said image processor as a first digital image;

d) preprocessing said first digital image to provide a first normalized image which is related to the point spread function of said camera system by a convolution operation;

e) calculating a first set o Fourier coefficients of said first normalized image at a set of discrete spatial frequencies;

f) changing at least one of said camera parameters;

g) forming a second image of said surface patch on said image detector;

h) recording said second image in said image processor as a second digital image;

i) preprocessing said second digital image to provide a second normalized image which is related to the point spread function of said camera by a convolution operation;

j) calculating a second set of Fourier coefficients of said second normalized image at said set of discrete spatial frequencies;

k) dividing said first set of Fourier coefficients by said second set of Fourier coefficients to provide a set of ratio values;

l) comparing the set of ratio values resulting from step (k) to a look-up table in said image processor, said look-up table having stored ratio values corresponding to distances from said camera system, and

m) determining said distance of said surface patch from said camera system by the selection of that distance which corresponds most closely to the ratio values from step (k).

9. The method of claim 8 wherein said look-up table has been provided by calculating discrete distances corresponding to said ratio values resulting from step (k) and storing same in said image processor.

10. The method of claim 8 wherein said look-up table has been experimentally provided by measuring the distances from said camera system for the set of ratio values resulting from step (k).

11. A method for focusing a surface patch of an object on an image detector of a camera system having an aperture through which light enters, an image forming optical system having first and second principal planes and a focal length, said second principal plane arranged closer to said image detector than said first principal plane, a light filter, a camera controller, and an image processor operatively connected to said image detector and to said camera controller, said method comprising:

a) setting said camera to a first set of camera parameters including at least one of (i) distance (s) between said second principal plane and said image detector, (ii) diameter (D) of said camera aperture, (iii) focal length (f) of said camera system, and (iv) spectral characteristic (λ

) of light transmitted by said light filter;

b) forming a first image of said surface patch on said image detector;

c) recording said first image in said image processor as a first digital image;

d) preprocessing said first digital image to provide a first normalized image which is related to the point spread function of said camera system by a convolution operation;

e) calculating a first set of Fourier coefficients of said first normalized image at a set of discrete spatial frequencies;

f) changing at least one of said camera parameters;

g) forming a second image of said surface patch on said image detector;

h) recording said second image in said image processor as a second digital image;

i) preprocessing said second digital image to provide a second normalized image which is related to the point spread function of said camera by a convolution operation;

j) calculating a second set of Fourier coefficients of said second normalized image at said set of discrete spatial frequencies;

k) dividing said first set of Fourier coefficients by said second set of Fourier coefficients to provide a set of ratio values;

l) providing a single mathematical relationship corresponding to each number of said set of ratios, said mathematical relationship provided without any intermediate parameter and representing the ratio of the optical transfer function of said first normalized image to the optical transfer function of said second normalized image, each said mathematical relationship dependent only on said camera parameters and having one unknown equal to said distance of said surface patch from said camera system;

m) substituting each ratio value resulting from step (k) in the single corresponding mathematical relationship provided in step (l) to obtain a set of equations;

n) solving said set of equations resulting from step (m) to determine said distance of said surface patch from said camera system;

o) based on the distance determined in step (n), determining the camera parameters corresponding to a focused image of said surface patch; and

p) adjusting said camera parameters to the values determined in step (o).

12. The method of claim 11 wherein said camera parameters are determined in step (o) by calculation using a focusing formula.

13. The method of claim 11 wherein said camera parameters are determined in step (o) by indexing the calculated distance into a prestored look-up table in said image processor which provides camera parameter values which bring said surface patch into focus.

14. A method for focusing a surface patch of an object on an image detector of a camera system having an aperture through which light enters, an image forming optical system having first and second principal planes and a focal length, said second principal plane arranged closer to said image detector than said first principal plane, a light filter, a camera controller, and an image processor operatively connected to said image detector and to said camera controller, said method comprising:

a) setting said camera to a first set of camera parameters including at least one of (i) distance (s) between said second principal plane and said image detector, (ii) diameter (D) of said camera aperture, (iii) focal length (f) of said camera system, and (iv) spectral characteristic (λ

) of light transmitted by said light filter;

b) forming a first image of said surface patch on said image detector;

c) recording said first image in said image processor as a first digital image;

d) preprocessing said first digital image to provide a first normalized image which is related to the point spread function of said camera system by a convolution operation;

e) calculating a first set of Fourier coefficients of said first normalized image at a set of discrete spatial frequencies;

f) changing at least one of said camera parameters;

g) forming a second image of said surface patch on said image detector;

h) recording said second image in said image processor as a second digital image;

i) preprocessing said second digital image to provide a second normalized image which is related to the point spread function of said camera by a convolution operation;

j) calculating a second set of Fourier coefficients of said second normalized image at said set of discrete frequencies;

k) dividing said first set of Fourier coefficients by said second set of Fourier coefficients to provide a set of ratio values;

l) comparing the set of ratio values resulting from step (k) to a look-up table in said image processor, said look-up table having stored ratio values corresponding to distance from said camera system;

m) determining said distance of said surface patch from said camera system by the selection of that distance which corresponds most closely to the ratio values from step (k);

n) based on said distance determined in step (m), determining camera parameters corresponding to a focused image of said surface patch; and

p) adjusting said camera parameters to the values determined in step (n).

15. The method of claim 14 wherein said camera parameters are determined in step (n) by calculation using a focusing formula.

16. The method of claim 14 wherein said camera parameters are determined in step (n) by indexing the calculated distance into a prestored look-up table in said image processor which provides the camera parameter values which bring said surface patch into focus.

17. A method for obtaining an improved focus image of a surface patch of an object from its blurred image in a camera system having an aperture through which light enters, an image detector, an image forming optical system having first and second principal planes and a focal length, said second principal plane arranged closer to said image detector than said first principal plane, a light filter, a camera controller, and an image processor operatively connected to said image detector and to said camera controller, said method comprising:

a) setting said camera to a first set of camera parameters including at least one of (i) distance (s) between said second principal plane and said image detector, (ii) diameter (D) of said camera aperture, (iii) focal length (f) of said camera system, and (iv) spectral characteristic (λ

) of light transmitted by said light filter;

b) forming a first image of said surface patch on said image detector;

c) recording said first image in said image processor as a first digital image;

d) preprocessing said first digital image to provide a first normalized image which is related to the point spread function of said camera system by a convolution operation;

e) calculating a first set of Fourier coefficients of said first normalized image at a set of discrete spatial frequencies;

f) changing at least one of said camera parameters;

g) forming a second image of said surface patch on said image detector;

h) recording said second image in said image processor as a second digital image;

i) preprocessing said second digital image to provide a second normalized image which is related to the point spread function of said camera by a convolution operation;

j) calculating a second set of Fourier coefficients of said second normalized image at said set of discrete spatial frequencies;

`k) dividing said firs set of Fourier coefficients by said second set of Fourier coefficients to provide a set of ratio values;

l) providing a single mathematical relationship corresponding to each member of said set of ratios, said mathematical relationship provided without any intermediate parameter and representing the ratio of the optical transfer function of said first normalized image to the optical transfer function of said second normalized image, each said mathematical relationship dependent only on said camera parameters and having one unknown equal to said distance of said surface patch from said camera system;

m) substituting each ratio value resulting from step (k) in the single corresponding mathematical relationship provided in step (l) to obtain a set of equations;

n) solving said set of equations resulting from step (m) to determine said distance of said surface patch from said camera system;

o) based on the distance of said surface patch calculated in step (n), determining the optical transfer function of said first image formed in step (b);

p) determining the Fourier transform of said first image;

q) dividing said Fourier transform resulting from step (p) by said optical transfer function resulting from step (o); and

r) computing the inverse Fourier transform of the result from step (q), whereby said improved focus image is determined from its blurred image.

18. The method of claim 17 which further includes dividing the complete field-of-view of said camera system into many subfields-of-view each containing a surface patch and repeating steps (a) through (r) for each surface patch thereby determining the improved focus image of all surface patches, and from these improved focus images synthesizing a focused image of the entire scene in the field-of-view of said camera system.

19. An apparatus for determining the distance between a surface patch of an object and a camera system comprising:

a) a camera system including an image detector, a camera aperture, an image forming optical system having first and second principal plane, a light filter and a focal length, said second principal plane arranged closer to said image detector than said first principal plane;

b) a camera controller operatively connected to said camera system for movement thereof to change one or more of the following camera parameters,

(i) the distance between said principal plane and said image detector,

(ii) the diameter of said camera aperture, and

(iii) said focal length of said image forming optical system;

) an image processor operatively connected to said image detector and to said camera controller for receipt of signals from said image detector, the image processor varying said camera parameters by means of said camera controller, wherein said image processor includes;

(i) means for converting the image detector signal to digital data and normalizing the resulting digital image for further image processing;

(ii) means for Fourier transforming signals, and

(iii) means for dividing Fourier coefficients to provide a set of Fourier coefficient ratio values corresponding to signals from a first and a second image; and

d) means for substituting Fourier coefficient ratio values gathered from said first and second images into a mathematical relationship involving the camera parameters, the distance of said surface patch and the optical transfer function of said camera system, the means for substituting Fourier coefficient ratio values also solving the resulting relationships to provide the distance of said surface patch from said camera system.

20. An apparatus for determining the distance between a surface patch of an object and a camera system as defined by claim 19 wherein said image processor further includes at least one of:

a) means for correcting sensor-response characteristics of said image detector,

b) means for correcting non-uniform light transmission of said image forming optical system,

c) means for normalization of brightness of images, and

d) means for normalization of magnification of images recorded on said image detector.

21. An apparatus for determining the distance between a surface patch of an object and a camera system comprising:

a) a camera system including an image detector, a camera aperture, an image forming optical system having first and second principal planes, a light filter and a focal length, said second principal plane arranged closer to said image detector than said first principal plane;

b) a camera controller operatively connected to said camera system for movement thereof to change one or more of the following camera parameters,

(i) the distance between said second principal plane and said image detector,

(ii) the diameter of said camera aperture, and

(iii) the focal length of said image forming optical system;

c) an image processor operatively connected to said image detector and to said camera controller for receipt of signals from said image detector, the image processor varying said camera parameters by means of said camera controller, wherein said image processor includes;

(i) means for converting the image detector signal to digital data and normalizing the resulting digital image for further image processing,

(ii) means for Fourier transforming signals, and

(iii) means for dividing Fourier coefficients to provide a set of Fourier coefficient ratio values corresponding to signals from a first and a second image,

d) a look-up table computed from the optical transfer function of said camera system, and

e) correlation means to determine distances by comparing said set of Fourier coefficient ratio values with values stored in said look-up table.

22. An apparatus for determining the distance between a surface patch of an object and a camera system as defined by claim 21 wherein said image processor further includes at least one of:

a) means for correcting sensor-response characteristics of said image detector,

b) means for correcting non-uniform light transmission of said image forming optical system,

c) means for normalization of brightness of images, and

d) means for normalizing magnification of images recorded on said image detector.