Method and apparatus for image mosaicing

• US 20040056966A1
• Filed: 09/09/2003
• Published: 03/25/2004
• Est. Priority Date: 07/21/2000
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Nayar, Shree K., Schechner, Yoav Y.
• Granted Patent: US 7,440,637 B2
• Time in Patent Office: Days
• US Class Current: 348/229.1
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

A method and apparatus is provided for capturing images using a camera or other imager having imaging sensitivity characteristics which vary across the imager'"'"'s viewing angle. The imager'"'"'s characteristics can be non-uniform with respect to exposure, color sensitivity, polarization sensitivity, focal distance, and/or any other aspect of image detection. The imager is rotated or translated in order to capture different portions of the scene being imaged. Because the imager is in multiple positions when the respective scene portions are captured, each scene portion is imaged by multiple portions of the imager'"'"'s sensitivity profile.

Claims

1. A method for imaging, comprising:

a first step of using an imager to perform a first set of measurements for generating a first image value, the first set of measurements including at least one measurement of an intensity of a first radiation ray bundle from a first scene region, the first radiation ray bundle having a first chief ray in a reference frame of the imager, the imager having a first intensity sensitivity characteristic with respect to radiation ray bundles having the first chief ray, and the imager having a first dynamic range with respect to intensities of the radiation ray bundles having the first chief ray;

a second step of using the imager to perform a set of second measurements for generating a second image value, the second set of measurements comprising at least one measurement of an intensity of a second radiation ray bundle emanating from the first scene region, the second radiation ray bundle having a second chief ray in the reference frame of the imager, the second chief ray being different from the first chief ray, the imager having a second intensity sensitivity characteristic with respect to radiation ray bundles having the second chief ray, the second intensity sensitivity characteristic being different from the first intensity sensitivity characteristic, and the imager having a second dynamic range with respect to intensities of the radiation ray bundles having the second chief ray; and

applying a mosaicing operation to the first and second image values, for generating a third image value having associated therewith a third dynamic range of the imager with respect to at least one of the intensities of the first and second radiation ray bundles, the third dynamic range being greater than at least one of the first and second dynamic ranges of the imager.

2. A method according to claim 1, wherein the first radiation ray bundle comprises a first electromagnetic radiation ray bundle, the second radiation ray bundle comprising a second electromagnetic radiation ray bundle.

3. A method according to claim 1, further comprising one of:

rotating the imager with respect to the first scene region between the first and second steps; and

translating the imager with respect to the first scene region between the first and second steps.

4. A method according to claim 1, further comprising calibrating the imager, the calibrating step comprising:

using the imager to perform measurements of intensities of a first plurality of radiation ray bundles having the first chief ray, for generating a first set of calibration measurement values;

determining a first estimate of the first intensity sensitivity characteristic by determining one of;

a) a sum of the first set of calibration measurement values, and b) a mean of the first set of calibration measurement values;

using the imager to perform measurements of intensities of a second plurality of radiation ray bundles having the second chief ray, for generating a second set of calibration measurement values; and

determining a second estimate of the second intensity sensitivity characteristic by determining one of;

a) a sum of the second set of calibration measurement values, and D) a mean of the second set of calibration measurement values.

5. A method according to claim 1, further comprising calibrating the imager the calibrating step comprising:

using the imager to perform a third set of measurements for generating a fourth image value, the third set of measurements comprising at least one measurement of an intensity of a third radiation ray bundle, the third radiation ray bundle emanating from a second scene region and having the first chief ray;

using the imager to perform a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of a fourth radiation ray bundle, the fourth radiation ray bundle emanating from the second scene region and having the second chief ray; and

estimating a relationship between the first and second intensity sensitivity characteristics, the estimating step comprising determining one of;

a) a difference of the fourth and fifth image values, and b) a ratio of the fourth and fifth image values.

6. A method according to claim 1, further comprising:

using the imager to perform a third set of measurements for generating a fourth image value, the third set of measurements comprising at least one measurement of an intensity of at least one selected spectral component of a third radiation ray bundle, the third radiation ray bundle emanating from the first scene region and having a third chief ray in the reference frame of the imager, the imager having a first spectral sensitivity characteristic with respect to radiation ray bundles having the third chief ray, the first spectral sensitivity characteristic comprising a bandpass characteristic having a first wavelength sensitivity band, and the at least one selected spectral component of the third radiation ray bundle having a wavelength within the first wavelength sensitivity band; and

using the imager to perform a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of at least one selected spectral component of a fourth radiation ray bundle, the fourth radiation ray bundle emanating from the first scene region and having a fourth chief ray in the reference frame of the imager, the fourth chief ray being different from the third chief ray, the imager having a second spectral sensitivity characteristic with respect to radiation ray bundles having the fourth chief ray, the second spectral sensitivity characteristic comprising a bandpass characteristic having a second wavelength sensitivity band, the at least one selected spectral component of the fourth radiation ray bundle having a wavelength within the second wavelength sensitivity band, and the second wavelength sensitivity band being different from the first wavelength sensitivity band.

7. A method according to claim 1, further comprising:

using the imager to perform a third set of measurements for generating a fourth iamge value, the third set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a third radiation ray bundle emanating from the first scene region, the third radiation ray bundle having a third chief ray in the reference frame of the imager, the imager having a first polarization sensitivity characteristic with respect to radiation ray bundles having the third chief ray, the first polarization sensitivity characteristic comprising reduced sensitivity to radiation ray components having polarization angles outside a first angular range, and the at least one selected polarization component of the third radiation ray bundle having a polarization angle within the first angular range; and

using the imager to perform a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a fourth radiation ray bundle, the fourth radiation ray bundle emanating from the first scene region and having a fourth chief ray in the reference frame of the imager, the fourth chief ray being different from the third chief ray, the imager having a second polarization sensitivity characteristic with respect to radiation ray bundles having the fourth chief ray, the second polarization sensitivity characteristic comprising reduced sensitivity to signal components having polarization angles outside a second angular range, the at least one selected polarization component of the fourth radiation ray bundle having a polarization angle within the second angular range, and the second angular range being different from the first angular range.

8. A method according to claim 1, further comprising:

using the imager to perform a third set of measurements for generating a fourth image value, the third set of measurements comprising at least one measurement of an intensity of a third radiation ray bundle emanating from the first scene region, the third radiation ray bundle having a third chief ray in the reference frame of the imager, and the imager having a first focal characteristic with respect to radiation ray bundles having the third chief ray, the first focal characteristic comprising a first focal distance; and

using the imager to perform a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of a fourth radiation ray bundle emanating from the first scene region, the fourth radiation ray bundle having a fourth chief ray in the reference frame of the imager, the fourth chief ray being different from the third chief ray, the imager having a second focal characteristic with respect to radiation ray bundles having the fourth chief ray, the second focal characteristic comprising a second focal distance, and the second focal distance being different from the first focal distance.

9. A method for imaging, comprising:

a first step of using an imager to perform a first set of measurements for generating a first image value, the first set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a first radiation ray bundle emanating from a first scene region, the first radiation ray bundle having a first chief ray in a reference flame of the imager, the imager having a first polarization sensitivity characteristic with respect to radiation ray bundles having the first chief ray, and the first polarization sensitivity characteristic comprising reduced sensitivity to signal components having polarization angles outside a first angular range, the at least one selected polarization component of the first radiation ray bundle having a polarization angle within the first angular range;

a second step of using the imager to perform a second set of measurements for generating a second image value, the second set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a second radiation ray bundle emanating from the first scene region, the second radiation ray bundle having a second chief ray in the reference frame of the imager, the second chief ray being different from the first chief ray, the imager having a second polarization sensitivity characteristic with respect to radiation ray bundles having the second chief ray, the second polarization sensitivity characteristic comprising reduced sensitivity to signal components having polarization angles outside a second angular range, the at least one selected polarization component of the second radiation ray bundle having a polarization angle within the second angular range, and the second angular range being different from the first angular range;

a third step of moving the imager, comprising one of;

rotating the imager with respect to the first scene region between the first and second steps; and

translating the imager with respect to the first scene region between the first and second steps; and

using the first and second image values to determine a polarization state of one of the first and second radiation ray bundles.

10. A method according to claim 9, the first radiation ray bundle comprising a first electromagnetic radiation ray bundle, and the second radiation ray bundle comprising a second electromagnetic radiation ray bundle.

11. An apparatus for imaging, comprising:

a first imager for performing a first set of measurements for generating a first image value, the first set of measurements comprising at least one measurement of an intensity of a first radiation ray bundle emanating from a first scene region, the first radiation ray bundle having a first chief ray in a reference frame of the imager, the imager having a first intensity sensitivity characteristic with respect to radiation ray bundles having the first chief ray, the imager having a first dynamic range with respect to intensities of the radiation ray bundles having the first chief ray;

a second imager for performing a second set of measurements for generating a second image value, the second set of measurements comprising at least one measurement of an intensity of a second radiation ray bundle emanating from the first scene region, the second radiation ray bundle having a second chief ray in the reference frame of the imager, the second chief ray being different from the first chief ray, the imager having a second intensity sensitivity characteristic with respect to radiation ray bundles having the second chief ray, the second intensity sensitivity characteristic being different from the first intensity sensitivity characteristic, the imager having a second dynamic range with respect to intensities of the radiation signal sets having the second chief ray; and

a processor for applying a mosaicing operation to the first and second measurement values, for generating a third image value having associated therewith a third dynamic range with respect to at least one of the intensities of the first and second radiation ray bundles, the third dynamic range being greater than at least one of the first and second dynamic ranges.

12. An apparatus according to claim 11, wherein the first radiation ray bundle comprises a first electromagnetic radiation ray bundle, the second radiation ray bundle comprising a second electromagnetic radiation ray bundle.

13. An apparatus according to claim 11, wherein the second imager is the first imager, the first set of measurements being performed no later than a first time, the second set of measurements being performed no earlier than a second time, the second time being later than the first time, and the apparatus further comprising one of:

an arrangement for rotating the first imager with respect to the first and second scene regions between the first and second times; and

an arrangement for translating the first imager with respect to the first and second scene regions between the first and second times.

14. An apparatus according to claim 11, further comprising:

a processor for using the imager to measure intensities of a first plurality of radiation ray bundles having the first chief ray, for generating a first set of calibration measurement values;

a processor for determining a first estimate of the first intensity sensitivity characteristic, comprising one of;

a) a processor for determining a sum of the first set of calibration measurement values, and b) a processor for determining a mean of the first set of calibration measurement values;

a processor for using the imager to measure intensities of a second plurality of radiation ray bundles having the second chief ray, for generating a second set of calibration measurement values; and

a processor for determining a second estimate of the second intensity sensitivity characteristic, comprising one of;

a) a processor for determining a sum of the second set of calibration measurement values, and b) a processor for determining a mean of the second set of calibration measurement values.

15. An apparatus according to claim 11, further comprising:

a third imager for performing a third set of measurements for generating a fourth image value, the third set of measurements comprising at least one measurement of an intensity of a third radiation ray bundle emanating from a second scene region, and the third radiation ray bundle having the first chief ray;

a fourth imager for performing a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of a fourth radiation ray bundle emanating from the second scene region, the fourth radiation ray bundle having the second chief ray; and

a processor for estimating a relationship between the first and second intensity sensitivity characteristics, comprising one of;

a) a processor for determining a difference of the fourth and fifth image values, and b) a processor for determining a ratio of the fourth and fifth image values.

16. An apparatus according to claim 11, further comprising:

a third imager for performing a set of third measurements for generating a fourth image value, the third measurement set comprising at least one measurement of an intensity of at least one selected spectral component of a third radiation ray bundle emanating from the first scene region, the third radiation ray bundle having a third chief ray in the reference frame of the imager, the imager having a first spectral sensitivity characteristic with respect to radiation ray bundles having the third chief ray, the first spectral sensitivity characteristic comprising a bandpass characteristic having a first wavelength sensitivity band, the at least one selected spectral component of the third radiation ray bundle having a wavelength within the first wavelength sensitivity band;

a fourth imager for performing a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of at least one selected spectral component of a fourth radiation ray bundle emanating from the first scene region, the fourth radiation ray bundle having a fourth chief ray in the reference frame of the imager, the fourth chief ray being different from the third chief ray, the imager having a second spectral sensitivity characteristic with respect to radiation ray bundles having the fourth chief ray, the second spectral sensitivity characteristic comprising a bandpass characteristic having a second wavelength sensitivity band, the at least one selected spectral component of the fourth radiation ray bundle having a wavelength within the second wavelength sensitivity band, and the second wavelength sensitivity band being different from the first wavelength sensitivity band.

17. An apparatus according to claim 11, further comprising:

a third imager for performing a third set of measurements for generating a fourth image value, the third set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a third radiation ray bundle emanating from the first scene region, the third radiation ray bundle having a third chief ray in the reference frame of the imager, the imager having a first polarization sensitivity characteristic with respect to radiation ray bundles having the third chief ray, the first polarization sensitivity characteristic comprising reduced sensitivity to signal components having polarization angles outside a first angular range, and the at least one selected polarization component of the third radiation ray bundle having a polarization angle within the first angular range;

a fourth imager for performing a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a fourth radiation ray bundle emanating from the first scene region, the fourth radiation ray bundle having a fourth chief ray in the reference frame of the imager, the fourth chief ray being different from the third chief ray, the imager having a second polarization sensitivity characteristic with respect to radiation ray bundles having the fourth chief ray, the second polarization sensitivity characteristic comprising reduced sensitivity to signal components having polarization angles outside a second angular range, the at least one selected polarization component of the fourth radiation ray bundle having a polarization angle within the second angular range, and the second angular range being different from the first angular range.

18. An apparatus according to claim 11, further comprising:

a third imager for performing a third set of measurements for generating a fourth image value, the third set of measurements comprising at least one measurement of an intensity of a third radiation ray bundle emanating from the first scene region, the third radiation ray bundle having a third chief ray in the reference frame of the imager, and the imager having a first focal characteristic with respect to radiation ray bundles having the third chief ray, the first focal characteristic comprising a first focal distance;

a fourth imager for performing a fourth set of measurements for generating a fifth image value, the fourth set of measurements comprising at least one measurement of an intensity of a fourth radiation ray bundle emanating from the first scene region, the fourth radiation ray bundle having a fourth chief ray in the reference frame of the imager, the fourth chief ray being different from the third chief ray, the imager having a second focal characteristic with respect to radiation ray bundles having the fourth chief ray, the second focal characteristic comprising a second focal distance, and the second focal distance being different from the first focal distance.

19. An apparatus for imaging, comprising:

a first imager for performing a first set of measurements for generating a first image value, the first set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a first radiation ray bundle emanating from a first scene region, the first radiation ray bundle having a first chief ray in a reference frame of the imager, the imager having a first polarization sensitivity characteristic with respect to radiation ray bundles having the first chief ray, the first polarization sensitivity characteristic comprising reduced sensitivity to signal components having polarization angles outside a first angular range, the at least one selected polarization component of the first radiation ray bundle having a polarization angle within the first angular range;

a second imager for performing a second set ofI measurements for generating a second image value, the second set of measurements comprising at least one measurement of an intensity of at least one selected polarization component of a second radiation ray bundle emanating from the first scene region, the second radiation ray bundle having a second chief ray in the reference frame of the imager, the second chief ray being different from the first chief ray, the imager having a second polarization sensitivity characteristic with respect to radiation ray bundles having the second chief ray, the second polarization sensitivity characteristic comprising reduced sensitivity to signal components having polarization angles outside a second angular range, the at least one selected polarization component of the second radiation ray bundle having a polarization angle within the second angular range, and the second angular range being different from the first angular range, wherein the first set of measurements is performed no later than a first time, the second measurement set being performed no earlier than a second time, the second time being later than the first time;

a movement device comprising one of;

an arrangement for rotating the first imager with respect to the first and second scene regions between the first and second times, and

an arrangement for translating the first imager with respect to the first and second scene regions between the first and second times; and

a processor for using the first and second image values to determine a polarization state of one of the first and second radiation ray bundles.

20. An apparatus according to claim 19, the first radiation ray bundle comprising a first electromagnetic radiation ray bundle, and the second radiation ray bundle comprising a second electromagnetic radiation bundle.