Image pickup method and apparatus, and image processing method and apparatus for adapting to synthesize a plurality of images

• US 7,202,892 B1
• Filed: 12/20/1999
• Issued: 04/10/2007
• Est. Priority Date: 12/21/1998
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ueda, Kazuhiko, Tsuchiya, Takashi, Ogata, Masami
• Primary Examiner(s): Srivastava; Vivek
• Assistant Examiner(s): JERABEK, KELLY L
• Application Number:
  US09/468,053
• Time in Patent Office: 2,668 Days
• Field of Search: 3482271-2301, 348298-299, 348/362, 348370-371
• US Class Current: 348/229.1
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 25/583 with different integration times
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures

Abstract

In an image pickup apparatus, a level compensator compensates the level of an image xL (i, j) sensed by an image sensor by exposure for a long time on the basis of the exposure of the image to produce a compensated image xL′ (i, j), while another level compensator compensates the level of an image xS (i, j) sensed by the image sensor by exposure for a short time on the basis of the exposure of the image to produce a compensated image xS′ (i, j). The image pickup apparatus also includes an image synthesizer to synthesize these compensates images xL′ (i, j) and xS′ (i, j) to produce a single synthetic image x (i, j). A dynamic range compressor also included in the image pickup apparatus compresses the synthetic image x (i, j) to produce a compressed image y (i, j) for delivery to outside.

Claims

1. An image pickup method of synthesizing a plurality of images acquired by sensing an object under different exposure conditions to produce a single image excellent in gradation reproducibility, the method comprising the steps of:

sensing an object under different exposure conditions to acquire a plurality of images;

calculating a pixel value for each of the plurality of images;

multiplying the pixel value obtained at the calculating step by a factor set based on the exposure condition to calculate a positive value compensation amount;

compensating the levels of the plurality of images on the basis of the exposure conditions under which they have been sensed respectively, to provide a plurality of compensated images by subtracting the positive value compensation amount from each image;

said positive valued compensation amount being calculated by multiplying a level average of each image of a coefficient based on the corresponding exposure condition;

said coefficient being selected for each image based on the exposure condition of that image;

synthesizing the plurality of compensated images to produce a single synthetic image having a wide dynamic range; and

compressing the synthetic image to an extent depending upon the performance of its output destination to produce a compressed image.

2. The method as set forth in claim 1, further comprising the steps of:

calculating a mean pixel value of each of the plurality of images; and

multiplying the mean pixel value obtained at the mean calculating step by a factor set based on the exposure condition to calculate the positive value compensation amount.

3. The method as set forth in claim 2, further comprising the steps of:

time-smoothing the positive value obtained at the multiplying step; and

subtracting the time-smoothed positive value from the pixel level of each of the plurality of images.

4. The method as set forth in claim 2, wherein the factor is set larger for the image having been sensed with a larger exposure at the image sensing step.

5. The method as set forth in claim 1, further comprising the steps of:

filtering the signal of each of the plurality of images by a predetermined low-pass filter; and

multiplying an output obtained at the filtering step by a factor set based on the exposure condition under which the image has been sensed to calculate the positive value compensation amount.

6. The method as set forth in claim 5, further comprising the steps of:

time-smoothing the positive value obtained at the multiplying step; and

subtracting, at the subtracting step, the positive value obtained at the time-smoothing step from the pixel level of each of the plurality of images.

7. The method as set forth in claim 5, wherein the factor is set larger for the image having been sensed with a larger exposure at the image sensing step.

8. The method as set forth in claim 1, wherein each of the plurality of images acquired at the image sensing step is an image signal in which a frequency-modulated color signal is superposed on a brightness signal, the method further comprising the step of separating the image signal into the brightness and color signals;

and

the brightness signal and color signal separated at the signal separating step being compensated at the level compensating step to produce a compensated brightness signal and a compensated color signal, respectively;

the compensated brightness and color signals being synthesized at the image synthesizing step to produce a synthetic brightness signal and a synthetic color signal, respectively;

wherein

a compensation amount is calculated based on the brightness signal separated at the signal separating step

the level of brightness signal is compensated with the compensation amount to produce a compensated brightness signal; and

the compensated color signal is produced based on the compensated brightness signal; and

the synthetic brightness and color signals being compressed at the image compressing step to produce a compressed brightness signal and a compressed color signal, respectively.

9. The method as set forth in claim 8, further comprising the step of mixing the compressed brightness and color signals.

10. An image pickup apparatus adapted to produce a single image excellent in gradation reproducibility from a plurality of images acquired by sensing an object under different exposure conditions, the apparatus comprising:

means for sensing an object under different exposure conditions to acquire a plurality of images;

means for calculating a pixel value for each of the plurality of images;

means for multiplying the pixel value obtained at the calculating step by a factor set based on the exposure condition to calculate a positive value compensation amount;

means for compensating the levels of the plurality of images on the basis of the exposure conditions under which they have been sensed respectively, to provide a plurality of compensated images by subtracting the positive value compensation amount from each image;

said positive valued compensation amount being calculated by multiplying a level average of each image of a coefficient based on the corresponding exposure condition;

said coefficient being selected for each image based on the exposure condition of that image;

means for synthesizing the plurality of compensated images to produce a single synthetic image having a wide dynamic range; and

means for compressing the synthetic image to an extent depending upon the performance of its output destination to produce a compressed image.

11. The apparatus as set forth in claim 10, wherein the level compensating means comprises:

means for calculating a mean pixel value of each of the plurality of images; and

means for multiplying the mean pixel value provided by the mean calculating means by a factor set based on the exposure condition to calculate the positive value compensation amount.

12. The apparatus as set forth in claim 11, wherein the level compensating means comprises:

means for time-smoothing the positive value provided by multiplying means; and

means for subtracting the time-smoothed positive value from the pixel level of each of the plurality of images.

13. The method as set forth in claim 11, wherein the factor is set larger for the image having been sensed with a larger exposure by the image sensing means.

14. The apparatus as set forth in claim 10, wherein the level compensating means comprises:

means for allowing to pass the signal of each of the plurality of images through a predetermined low-pass filter; and

means for multiplying an output of the signal filtering means by a factor set based on the exposure condition under which the image has been sensed to calculate the positive value compensation amount.

15. The apparatus as set forth in claim 14, wherein the level compensating means comprises means for time-smoothing time positive value provided by the multiplying means;

the subtracting means subtracting the time-smoothed positive value from the pixel level of each of the plurality of images.

16. The apparatus as set forth in claim 14, wherein the factor is set larger for the image having been sensed with a larger exposure by the image sensing means.

17. The apparatus as set forth in claim 10, wherein the image sensing means is adapted to output an image signal in which a frequency-modulated color signal is superposed on a brightness signal, the apparatus further comprising means for separating the image signal into the brightness and color signals;

and

the level compensating means compensating the brightness signal and color signal separated by the signal separating means to produce a compensated brightness signal and a compensated color signal, respectively;

wherein;

a compensation amount is calculated based on the brightness signal separated at the signal separating step;

the level of brightness signal is compensated with the compensation amount to produce a compensated brightness signal; and

the compensated color signal is produced based on the compensated brightness signal;

the signal synthesizing means synthesizing the compensated brightness and color signals to produce a synthetic brightness signal and a synthetic color signal, respectively; and

the image compressing means the synthetic brightness and color signals to produce a compressed brightness signal and a compressed color signal, respectively.

18. The method as set forth in claim 17, further comprising means for mixing the compressed brightness and color signals.

19. An image processing method of synthesizing a plurality of input images acquired by sensing an object under different exposure conditions to produce a single image excellent in gradation reproducibility, the method comprising the steps of:

receiving a plurality of images acquired by sensing an object under different exposure conditions and compensating the levels of the plurality of input images on the basis of the exposure conditions under which they have been sensed respectively, to provide a plurality of compensated images by subtracting a positive value compensation amount from each image;

said positive valued compensation amount being calculated by multiplying a level average of each image by a coefficient based on the corresponding exposure condition;

said coefficient being selected for each image based on the exposure condition of that image;

calculating a pixel value for each of the plurality of images;

multiplying the pixel value obtained at the calculating step by a factor set based on the exposure condition to calculate the positive value compensation amount;

synthesizing the plurality of compensated images to produce a single synthetic image having a wide dynamic range; and

compressing the synthetic image to an extent depending upon the performance of its output destination to produce a compressed image.

20. The method as set forth in claim 19, further comprising the steps of:

calculating a mean pixel value of each of the plurality of input images; and

multiplying the mean pixel value obtained at the mean calculating step by a factor set based on the exposure condition to calculate the positive value compensation amount.

21. The method as set forth in claim 20, further comprising the steps of:

time-smoothing the positive value obtained at the multiplying step; and

subtracting the time-smoothed positive value from the pixel level of each of the plurality of input images.

22. The method as set forth in claim 20, wherein the factor is set larger for the input image having been sensed with a larger exposure at the image sensing step.

23. The method as set forth in claim 19, further comprising the steps of:

filtering the signal of each of the plurality of input images by a predetermined low-pass filter; and

multiplying an output obtained at the filtering step by a factor set based on the exposure condition under which the image has been sensed to calculate the positive value compensation amount.

24. The method as set forth in claim 23, further comprising the steps of:

time-smoothing the positive value obtained at the multiplying step; and

subtracting, at the subtracting step, the positive value obtained at the time-smoothing step from the pixel level of each of the plurality of input images.

25. The method as set forth in claim 23, wherein the factor is set larger for the input image having been sensed with a larger exposure at the image sensing step.

26. The method as set forth in claim 19, wherein each of the plurality of input images acquired at the image sensing step is an image signal in which a frequency-modulated color signal is superposed on a brightness signal, the method further comprising the step of separating the image signal into the brightness and color signals;

and

the brightness signal and color signal separated at the signal separating step being compensated at the level compensating step to produce a compensated brightness signal and a compensated color signal, respectively;

the compensated brightness and color signals being synthesized at the image synthesizing step to produce a synthetic brightness signal and a synthetic color signal, respectively;

wherein

a compensation amount is calculated based on the brightness signal separated at the signal separating step

the level of brightness signal is compensated with the compensation amount to produce a compensated brightness signal; and

the compensated color signal is produced based on the compensated brightness signal; and

the synthetic brightness and color signals being compressed at the image compressing step to produce a compressed brightness signal and a compressed color signal, respectively.

27. The method as set forth in claim 26, further comprising the step of mixing the compressed brightness and color signals.

28. An image processing apparatus adapted to produce a single image excellent in gradation reproducibility from a plurality of input images acquired by sensing an object under different exposure conditions, the apparatus comprising:

means for receiving a plurality of input images acquired by sensing an object under different exposure conditions and compensating the levels of the plurality of input images on the basis of the exposure conditions under which they have been sensed respectively, to provide a plurality of compensated images by subtracting a positive valued compensation amount for each image;

said positive valued compensation amount being calculated by multiplying a level average of each image by a coefficient based on the corresponding exposure condition;

said coefficient being selected for each image based on the exposure condition of that image;

means for calculating a pixel value for each of the plurality of images;

means for multiplying the pixel value obtained at the calculating step by a factor set based on the exposure condition to calculate the positive value compensation amount;

means for synthesizing the plurality of compensated images to produce a single synthetic image having a wide dynamic range; and

means for compressing the synthetic image to an extent depending upon the performance of its output destination to produce a compressed image.

29. The apparatus as set forth in claim 28, wherein the level compensating means comprises:

means for calculating a mean pixel value of each of the plurality of input images; and

means for multiplying the mean pixel value provided by the mean calculating means by a factor set based on the exposure condition to calculate the positive value compensation amount.

30. The apparatus as set forth in claim 29, wherein the level compensating means comprises:

means for time-smoothing the positive value provided by the multiplying means; and

means for subtracting the time-smoothed positive value from the pixel level of each of the plurality of input images.

31. The apparatus as set forth in claim 29, wherein the factor is set larger for the input image having been sensed with a larger exposure by the image sensing means.

32. The apparatus as set forth in claim 28, wherein the level compensating means comprises:

means for filtering the signal of each of the plurality of input images by a predetermined low-pass filter; and

means for multiplying an output of the signal filtering means by a factor set based on the exposure condition under which the image has been sensed to calculate the positive value compensation amount.

33. The apparatus as set forth in claim 32, wherein the level compensating means comprises means for time-smoothing the positive value provided by the multiplying means;

the subtracting means subtracting the time-smoothed positive value from the pixel level of each of the plurality of input images.

34. The apparatus as set forth in claim 32, wherein the factor is set larger for the input image having been sensed with a larger exposure by the image sensing means.

35. The apparatus as set forth in claim 28, wherein the image sensing means is adapted to output an image signal in which a frequency-modulated color signal is superposed on a brightness signal, the apparatus further comprising means for separating the image signal into the brightness and color signals;

and

the level compensating means compensating the brightness signal and color signal separated by the signal separating means to produce a compensated brightness signal and a compensated color signal, respectively;

wherein;

a compensation amount is calculated based on the brightness signal separated at the signal separating step

the level of brightness signal is compensated with the compensation amount to produce a compensated brightness signal; and

the compensated color signal is produced based on the compensated brightness signal; and

the signal synthesizing means synthesizing the compensated brightness and color signals to produce a synthetic brightness signal and a synthetic color signal, respectively; and

the image compressing means the synthetic brightness and color signals to produce a compressed brightness signal and a compressed color signal, respectively.

36. The apparatus as set forth in claim 35, further comprising means for mixing the compressed brightness and color signals.

37. An image pickup apparatus adapted to synthesize a plurality of images acquired by sensing an object under different exposure conditions to produce a single image excellent in gradation and reproducibility, the apparatus comprising:

means for sensing an object under different exposure conditions to acquire a plurality of images;

means for calculating a pixel value for each of the plurality of images;

means for multiplying the pixel value obtained at the calculating step by a factor set based on the exposure condition to calculate a positive value compensation amount;

means for synthesizing plural images to produce a single synthetic image having a wide dynamic range;

means for compressing either the synthetic image or the dynamic range of the synthetic image to an extent depending upon the performance of its output destination to produce a compressed image; and

means for compensating the level of images supposed thereto to provide compensated images, whereby said compensating means is supplied with either (a) the images acquired by said sensing means to compensate the levels of said images on the basis of the exposure conditions under which said images have been sensed, respectively, by subtracting the positive valued compensation amount from each image;

said positive valued compensation amount being calculated by multiplying a level average of each image by a coefficient based on the corresponding exposure condition, and said synthesizing means is supplied with the provided compensated images, or (b) the compressed image and said synthesizing means is supplied with the acquired uncompensated images;

said coefficient being selected for each image based on the exposure condition of that image.