Method and apparatus for expanding the dynamic range of sensed color images

• US 6,040,858 A
• Filed: 11/16/1995
• Issued: 03/21/2000
• Est. Priority Date: 11/18/1994
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ikeda, Eiichiro
• Primary Examiner(s): Ho, Tuan
• Assistant Examiner(s): Vu, Ngoc-Yen, VU, NGOC YEN T
• Application Number:
  US08/559,074
• Time in Patent Office: 1,587 Days
• Field of Search: 348/207, 348/222, 348/224, 348/225, 348/241, 348/242, 348/250, 348/255, 348/256, 348/266, 348/272, 348/273, 348/277, 348/279, 348/362, 348/645-648, 348/675, 348/679
• US Class Current: 348/242
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/136 using complementary colours
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Disclosed is an image processing method of expanding the dynamic range by determining a saturated or noise region in an image signal and replacing the image signal with a proper image signal. A threshold value is set in units of colors on the basis of a standard image signal for each color, which is obtained upon sensing an image in a standard exposure, and a non-standard image signal for each color, which is obtained upon sensing the image in a non-standard exposure. A saturated or noise region is determined by applying the threshold value set in units of colors to the standard image signal. A region determined as a saturated or noise region is replaced with the non-standard image signal.

Claims

1. An image processing method of forming an image based on color signals output from an image sensing device, comprising:

the image picking-up step of picking up, in advance by said image sensing device, a first plurality of color signals in a standard exposure and a second plurality of color signals in a non-standard exposure;

the threshold value setting step of setting a different threshold value for each color; and

the combining step of generating a combined color signal by replacing, for each color, a part of a color signal out of the first plurality of color signals obtained in the standard exposure with a corresponding part of a corresponding color signal of the second plurality of color signals, a replacement of a color signal part being made on the basis of the threshold value set for the color.

2. The method according to claim 1, wherein the combining step further comprises the steps of:

determining whether or not a part of a color signal of the first plurality of color signals is saturated using a threshold value for the color signal set in the threshold value setting step; and

replacing the part of the color signal which is determined as saturated with the corresponding part of the corresponding color signal of the second plurality of color signals.

3. The method according to claim 2, wherein an exposure amount of the non-standard exposure is smaller than an exposure amount of the standard exposure.

4. The method according to claim 1, wherein the combining step further comprises the steps of:

determining whether or not a part of a color signal of the first plurality of color signals which have been picked up in the standard exposure exhibits a dark noise with respect to a threshold value for the color signal set in the threshold value setting step; and

replacing the part of the color signal which is determined as of low-sensitivity with the corresponding part of the corresponding color signal of the second plurality of color signals.

5. The method according to claim 4, wherein an exposure amount of the non-standard exposure is larger than an exposure amount of the standard exposure.

6. The method according to claim 1, wherein the threshold value setting step includes the step of determining a threshold value corresponding to each color on the basis of the first and second plurality of color signals.

7. The method according to claim 1, wherein the threshold value setting step includes the step of calculating the threshold value corresponding to each color by using an average value of a difference between the first and second plurality of color signals.

8. The method according to claim 1, wherein the threshold value setting step includes the step of determining the threshold value corresponding to each color depending on whether a degree of change in the color signal out of the second plurality of color signals with respect to the color signal out of the first plurality of color signals obtained in the picking-up step is large.

9. An image processing method of forming an image based on color signals output from an image sensing device, comprising:

the image picking-up step of picking up, in advance by said image sensing device, a first plurality of color signals in a standard exposure and a second plurality of color signals in a non-standard exposure;

the threshold value setting step of setting a threshold value for each color on the basis of the first plurality of color signals obtained in the standard exposure and the second plurality of color signals obtained in the non-standard exposure, in the image picking-up step; and

the combining step of generating a combined color signal by replacing, for each color, a part of a color signal out of the first plurality of color signals obtained in the standard exposure with the corresponding part of a corresponding color signal of the second plurality of color signals, a replacement of a color signal part being made on the basis of a lowest threshold value out of threshold values for the color set in the threshold value setting step.

10. The method according to claim 9, wherein the combining step further comprises the steps of:

determining whether or not a part of a color signal of the first plurality of color signals is saturated using a threshold value for the color signal set in the threshold value setting step; and

replacing the part of the color signal which is determined as saturated with the corresponding part of the corresponding color signal of the second plurality of color signals.

11. The method according to claim 10, further comprising the step of performing smoothing processing of the replaced image.

12. The method according to claim 9, wherein the threshold value setting step includes the step of calculating a threshold value corresponding to each color by using an average value of a difference between the first and second plurality of color signals.

13. The method according to claim 9, wherein the threshold value setting step includes the step of determining a threshold value corresponding to each color depending on whether a degree of change in the color signal out of the second plurality of color signals with respect to the color signal out of the first plurality of color signals is large.

14. The method according to claim 9, wherein an exposure amount of the non-standard exposure is smaller than an exposure amount of the standard exposure.

17. The method according to claim 10, further comprising the step of performing smoothing processing of the replaced image.

15. An image processing method of forming an image based on color signals output from an image sensing device, comprising:

the image picking-up step of picking up, in advance by said image sensing device, a first plurality of color signals in a standard exposure and a second plurality of color signals in a non-standard exposure;

the threshold value setting step of setting a threshold value for each color on the basis of the first plurality of color signals obtained in the standard exposure and the second plurality of color signals obtained in the non-standard exposure, in the image picking-up step; and

the combining step of generating a combined color signal by replacing, for each color, a part of a color signal out of the first plurality of color signals obtained in the standard exposure with a corresponding part of a corresponding color signal of the second plurality of color signals, a replacement of a color signal part being made on the basis of a highest threshold value out of threshold values for the color set in the threshold value setting step.

16. The method according to claim 15, wherein the combining step further comprises the steps of:

determining whether or not a part of a color signal of the first plurality of color signals exhibits a dark noise with respect to a threshold value for the color signal set in the threshold value setting step; and

replacing the part of the color signal which is determined as of low-sensitivity with the corresponding part of the corresponding color signal of the second plurality of color signals.

18. The method according to claim 15, wherein the threshold value setting step includes the step of calculating a threshold value corresponding to each color by using an average value of a difference between the first and second plurality of color signals.

19. The method according to claim 15, wherein the threshold value setting step includes the step of determining a threshold value corresponding to each color depending on whether a degree of change in the color signal out of the second plurality of color signals with respect to the color signal out of the first plurality of color signals is large.

20. The method according to claim 15, wherein an exposure amount of the non-standard exposure is larger than an exposure amount of the standard exposure.

21. An image processing apparatus which forms an image based on color signals output from an image sensing device, comprising:

image picking-up means for picking up, in advance by said image sensing device, a first plurality of color signals in a standard exposure and a second plurality of color signals in a non-standard exposure;

threshold value setting means for setting a threshold value for each color signal; and

combining means for generating a combined color signal by replacing, for each color, with a corresponding part of a corresponding color signal of the second plurality of color signals, a replacement of a color signal part being made on the basis of the threshold value set for the color.

22. An image processing apparatus for forming an image based on color signals output from an image sensing device, comprising:

image picking-up means for picking up, in advance by said image sensing device, a first plurality of color signals in a first exposure and a second plurality of color signals in a second exposure, an amount of the second exposure being larger than an amount of the first exposure;

threshold value setting means for setting a threshold value for each color on the basis of the first plurality of color signals obtained in the first exposure and the second plurality of color signals obtained in the second exposure, by the image picking-up means; and

combining means for generating a combined color signal by replacing, for each color, a part of a color signal out of the first plurality of color signals obtained in the first exposure with a corresponding part of a corresponding color signal of the second plurality of color signals, a replacement of a color signal part being made on the basis of a lowest threshold value out of threshold values for the color set by the threshold value setting means.

23. An image processing apparatus for forming an image based on color signals output from an image sensing device, comprising:

image picking-up means for picking up, in advance by said image sensing device, a first plurality of color signals in a first exposure and a second plurality of color signals in a second exposure, an amount of the second exposure being larger than an amount of the first exposure;

threshold value setting means for setting a threshold value predetermined color filter of said image sensing device for each color on the basis of the first plurality of color signals obtained in the first exposure and the second plurality of color signals obtained in the second exposure, by the image picking-up means; and

combining means for generating a combined color signal by replacing, for each color, a part of a color signal out of the first plurality of color signals obtained in the first exposure with a corresponding part of a corresponding color signal of the second plurality of color signals, a replacement of a color signal part being made on the basis of a highest threshold value out of threshold values for the color set by the threshold value setting means.

24. An image processing method of forming an image based on color signals output from an image sensing device, comprising:

an image picking-up step of picking up, in advance by said image sensing device, a plurality of color signals in respective exposures, the respective exposures being different from each other;

a threshold value setting step of setting a threshold value for each color on the basis of a predetermined kind of color signal from among the plurality of color signals obtained in the image picking-up step; and

the combining step of generating a combined color signal by selecting, for each color, a color signal from among the plurality of color signals obtained in the image picking-up step, the color signal being selected using the one of the threshold values set in the threshold value setting step which corresponds to the selected color signal.

25. The method according to claim 24, wherein the threshold value is a value that does not saturate all color signals obtained in at least two different exposures from among the respective exposures.

26. The method according to claim 24, wherein the threshold value is a value that does not generate a noise in all color signals obtained in at least two different exposures from among the respective exposures.

27. The method according to claim 24, wherein the predetermined kind of color signal used in the threshold value setting step is yellow.

28. An image processing apparatus for forming an image based on color signals output from an image sensing device, comprising:

image picking-up means for picking up, in advance by said image sensing device, a plurality of color signals in respective exposures, the respective exposures being different from each other;

threshold value setting means for setting a threshold value for each color on the basis of a predetermined kind of color signal from among the plurality of color signals obtained by the image picking-up means; and

combining means for generating a combined color signal by selecting, for each color, a color signal from among the plurality of color signals obtained by the image picking-up means, the color signal being selected using the one of the threshold values set in the threshold value setting means which corresponds to the selected color signal.

29. The apparatus according to claim 28, wherein the threshold value is a value that does not saturate all color signals obtained in at least two different exposures from among the respective exposures.

30. The apparatus according to claim 28, wherein the threshold value is a value that does not generate a noise in all color signals obtained in at least two different exposures from among the respective exposures.

31. The apparatus according to claim 28, wherein the predetermined kind of color signal used by the threshold value setting means is yellow.