Method for improving digital images and an image sensor for sensing the same

• US 7,876,974 B2
• Filed: 08/30/2004
• Issued: 01/25/2011
• Est. Priority Date: 08/29/2003
• Status: Active Grant

• Current Assignee: Vladimir Brajovic
• Original Assignee: Vladimir Brajovic
• Inventors: Brajovic, Vladimir
• Primary Examiner(s): Bhatnagar; Anand
• Assistant Examiner(s): CHU, RANDOLPH I
• Application Number:
  US10/929,188
• Publication Number:
  US 20050089239A1
• Time in Patent Office: 2,339 Days
• Field of Search: 382/264, 382/254, 382/275, 382/260, 348/597
• US Class Current: 382/264
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 1/4072 dependent on the contents of the original
• CPC Class Codes: H04N 1/6027 Correction or control of colour gradation or colour contrast H04N1/6058 takes precedence

Abstract

The present invention provides an image processing method. In one embodiment, the method provides discontinuity-preserving image smoothing and segmentation, noise reduction, reduction of image variations particularly those variations caused by illumination conditions, exposure compensation, and dynamic range compression. The present invention also provides an electronic image sensor that is able to detect high dynamic range optical images and produce electronic images with reduced dynamic range.

Claims

1. An electronic image sensor, comprising:

a detector array for detecting an input image and for producing an electronic image signal including a plurality of signal samples; and

an image intensity processing array having a plurality of nodes, each node connected to a corresponding detector in the detector array, comprising;

a discontinuity-preserving smoothing system for producing a smooth image signal from the electronic image signal, wherein large discontinuities in the image signal are substantially preserved in the smooth image signal and small discontinuities in the electronic image signal are substantially smoothed over in the smooth image signal;

the discontinuity-preserving smoothing system comprising;

a plurality of horizontal resistive elements for imposing a smoothness constraint by restricting a signal difference across a pair of nodes;

a plurality of vertical resistive elements for imposing a closeness constraint by restricting a difference between the electronic image signal and the signal on a corresponding node; and

a balancing system for producing a smooth image signal by favoring the closeness constraint over the smoothness constraint according to a permeability function, the permeability function substantially being a function of the electronic image signal.

2. The image sensor of claim 1, wherein the input image is visible white light, visible colored light, X-ray radiation or infrared radiation.

3. The image sensor of claim 2, further comprising a luminance-color computing system for separating a colored light image into a color component and a luminance component, wherein the electronic image signal is representative of the luminance component.

4. The image sensor of claim 3, wherein the smooth image signal is colored according to the color component to produce an enhanced color image.

5. The image sensor of claim 1, wherein the image intensity processing array further comprises a sample preamplifier for amplifying the signal samples before the electronic image signal is communicated to the image intensity processing array.

6. The image sensor of claim 5, wherein the sample preamplifier utilizes one of:

nonlinear or linear signal mapping.

7. The image sensor of claim 6, wherein the nonlinear signal mapping includes at least one of:

gamma curve mapping, logarithmic curve mapping, and S-curve type mapping.

8. The image sensor of claim 1, wherein at least a part of the image intensity processing array and at least part of the detector array are located within the same integrated circuit.

9. The image sensor of claim 1, wherein the horizontal and vertical resistive elements are MOS devices.

33. The image sensor of claim 1, wherein the permeability function is controlled by a measure of the strength of discontinuities in the electronic image signal, the measure including an absolute strength of discontinuities in the input image, the absolute discontinuity strength being determined as a vector norm of the gradient of the input image, or vector norm of a directional derivate of the input image.

34. The image sensor of claim 1, wherein the permeability function is controlled by a measure of the strength of discontinuities in the electronic image signal, the measure including a relative strength of discontinuities in the input image, the relative discontinuity strength being determined by a ratio of a vector norm of the gradient of the input image, and corresponding signal strength of the input image.

35. The image sensor of claim 1, wherein the permeability function is a monotonic function of a relative strength of discontinuities in the electronic image signal.

36. The image sensor of claim 1, wherein the permeability function is a monotonic function of an absolute strength of discontinuity in the electronic image signal.

37. The image sensor of claim 1, further comprising a compensation system for compensating the electronic image signal with the smooth image signal to produce a compensated image signal.

38. The image sensor of claim 37, further comprising a blending system for producing a sensor image signal as a function of one or more of:

the compensated image signal, the smooth image signal, and the electronic image signal.

10. A non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the steps of:

receiving first signal samples, representative of an image;

solving for voltages by solving a system of equations describing a discontinuity-preserving smoothing circuit for producing a second signal from the first signal, wherein large discontinuities in the first signal are substantially preserved in the second signal and small discontinuities in the first signal are substantially smoothed over in the second signal;

the discontinuity-preserving smoothing circuit comprising;

a plurality of nodes, the first signal samples received as current being injected into the nodes;

a first plurality of resistors connected between each said node and a common voltage level thus imposing a closeness constraint by restricting a difference between the nodal voltages and the common voltage level;

a second plurality of resistors connected between pairs of the nodes, thus imposing a smoothness constraint by restricting a signal difference between the corresponding pairs of nodes;

wherein the values of each resistor in the second plurality of resistors is controlled by the strength of discontinuities in the first signal, the strength of discontinuities being determined as a Weber-Fechner contrast of the first signal, thus balancing between the smoothness and closeness constraints to produce voltages on said nodes, the nodal voltages representing the second signal.

11. An apparatus comprising an analog circuit comprising:

means for supplying a first signal;

means for performing a discontinuity-preserving smoothing over at least a portion of the first signal to produce a second signal, comprising;

a first plurality of resistive elements Rh imposing a smoothness constraint by selectively restricting a difference between samples of the produced second signal;

a second plurality of resistive elements Rv imposing a closeness constraint by selectively restricting a difference between samples of the second signal and samples of the first signal;

wherein the smoothness constraint is controlled by resistance values of individual resistive elements in the first plurality of resistive elements, or the closeness constraint is controlled by resistance values of individual resistive elements in the second plurality of resistive elements, or both the smoothness and closeness constraints are controlled by resistance values of individual resistive elements in the first and second pluralities of resistive elements, thus balancing between the closeness constraint and the smoothness constraint to produce the second signal whereby large discontinuities present in the first signal are substantially preserved in the second signal, and small discontinuities present in the first signal are substantially smoothed over in the second signal, and

wherein controlling individual resistances is done in response to a permeability function, the permeability function substantially being a function of the strength of discontinuities in the first signal.

12. The apparatus of claim 11, wherein the ratio Rh/Rv, the Rh/Rv being the ratio between resistive values of each resistive element Rh and corresponding resistive element Rv, is controlled by the permeability function.

13. The apparatus of claim 11, wherein at least one of the resistive elements Rh and Rv are circuit networks, each network having at least one MOS device.

14. The apparatus as defined in claim 11, wherein at least a portion of the means for imposing a discontinuity-preserving smoothing resides within a single integrated circuit.

25. The apparatus of claim 11, further comprising:

means for compensating the first signal with the second signal to produce a third signal; and

means for producing blended signal as a function of one or more of;

the first signal, the second signal, and the third signal.

26. The apparatus as defined in claim 25, wherein the means for supplying the first signal resides within a single integrated circuit together with at least one of means selected from the group consisting of:

the means for performing a discontinuity-preserving smoothing;

the means for compensating the first signal with the second signal; and

the means for producing a blended signal.

27. The apparatus as defined in claim 26, further including a plurality of said single integrated circuits in close electric communication and operating simultaneously.

28. The apparatus as defined in claim 11, wherein the controlling permeability function includes an absolute strength of discontinuities in the first signal, the absolute discontinuity strength being determined as a vector norm of the gradient of the first signal, or vector norm of a directional derivate of the first signal.

29. The apparatus as defined in claim 11, wherein the controlling permeability function includes a relative strength of discontinuities in the first signal, the relative discontinuity strength being determined by a ratio of a vector norm of the gradient of the first signal, and corresponding local signal strength of the first signal.

30. The apparatus as defined in claim 28, wherein the controlling permeability function of the first signal is a monotonic function of a relative strength of discontinuity.

31. The apparatus as defined in claim 29, wherein the controlling permeability function of the first signal is a monotonic function of an absolute strength of discontinuity.

32. The apparatus as defined in claim 14, further including a plurality of said single integrated circuits in close electric communication operating simultaneously.

15. A method of producing an improved electronic image comprising the steps of:

supplying a first image;

producing a second image by solving for nodal voltages of a circuit comprising;

a plurality of input nodes, samples of the first image being supplied to the input nodes as a voltage source connected between the input nodes and a common voltage level;

a plurality of output nodes, wherein samples of a second image are produced as voltages on the output nodes;

a first plurality of resistors Rv connecting each input node to a corresponding output node, thereby imposing a closeness constraint by restricting a voltage difference between said input nodes and said corresponding output nodes; and

a second plurality of resistors Rh connecting pairs of output nodes, thereby imposing a smoothness constraint by restricting a voltage difference between connected nodes in the said pairs of output nodes;

controlling the values of each Rh resistor with a relative measure of discontinuities in the first image, the controlling being determined as a local spatial change of the first image relative to a local signal strength of the first image, thereby balancing between the smoothness and closeness constraints across said pluralities of input and output nodes to produce voltages on said output nodes representing the second image, whereby the second image becomes a discontinuity-preserved version of the first image in which discontinuities with a large value of the relative measure of discontinuities in the first image are preserved in the second image and discontinuities with a small value of the relative measure of discontinuities in the first image are smoothed over in the second image; and

compensating the first image with the second image to produce a compensated image.

16. The method of claim 15, further including producing a blended image as a function of one or more signals selected from the group consisting of the first image, the second image, and the compensated image.

17. The method of claim 16, wherein the first image is a color image, further comprising separating the color image into a luminance component and a color component, the luminance component corresponding to the first image, and colorizing the blended image according to the color component.

18. The method of claim 15, wherein compensating includes dividing the magnitude of the first image by the magnitude of the second image.

19. The method of claim 15, wherein compensating includes subtracting the magnitude of the second image from the magnitude of the first image.

20. The method of claim 16, wherein supplying a first image, producing a second image, compensating the first image and blending are performed for each spectral component of the first image.

21. The method of claim 20, further including color balancing, wherein the color balancing is performed using spectral components of the blended image.

22. The method of claim 15, wherein supplying the first image further includes the steps of:

forming at least one radiance image, each radiance image corresponding to at least one spectral band;

projecting each said radiance image on a corresponding sensitive surface of a sensing means; and

sensing each radiance image with the corresponding sensitive surface of the sensing means, and

emitting the first image as a plurality of images, one image emitted for each spectral band.

23. The method of claim 15, further including reading instructions stored on a computer readable medium, and executing the instructions by a processor.

24. The method of claim 15, wherein the relative measure is a ratio of a vector norm of the gradient of the first image, and corresponding local signal strength of the first image.