Method for improving digital images and an image sensor for sensing the same

• US 20050089239A1
• Filed: 08/30/2004
• Published: 04/28/2005
• Est. Priority Date: 08/29/2003
• Status: Active Grant

• Current Assignee: Vladimir Brajovic
• Original Assignee: Vladimir Brajovic
• Inventors: Brajovic, Vladimir
• Granted Patent: US 7,876,974 B2
• Time in Patent Office: Days
• US Class Current: 382/266
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 1/4072 dependent on the contents of the original
• CPC Class Codes: H04N 1/6027 Correction or control of colour gradation or colour contrast H04N1/6058 takes precedence

Abstract

The present invention provides an image processing method. In one embodiment, the method provides discontinuity-preserving image smoothing and segmentation, noise reduction, reduction of image variations particularly those variations caused by illumination conditions, exposure compensation, and dynamic range compression. The present invention also provides an electronic image sensor that is able to detect high dynamic range optical images and produce electronic images with reduced dynamic range.

Claims

1. An electronic image sensor, comprising:

a detector array for detecting an input image and for producing an electronic image signal including a plurality of signal samples; and

an image intensity processing array having a plurality of nodes in electronic communication with the signal samples, comprising;

a discontinuity-preserving smoothing system for producing a smooth image signal from the electronic image signal;

a compensation system for compensating the electronic image signal with the smooth image signal to produce a compensated image signal; and

a blending system for blending at least two of;

the compensated image signal, the smooth image signal, and the electronic image signal to produce a sensor image signal.

2. The image sensor of claim 1, wherein the input image is visible white light, visible colored light, x-ray radiation or infrared radiation.

3. The image sensor of claim 2, fuirther comprising a luminance-color computing system for separating a colored light image into a color component and a luminance component, wherein the electronic image signal is representative of the luminance component.

4. The image sensor of claim 3, wherein the sensor image signal is colored according to the color component to produce an enhanced color image.

5. The image sensor of claim 1, wherein the image intensity processing array further comprises a sample preamplifier for amplifying the samples signals before the electronic image signal is received by the discontinuity-preserving smoothing system.

6. The image sensor of claim 5, wherein the sample preamplifier utilizes one of:

non-linear or linear signal mapping.

7. The image sensor of claim 6, wherein the non-linear signal mapping includes at least one of:

gamma curve mapping, logarithmic curve mapping, and S-curve type mapping.

8. The image sensor of claim 1, wherein at least a part of the image intensity processing array and at least part of the detector array are located within the same integrated circuit.

9. The image sensor of claim 1, wherein the discontinuity-preserving smoothing system further comprises:

a horizontal resistive element for imposing a smoothness constraint by restricting a signal difference across a pair of nodes;

a vertical resistive element for imposing a closeness constraint by restricting a difference between the electronic image signal and the signal difference across a pair of nodes; and

a balancing system for producing a smooth image signal by favoring the closeness constraint over the smoothness constraint according to a permeability function.

10. The image sensor of claim 9, wherein the permeability function is controlled by a measure of the discontinuities in the electronic image signal.

11. The image sensor of claim 9, wherein the horizontal and vertical resistive elements are MOS devices.

12. A computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the steps of:

supplying a first signal, representative of an image, to a processor;

imposing a discontinuity-preserving smoothing constraint on the first signal to produce a second signal;

compensating the first signal with the second signal to produce a third signal; and

blending at least two of;

the first signal, the second signal and the third signal to produce a blended signal.

13. A method for producing an improved signal comprising the steps of:

supplying a first signal; and

imposing a discontinuity-preserving smoothness constraint on the first signal to produce a second signal, the discontinuity-preserving constraint controlled by a permeability function in response to the first signal.

14. The method of claim 13, further comprising compensating the first signal with the second signal to produce a third signal.

15. The method of claim 14, further comprising blending at least two of:

the first signal, the second signal and the third signal to produce a blended signal.

16. The method of claim 15, wherein the blending step is controlled by the second signal, whereby the blending of at least two of:

the first signal, the second signal and the third signal are blended in proportions influenced by a magnitude of the second signal.

17. The method of claim 15, wherein the steps of supplying a first signal, imposing a discontinuity-preserving smoothing constraint on the first signal to produce a second signal, compensating the first signal with the second signal to produce a third signal are performed for each spectral component of the first signal.

18. The method of claim 13, wherein the step of imposing the discontinuity preserving smoothing constraint includes employing an absolute discontinuity measure comprising vector norms of a gradient vector of the first signal, and absolute values of a directional derivative of the first signal.

19. The method of claim 13, wherein the step of supplying the first signal further includes:

forming at least one radiance image;

sensing the at least one radiance image by a surface of a sensor; and

emitting the first signal responsive to the radiance image.

20. The method of claim 13, wherein the first signal is a color image, further comprising the steps of:

separating the color image into a luminance component and a color component, the luminance component corresponding to the first image signal; and

colorizing the second signal according to the color component.

21. The method of claim 13, wherein imposing the discontinuity-preserving smoothness constraint further comprises:

supplying a first signal to a first plurality of nodes;

producing a second signal at a second plurality of nodes, the second plurality of nodes corresponding to the first plurality of nodes;

imposing a smoothness constraint on the second signal, restricting a signal difference between a pair of second nodes;

imposing a closeness constraint on the second signal, restricting a signal difference between the first nodes and the corresponding second nodes; and

balancing the smoothness constraint and the closeness constraint.

22. The method of claim 21, wherein balancing the smoothness constraint and the closeness constraint includes favoring the closeness constraint over the smoothness constraint according to the permeability function.

23. The method of claim 22, wherein the permeability function has as its argument at least a measure of discontinuities in said first signal, whereby the permeability function causes the second signal to be substantially close to the first signal at locations where the first signal exhibits substantially large discontinuities.

24. The method of claim 21, wherein producing the second signal includes an energy function optimization, including the smoothness constraint, the closeness constraint, and the balancing of the smoothness constraint and the closeness constraint.

25. The method of claim 24, wherein the energy function optimization includes a gradient descent.

26. The method of claim 25, wherein the energy function optimization further includes a Euler-Lagrange equation corresponding to the energy function.

27. The method of claim 26, wherein the step of compensating includes dividing a magnitude of the first signal by a magnitude of the second signal.

28. The method of claim 14, wherein the step of compensating includes subtracting a magnitude of the second signal from a magnitude of the first signal.

29. An apparatus comprising a circuit for producing an improved signal, comprising:

means for supplying a first signal;

means for imposing a discontinuity-preserving smoothness constraint on the first signal to produce a second signal;

means for compensating the first signal with the second signal to produce a third signal; and

means for blending at least two of;

the first signal, the second signal and the third signal to produce a blended signal.

30. The apparatus of claim 29, wherein the means for supplying a first signal includes providing input signals including a plurality of signal samples to nodes in an array.

31. The apparatus of claim 30, wherein the means for imposing a discontinuity-preserving smoothness constraint includes restricting a difference of the signal samples between pairs of the nodes to produce a smooth image signal.

32. The apparatus of claim 29, wherein the means for imposing a discontinuity-preserving smoothness constraint includes imposing a closeness constraint for restricting a difference between the smooth image signal and corresponding samples of the first signal for at least one said node in the array.

33. The apparatus of claim 32, wherein the means for blending at least two of the first signal, the second signal and the third signal to produce a blended signal includes controlling a closeness constraint as a function of the first signal.

34. The apparatus of claim 29, wherein the means for imposing a discontinuity-preserving smoothness constraint includes a vertical resistive element Rv, and a horizontal resistive element Rh, wherein the ratio of Rh/Rv is substantially a function of the first signal.

35. The apparatus of claim 34, wherein the vertical and horizontal resistive elements are networks, each network having at least one MOS device.

36. The apparatus as defined in claim 29, wherein the means for imposing a discontinuity-preserving smoothness constraint, means for imposing a discontinuity-preserving smoothness constraint, means for compensating the first signal with the second signal, and the means for blending reside within a single integrated circuit or on a plurality of integrated circuits in electrical communication.

37. A method of producing an improved signal, the signal having at least one spectral component comprising the steps of:

supplying a first signal to a first plurality of nodes;

producing a second signal at a second plurality of nodes, the first plurality of nodes substantially corresponding to the second plurality of nodes, wherein the step of producing the second signal comprises;

imposing a smoothness constraint on the second signal restricting a signal difference between the nodes of the second signal;

imposing a closeness constraint on the second signal, thereby restricting a signal difference between the corresponding nodes in the first signal and the second signal; and

balancing the smoothness constraint and the closeness constraint by controlling at least a portion of the first signal.

38. The method of claim 37, wherein the step of balancing includes favoring the closeness constraint over the smoothness constraint according to a permeability function responsive to a portion of the first signal.

39. The method of claim 37, wherein the permeability function is a monotonic strictly rising or non decreasing function and the permeability function has an argument at least a measure of discontinuities in the first signal, whereby the permeability function causes the second signal to be substantially close to the first signal at locations where the first signal exhibits substantially large discontinuities.

40. The method of claim 39, wherein the measure of discontinuities includes an absolute discontinuity measure which determines the strength of signal discontinuities irrespective of the local signal magnitude at the location where the discontinuities occur.

41. The method of claim 40, wherein the absolute discontinuity measure is selected from the group consisting of all vector norms of a gradient vector of said first signal and an absolute value of a directional derivative of the first signal.

42. The method of claim 39, wherein the measure of discontinuities includes a relative measure of signal discontinuities for determining the strength of signal discontinuities relative to a local signal magnitude where the discontinuities occur.

43. The method of claim 42, wherein the relative measure of signal discontinuities is determined as a ratio having the magnitude of the first signal in the denominator, and having an absolute discontinuity measure selected from the group consisting of all vector norms of a gradient vector of the first signal and an absolute value of a directional derivative of the first signal in the numerator.

44. The method of claim 42, wherein the relative discontinuity measure is a Weber-Fechner contrast or a Michelson contrast.

45. The method of claim 38, wherein the permeability function flurther includes as its argument a global smoothness parameter.

46. The method of claim 38, wherein the permeability function further includes as its argument a noise statistics of the first signal.

47. The method of claim 37, wherein producing the second signal includes an energy function optimization, the energy function mathematically incorporating the smoothness constraint, the closeness constraint, and the balancing of the smoothness and closeness constraints.

48. The method of claim 47, wherein the energy function optimization is accomplished by a gradient descent method.

49. The method of claim 47, wherein the energy function optimization further includes an Euler-Lagrange equation corresponding to the energy function.

50. The method of claim 49, wherein solving the Euler-Lagrange equation is accomplished by a multigrid method for partial differential equations.

51. The method of claim 37, wherein supplying a first signal includes forming at least one radiance image which impinges on a sensitive surface of a sensing device, and sensing the radiance image, wherein the sensing device emits the first signal responsive to the radiance image.

52. The method of claim 37, further including compensating the first signal with the second signal to produce a compensated signal.

53. The method of claim 52, further including blending at least two signals selected from the group consisting of:

the first signal, the second signal and the compensated signal to produce a blended image.

54. The method of claim 53, wherein the blending is controlled by the second signal, wherein the at least two signals are blended in proportions influenced by the magnitude of the second signal.

55. The method of claim 37, wherein imposing the closeness constraint on the second signal requires the second signal to be close to a function or a transformation of the first signal.

56. The method of claim 37, further including reading instructions stored on a computer readable medium, and executing the instructions by a processor.

57. A method of producing an improved electronic image comprising the steps of:

supplying a first image;

producing a second image by smoothing the first image while substantially preventing smoothing across sharp discontinuities in the first image; and

compensating the first image with the second image to produce a compensated image.

58. The method of claim 57, further including blending at least two signals selected from the group consisting of the first image, the second image, and the compensated image, to produce a blended image.

59. The method of claim 58, wherein the first image is a color image, further comprising separating the color image into a luminance component and a color component, the luminance component corresponding to the first image, and colorizing the blended image according to the color component.

60. The method of claim 57, wherein compensating includes dividing the magnitude of the first image by the magnitude of the second image.

61. The method of claim 57, wherein compensating includes subtracting the magnitude of the second image from the magnitude of the first image.

62. The method of claim 58, wherein supplying a first image, producing a second image, compensating the first image and blending are performed for each spectral component of the first image.

63. The method of claim 62, further including color balancing, wherein the color balancing is performed using spectral components of the blended image.

64. The method of claim 57, wherein supplying the first image further includes:

forming a plurality of radiance images, each radiance image corresponding to a one spectral band and impinging on a corresponding sensitive surface of a sensing means, and

sensing each radiance image with the sensing means, wherein the sensing means emits a plurality of first images responsive to the plurality of radiance images.

65. The method of claim 57, further including reading instructions stored on a computer readable medium, and executing the instructions by a processor.

66. An apparatus comprising a circuit for simultaneously computing samples of output signals from samples of input signals comprising:

a plurality of nodes in an array, each sample of the output signal residing on one node;

means for providing the samples of the input signals, each input sample corresponding to each node in the array;

means for interconnecting a pair of nodes in the array, and restricting in part the signal difference across each pair of nodes;

means for restricting in part the difference between the samples of the output signal and the corresponding samples of the input signal for at least one node in the array; and

means for controlling the means for interconnecting a pair of nodes in the array and the means for restricting in part the difference between the samples of the output signal and the corresponding samples of the input signal, wherein the means for controlling derives a control signal as a function of the input signal.

67. The apparatus of claim 66, wherein the means for providing the samples includes an equivalent voltage source and the means for restricting is connected between the equivalent voltage source and a node in the array.

68. The apparatus of claim 66, wherein the means for providing includes an equivalent current source for introducing a signal current into each of the nodes and the means for restricting is connected between each of the nodes and a source of fixed potential.

69. The apparatus as defined in claim 66, wherein the means for restricting includes a vertical resistive element Rv and the means for interconnecting includes a horizontal resistive element Rh, wherein the ratio of resistances Rh/Rv is substantially a function of the input signal.

70. The apparatus of claim 69, wherein the function of the input signal is a nonnegative monotonic strictly increasing function of the strength of the discontinuities in the input signal.

71. The apparatus of claim 69, wherein the horizontal and vertical resistive elements are networks, each network comprised of at least one MOS device.

72. The apparatus of claim 66, wherein the means for providing further includes:

means for forming at least one radiance image corresponding to at least a one spectral band;

means for sensing each radiance image on a sensitive surface; and

means for emitting first signals responsive to each the radiance images.

73. The apparatus as defined in claim 72, wherein the means for providing, the means for interconnecting, the means for restricting and the means for controlling reside within a location comprising a single integrated circuit or a plurality of integrated circuits in electrical communication.