METHOD AND APPARATUS FOR EXTENDED DEPTH-OF-FIELD IMAGE RESTORATION

• US 20090297056A1
• Filed: 05/28/2008
• Published: 12/03/2009
• Est. Priority Date: 05/28/2008
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Micron Technology, Inc.
• Inventors: Venkataraman, Kartik, Chen, Junqing, Lelescu, Dan, Lu, Cheng, Mullis, Robert, Rao, Pravin
• Granted Patent: US 8,131,097 B2
• Time in Patent Office: Days
• US Class Current: 382/261
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation

Abstract

A method and apparatus are disclosed for restoring an image captured through an extended depth-of-field lens. Preprocessed data relating to image degradation is stored and used during an image restoration process.

Claims

1. A method for processing an image comprising:

receiving color channel pixel values from a pixel array which captures a degraded image through a lens system;

receiving information corresponding to at least one point spread function of the lens system with respect to the color channel;

using information corresponding to noise associated with capture of the image; and

processing the received color channel pixel values using the information on the at least one point spread function, noise, and at least one imaging system state to reduce the degradation of the image.

2. The method of claim 1, wherein the at least one imaging system state comprises gain information.

3. The method of claim 1, wherein the processing is separately conducted on each color channel of pixel values.

4. The method of claim 1, wherein the processing comprises using an adaptive linear, minimum mean square estimator to reduce image degradation.

5. The method of claim 4, wherein the estimator employs a non-iterative filter.

6. The method of claim 5, wherein the estimator comprises a Wiener filter.

7. The method of claim 4, wherein the estimator operates in a frequency domain.

8. The method of claim 4, wherein the estimator operates in a spatial domain.

9. The method of claim 1, further comprising combining the at least one point spread function information and the noise information to provide predetermined and prestored restoration information.

10. The method of claim 9, wherein the restoration information is based on an adaptive linear, minimum mean square restoration kernel.

11. The method of claim 1 wherein the color channel pixel values are divided into a plurality of tiles within the captured image and the pixels within each tile are processed using point spread function and noise information respectively associated with each tile.

12. The method of claim 11, wherein, if the current pixel is near a boundary between two tiles, then a restored current pixel value is interpolated according to:

13. The method of claim 4, wherein the processing applied to pixel values of one color channel is different from the processing applied to pixel values of another color channel.

14. The method of claim 1, wherein the noise information is associated with the received pixel values.

15. The method of claim 3, wherein the received point spread function information is different for different color channels.

16. The method of claim 11, wherein the point spread function information is different for different tiles of the captured image.

17. The method of claim 9, wherein the restoration information is represented as stored matrix values.

18. The method of claim 17, wherein the stored matrix values are represented through use of a dimensionality-reduction transform.

19. The method of claim 1, wherein the point spread function information is determined for a predetermined depth of field based on characteristics of the lens and desired performance parameters.

20. The method of claim 1, wherein the processing is performed according to a restoration filter.

21. The method of claim 20, wherein the restoration filter is determined according to:

22. The method of claim 20, wherein the restoration filter is determined according to:

23. The method of claim 20, wherein the restoration filter is determined according to:

24. The method of claim 20, wherein the restoration filter is prestored in a memory.

25. The method of claim 25, wherein the restoration filter comprises a look-up table.

26. A method of forming an imaging system comprising:

storing at least one eigen-kernel representing matrix values in a memory;

storing in the memory at least one scalar value corresponding to at least one point spread function (PSF); and

providing a processor configured to output a restored image based on a captured image the at least one eigen-kernel, and the at least one scalar value.

27. The method of claim 26, further comprising storing in the memory noise model values.

28. The method of claim 26, wherein:

the processor is configured to determine a processing order of pixels in the captured image during operation; and

the processing order is determined according to whether the PSF contains symmetry about an axis.

29. The method of claim 26, wherein the processor is configured to perform a processing order during manufacturing.

30. A method of processing a captured image comprising:

receiving a captured image;

dividing the captured image into a plurality of tiles;

reading at least one prestored eigen-kernel from memory;

reading at least one scalar value from memory; and

for each pixel in the tile;

reading image data in an M×

M window, where M is a number of pixels;

determining a mean of the windowed data;

estimating a variance of noise at a current pixel using the mean and a noise model;

estimating a variance of the image data in the window;

estimating a variance of an original image taking into account degrading blur, noise, and the variance of the image data; and

determining a signal-to-noise ratio at the current pixel.

31. The method of claim 30, further comprising reading noise model values from the memory.

32. The method of claim 30, further comprising reading imaging system state information.

33. The method of claim 30, wherein estimating a variance of an original image is performed according to:

σ

_x²=(σ

_y²−

σ

_n²)/φ

(h),

wherein φ

(h) is a scalar value representing a strength of the point spread function, σ

_n² is the estimated variance of noise, σ

_y² is the estimated variance of the image data, and σ

_x² is the estimated variance of the original image.

34. The method of claim 33 wherein the scalar value φ

(h) is determined according to;

35. The method of claim 30, further comprising:

comparing the determined signal-to-noise ratio with a predetermined threshold; and

filtering the captured image signal according to one of;

if the determined signal-to-noise ratio is lower than the threshold, then determining reconstruction coefficients by either preserving a value observed or using a smoothing filter; and

using the determined signal-to-noise ratio at the current pixel as an input to a polynomial representation to determine reconstruction coefficients to be used for determining a restoration kernel appropriate for the current pixel.

36. The method of claim 35, further comprising:

using the coefficients to linearly combine the eigen-kernels stored for the current tile to determine the restoration kernel that is to be applied at the current pixel;

performing a convolution at the current pixel using the determined restoration kernel; and

replacing a pixel value for the current pixel with the convolved value.

37. The method of claim 35, further comprising, if the current pixel is located within a tile boundary region, interpolating a value for the pixel.

38. The method of claim 35, wherein the at least one prestored eigen-kernel is different for each tile.

39. A method of processing a captured image comprising:

receiving a captured image;

dividing the captured image into a plurality of tiles;

dividing each of the plurality of tiles into at least one sub-tile;

reading at least one scalar value from memory;

reading prestored eigen-kernels from memory;

determining a signal-to-noise ratio for the current sub-tile;

for each sub-tile;

using the determined signal-to-noise ratio for the current sub-tile as an input to a polynomial representation to determine reconstruction coefficients to be used for determining a restoration kernel appropriate for the current sub-tile; and

using the coefficients to linearly combine the eigen-kernels stored for the current file to determine a restoration kernel that is to be applied at the current pixel; and

for each pixel in the tile;

reading image data in an M×

M window, where M is a number of pixels;

determining a mean of the windowed data;

performing one of;

estimating a variance of noise at a current pixel or reading an estimated variance of noise for the sub-tile from a memory;

determining an estimated variance of the image data in the window; and

determining an estimate of a variance of an original image taking into account both degrading blur and noise acting on the original image.

40. The method of claim 39, further comprising:

comparing the determined estimated original image variance with the estimated noise variance;

performing a convolution at the current pixel using the determined restoration kernel; and

replacing a pixel values for the current pixel with the convolved value.

41. The method of claim 40, further comprising if the current pixel is located within a tile boundary region, interpolating a value for the pixel.

42. A method of processing a captured image comprising:

acquiring an image;

reading from memory restoration information characterizing image blur and noise associated with a lens system used to acquire the image; and

processing the acquired image using the restoration information to reduce the image blur and noise in the acquired image.

43. The method of claim 42, wherein the image is acquired and processed by a single chip.

44. An imaging system comprising:

an imaging device having a pixel array for capturing an image through an extended depth-of-field lens;

a memory for storing data for restoring an image captured by the array; and

a processor configured to restore a captured image based on values stored in memory and the captured image.

45. The imaging system of claim 44, wherein the stored values comprise at least one eigen-kernel and at least one scalar value.

46. The imaging system of claim 45, wherein the stored values further comprise noise model information and a signal-to-noise ratio information.

47. The imaging system of claim 45, wherein the at least one eigen-kernel and at least one scalar value correspond to at least one point spread function (PSF).

48. The imaging system of claim 47, wherein the at least one PSF corresponds to characteristics of the extended depth-of-field lens.

49. The imaging system of claim 44, wherein the processor is further configured to restore the captured image according to a variation in PSFs of the extended depth-of-field lens.

50. The imaging system of claim 44, wherein the processor is further configured to restore the captured image according to an available processing rate of eigen-kernel computation.

51. An extended depth-of-field camera system comprising:

an extended depth-of-field lens;

a memory;

at least one eigen-kernel stored in the memory, the eigen-kernel representing a matrix value;

at least one scalar value corresponding to at least one point spread function stored in the memory, the point spread function being associated with the extended depth-of-field lens; and

a processor configured to output a restored image based on a captured degraded image, the at least one eigen-kernel and the at least one scalar value.

52. The camera system of claim 51, wherein the processor is further configured to restore the captured degraded image according to a stored linear non-iterative filter information.

53. The camera system of claim 51, further comprising a pixel array configured to provide color channel pixel values corresponding to the captured degraded image, the image being captured through the extended depth-of-field lens.

54. The camera system of claim 51, wherein the processor, memory, and pixel array are located on a single chip.

55. A semiconductor chip comprising:

a pixel array for capturing an image; and

a processor configure to perform the operations of;

receiving color channel pixel values from the pixel array;

receiving information corresponding to at least one point spread function with respect to the color channel;

using information corresponding to noise associated with capture of the image; and

processing the received color channel pixel values using the information on the at least one point spread function, noise, and at least one imaging system state to reduce degradation of the image.

56. The method of claim 55, wherein the at least one imaging system state comprises gain information.

57. The chip of claim 55, wherein the processing comprises using an adaptive linear, minimum mean square estimator.

58. The chip of claim 57, wherein the estimator employs a non-iterative filter.

59. The chip of claim 55, further comprising a memory for storing the at least one point spread function information, the point spread function information being stored prior to image capture.

60. The chip of claim 55, further comprising a memory for storing the noise information, the noise information being stored prior to image capture.

61. The chip of claim 55, wherein the processor is further configured to select from a plurality of processing options based on the point spread function information.

62. The chip of claim 55, wherein the processor is further configured to:

receive a captured image;

divide the captured image into a plurality of tiles;

read at least one prestored eigen-kernel from memory;

read at least one scalar value from memory; and

for each pixel in the tile;

read image data in an M×

M window, where M is a number of pixels;

determine a mean of the windowed data;

estimate an estimated variance of noise at a current pixel using the mean and a noise model,

determine an estimated variance of the image data in the window;

determine an estimate of a variance of an original image taking into account degrading blur, noise, and the variance of the image data; and

determine a signal-to-noise ratio at the current pixel.

63. The chip of claim 62, wherein the processor is further configured to:

compare the determined signal-to-noise ratio with a predetermined threshold;

determining reconstruction coefficients; and

filter the captured image signal.

64. The chip of claim 63, wherein the processor is further configured to:

use the coefficients to linearly combine the eigen-kernels stored for the current tile to determine a restoration kernel that is to be applied at the current pixel;

perform a convolution at the current pixel using the determined restoration kernel; and

replace a pixel value for the current pixel with the convolved value.

65. The chip of claim 63, wherein the processor is further configured to interpolate a value for the pixel if the current pixel is located within a tile boundary region.

66. The chip of claim 62, wherein the at least one prestored eigen-kernel is different for each tile.

67. The chip of claim 55, wherein the processor is further configured to:

receive a captured image;

divide the captured image into a plurality of tiles;

divide each of the plurality of tiles into at least one sub-tile;

read at least one scalar value from memory;

read prestored eigen-kernels from memory;

determine a signal-to-noise ratio for the current sub-tile;

for each sub-tile;

use the determined signal-to-noise ratio for the current sub-tile as an input to a polynomial representation to determine reconstruction coefficients to be used for determining a restoration kernel appropriate for the current sub-tile; and

use the coefficients to linearly combine the eigen-kernels stored for the current tile to determine a restoration kernel that is to be applied at the current pixel; and

for each pixel in the tile;

read image data in an M×

M window, where M is a number of pixels;

determine a mean of the windowed data;

perform one of;

estimating a variance of noise at a current pixel or reading a variance of noise for the sub-tile from a memory;

determine an estimated variance of the image data in the window; and

determine an estimate of a variance of an original image taking into account both degrading blur and noise acting on the original image.

68. The chip of claim 67, wherein the processor is further configured to:

compare the determined original image variance with the noise variance;

perform a convolution at the current pixel using the determined restoration kernel; and

replace a pixel values for the current pixel with the convolved value.