Image signal processor apparatus for restoring color signals at high definition by using image structural models

• US 6,343,146 B1
• Filed: 02/24/1998
• Issued: 01/29/2002
• Est. Priority Date: 02/27/1997
• Status: Expired due to Fees

• Current Assignee: Olympus Optical Corporation Limited (Olympus Corporation)
• Original Assignee: Olympus Optical Corporation Limited (Olympus Corporation)
• Inventors: Tsuruoka, Takao, Tsukioka, Taketo
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/027,590
• Time in Patent Office: 1,435 Days
• Field of Search: 382/163, 382/167, 358/518, 358/520, 358/525, 348/616, 348/638, 348/617, 348/642
• US Class Current: 382/163
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/447 by preserving the colour pattern with or without loss of information

Abstract

An image signal processor includes an area extracting unit for extracting from an image signal color signals of a local area of a predetermined size which includes a desired pixel. A maximum variance signal selecting unit selects one of the color signals which exhibits a maximum variance over the average variance of the color signals in the local area. In a signal modifying unit, the shape of the other color signals than the selected color signal is modified so that their parameters are equal to the parameters of the selected color signal. In a candidate image generating unit, the other modified color signals and the selected color signal are combined together to generate a plurality of reference image candidates. A reference image selecting unit than selects as a reference image one of the reference image candidates on the basis of a given evaluation function. In a restoring unit, dropout color signals are then restored with the use of the reference image determined by the reference image selecting unit.

Claims

1. An image signal processor for processing an image signal where each pixel is composed of more than one color signals and at least one of the color signals are dropped out according to the location of the pixel, comprising:

an extracting unit for extracting from the image signal the color signals of a local area of a predetermined size which includes a desired pixel;

a reference image generating unit for modifying and combining the color signals of the local area extracted by the extracting unit on the basis of a structure model and an evaluation function to generate an intermediate reference image, said structure model being commonly established for most applicable images in nature; and

a restoring unit for restoring the dropout color signal with the use of the intermediate reference image generated by the reference image generating unit.

2. An image signal processor according to claim 1, wherein the reference image generating unit employs as the structure model a color edge model in which the color signals have a common edge pattern.

5. An image signal processor according to claim 1, wherein the reference image generating unit exploits edge intensity calculated by Laplacian operation of each color signal, or energy, entropy, correlation, local uniformity, or moments given from density co-occurrence matrix for the evaluation function.

6. An image signal processor according to claim 2, wherein the reference image generating unit exploits edge intensity calculated by Laplacian operation of each color signal, or energy, entropy, correlation, local uniformity, or moments given from density co-occurrence matrix for the evaluation function.

9. An image signal processor according to claim 1, further comprising a selecting unit for examining the color signal, which is an output of the extracting unit, over a given evaluation criteria to select either the reference image generating unit or a linear interpolation restoring unit for processing the color signal.

10. An image signal processor according to claim 2, further comprising a selecting unit for examining the color signal, which is an output of the extracting unit, over a given evaluation criteria to select either the reference image generating unit or a linear interpolation restoring unit for processing the color signal.

12. An image signal processor according to claim 9, wherein the examination over the given evaluation criteria includes calculating edge intensity, variance, or maximum/minimum in the color signal which is an output of the extracting unit, and comparing a calculated result with a predetermined threshold.

13. An image signal processor according to claim 10, wherein the examination over the given evaluation criteria includes calculating edge intensity, variance, or maximum/minimum in the color signal which is an output of the extracting unit, and comparing a calculated result with a predetermined threshold.

14. An image input processor according to claim 1, further comprising an input unit having an imaging element for taking an image of an object to produce the color signals, the imaging element including an array of color filters, each color filter allocated to each pixel for the color signal, which is spatially arranged by repeatedly aligning square areas of an m×

n pixel size where m and n are integers of three or higher, the square area designed so that the rate of appearance is equal among the different color signals.

15. An image input processor according to claim 1, further comprising an input unit having an imaging element for taking an image of an object to produce the color signals, the imaging element including an array of color filters, each color filter allocated to each pixel for the color signal, which is spatially arranged by repeatedly aligning square areas of an jk×

jk pixel size where j is the number of the color signals used and k is an integer of two or higher, the square area designed so that the rate of appearance is equal among the different color signals and the number of the appearance of each color signals particularly designated to be larger than two and smaller than j in the horizontal and vertical directions.

16. An image input processor according to claim 1, further comprising an input unit having an imaging element for taking an image of an object to produce the color signals, the imaging element including an array of color filters, each color filter allocated to each pixel for the color signal, which is spatially arranged by repeatedly aligning square areas of an jk×

jk pixel size where j is the number of the color signals used and k is an integer of two or higher, the square area designed so that the rate of appearance is equal among the different color signals and that the arrangement of the color filters is defined by a remainder of the distance between a particular point in the square area and an integer coordinate point divided by j.

17. An image signal processor according to claim 1, wherein the restoring unit includes a luminance signal generating unit for generating a luminance signal from each the color signal restored corresponding to the reference image, and a convergence determining unit for repeating a restoring process with the luminance signals through reviewing a profile of the luminance signals.

3. An image signal processor for processing an image signal where each pixel is composed of more than one color signals and at least one of the color signals are dropped out according to the location of the pixel, comprising:

an extracting unit for extracting from the image signal the color signals of a local area of a predetermined size which includes a desired pixel;

a color signal selecting unit for calculating a parameter from each of the color signals of the local area extracted by the extracting unit and selecting one of the color signals according to the parameter;

a modifying unit for modifying the shape of the other color signals than the selected color signal so that their parameters are equal to the parameter of the selected color signal;

a candidate image generating unit for combining the other modified color signals modified by the modifying unit and the selected color signal selected by the color signal selecting unit to generate reference image candidates;

a reference image selecting unit for selecting as a reference image one of the reference image candidates generated by the candidate image generating unit on the basis of a given evaluation function; and

a restoring unit for restoring the dropout color signal with the use of the reference image determined by the reference image selecting unit.

4. An image signal processor according to claim 3, wherein the color signal selecting unit and the modifying unit exploit average and variance or maximum and minimum in the color signals as parameters to be calculated.

7. An image signal processor according to claim 3, wherein the reference image generating unit exploits edge intensity calculated by Laplacian operation of each color signal, or energy, entropy, correlation, local uniformity, or moments given from density co-occurrence matrix for the evaluation function.

8. An image signal processor according to claim 3, further comprising a selecting unit for examining the color signal, which is an output of the extracting unit, over a given evaluation criteria to select either the color signal selecting unit or a linear interpolation restoring unit for processing the color signal.

11. An image signal processor according to claim 8, wherein the examination over the given evaluation criteria includes calculating edge intensity, variance, or maximum/minimum in the color signal which is an output of the extracting unit, and comparing a calculated result with a predetermined threshold.

18. An image signal processor for processing an image signal where each pixel is composed of more than one color signals and at least one of the color signals are dropped out according to the location of the pixel, comprising:

an input unit including an optical lens system where chromatic aberration occurs and an imaging element where the rate of appearance of the color signals is adjusted based on the amount of their chromatic aberration;

an extracting unit for extracting from the image signal produced by the input means color signals of a local area of a predetermined size including a desired pixel;

an approximate reference image generating unit for modifying and combining color signals which have a low degree of the chromatic aberration on the basis of a structure model and an evaluation function to generate an approximate reference image;

a smoothing unit for smoothing the approximate reference image generated by the approximate reference image generating unit in accordance with a degree of the chromatic aberration of a high aberration color signal; and

a restoring unit for generating a reference image from the approximate reference image generated by the approximate reference image generating unit and the smoothed approximate reference image generated by the smoothing unit and utilizing it to restore the dropout color signal.

19. An image input processor according to claim 18, wherein the approximate reference image generating unit employs as the structure model a color edge model in which the color signals have a common edge pattern.

20. An image input processor according to claim 18, wherein the approximate reference image generating unit exploits edge intensity calculated by Laplacian operation of each color signal, or energy, entropy, correlation, local uniformity, or moments given from density co-occurrence matrix for the evaluation function.