Solid-state image sensor having pixels shifted and complementary-color filter and signal processing method therefor

• US 6,847,397 B1
• Filed: 06/29/2000
• Issued: 01/25/2005
• Est. Priority Date: 07/01/1999
• Status: Expired due to Fees

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Photo Film Co Limited (Fujifilm Holdings Corporation)
• Inventors: Osada, Masaru
• Primary Examiner(s): Christensen, Andrew
• Assistant Examiner(s): GENCO, BRIAN C
• Application Number:
  US09/606,005
• Time in Patent Office: 1,671 Days
• Field of Search: 348/273, 358/453, 358/511, 382/167
• US Class Current: 348/273
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements

Abstract

A solid-state image sensor capable of enhancing efficient use of incident light and increasing the resolution of an image and a signal processing method therefore are disclosed. A digital camera includes an image pickup section having a photosensitive array in which photosensitive cells or photodiodes are arranged. Signal charges, or pixel data, are read out of the photodiodes, two lines at a time, three lines at a time, or three lines at a time with line-by-line shift in accordance with a color filter using complementary colors. A signal processing section includes a data correcting circuit for correcting the pixel data. Pixel data of one of three primary colors R, G and B is interpolated in the position of each virtual photosensitive cell or that of each real photosensitive cell. The above color filer uses more efficiently incident light than a filter using the primary colors and improves the sensitivity of the photosensitive cells in a dense pixel arrangement, thereby contributing to the enhancement of image quality. Further, the generated pixel data are used to interpolate pixel data in the real photosensitive cells or the virtual photosensitive cells. This is successful to broaden the frequency band of the pixel data of the real photosensitive cells or those of the virtual photosensitive cells and therefore to improve image quality while obviating false colors.

Claims

1. A solid-state image sensor for separating colors of incident light at positions corresponding to apertures formed in a screen member, transforming color-separated light to electric signals representative of an image, and processing said electric signals to thereby output broadband signals, said solid-state image sensor comprising:

an image pickup section including color separating means comprising, among a plurality of color filters each having a particular spectral characteristic for separating the colors of the light incident via the apertures, a color filter having a spectral characteristic of complementary colors;

a photosensitive array comprising photosensitive cells, which photoelectrically transform the light passed through said color filter, and photosensitive cells adjoining said photosensitive cells, wherein said photosensitive cells are bidimensionally arranged while being shifted from each other in a vertical direction and/or a horizontal direction;

electrodes so arranged as to go around the apertures formed in said photosensitive array for reading signals out of said photosensitive cells, and;

transfer registers for sequentially transferring the signals read out via said electrodes in the horizontal direction or the vertical direction of said photosensitive array;

an analog-to-digital converter for converting the signals output from said image pickup section to digital data; and

a signal processing section for generating, assuming that positions on a plane, in which the digital data are arranged, void of the photosensitive cells due to a shifted arrangement of said photosensitive cells are virtual photosensitive cells, pixel data of any one of three primary colors red (R), green (G) and blue (B) for interpolating said virtual photosensitive cells or the actually present photosensitive cells on the basis of pixel data that include a plurality of complementary colors and are read out of said actually present photosensitive cells, two lines at a time, three lines at a time, or three lines at a time with a line-by-line shift in accordance with said color filter, and broadening a frequency band of pixel data at said actually present photosensitive cells and said virtual photosensitive cells on the basis of said pixel data interpolated.

2. An image sensor in accordance with claim 1, wherein said apertures are bidimensionally arranged such that, assuming that said apertures each have a shape of either one of a square lattice and a polygon and that a distance between said photosensitive cells respectively corresponding to said apertures is a pixel pitch, said apertures are shifted from each other by one half of said pixel pitch column by column in the vertical direction or row by row in the horizontal direction, or bidimensionally arranged with a shape of either one of a square lattice rotated by 45°

and a polygon.

3. An image sensor in accordance with claim 1, wherein said color filter uses a plurality of any of cyan (Cy), magenta (Mg), yellow (Ye), white (W) and green (G) particular to subtractive mixture and has a W square lattice, CyYe full-checker pattern, in which W is arranged in a square lattice while Cy and Ye are shifted from W by one half of the pixel pitch, with nearby photosensitive cells being shifted in accordance with the shifted arrangement of said photosensitive cells.

4. An image sensor in accordance with claim 1, wherein said color filter uses a plurality of any of Cy, Mg, Ye and G particular to subtractive mixture and has a G square lattice, CyYe full-checker pattern, in which G is arranged in a square lattice while Cy and Ye are shifted from G by one half of the pixel pitch, with nearby photosensitive cells being shifted in accordance with the shifted arrangement of said photosensitive cells.

5. An image sensor in accordance with claim 1, wherein said color filter uses a plurality of any of Cy, Mg, Ye and G particular to subtractive mixture and has a partly overlapping full-checker pattern in which part of a full-checker patter using Cy, Mg, Ye and G overlaps or a partly overlapping checker pattern in which W is substituted for G, with nearby photosensitive cells being shifted from each other by one half of the pixel pitch in accordance with the shifted arrangement of said photosensitive cells.

6. An image sensor in accordance with claim 1, wherein said color filter uses Cy, Ye, Mg and G particular to subtractive mixture and has three of Cy, Ye, Mg and G combined and arranged in the vertical direction or the horizontal direction with nearby photosensitive cells being shifted from each other by one half of the pixel pitch in accordance with the shifted arrangement of said photosensitive cells.

7. An image sensor in accordance with claim 1, wherein said color filter uses Cy, Ye and W particular to subtractive mixture and has Cy, Ye and W arranged in the vertical direction or the horizontal direction with nearby photosensitive cells being shifted from each other by one half of the pixel pitch in accordance with the shifted arrangement of said photosensitive cells.

8. An image sensor in accordance with claim 1, wherein said color filter uses a plurality of any of Cy, Mg, Ye and R, G and B and has a four-color full-checker pattern in which G is positioned at a center of a CyYe full-checker pattern while G surrounding said CyYe full-checker pattern is arranged in a square lattice or in which W is substituted for G, with nearby photosensitive cells being shifted from each other by one half of the pixel pitch in accordance with the shifted arrangement of said photosensitive cells.

9. An image sensor in accordance with claim 1, wherein said image pickup reads out, in accordance with an arrangement of said color filter, the signals output from said photosensitive cells, two lines at a time, three lines at a time, or three lines at a time with a line-by-line shift.

10. An image sensor in accordance with claim 1, wherein said signal processing comprises:

an interpolating circuit for calculating the pixel data for said virtual photosensitive cells by using two lines of pixel data output from said actually present photosensitive cells at a time, and calculating luminance data for said virtual photosensitive cells by using said pixel data calculated; and

a band broadening circuit for further broadening a frequency band included in said pixel data output from said interpolating circuit.

11. An image sensor in accordance with claim 10, wherein said interpolating circuit comprises:

a first interpolating circuit for interpolating a subject virtual pixel under consideration by performing subtraction at least once with pixel data output from two or three of the actually present photosensitive cells adjoining said subject virtual pixel and including the complementary colors; and

a second interpolating circuit for calculating luminance data for the subject virtual pixel by combining pixel data of the primary colors calculated by said first interpolating circuit for the virtual photosensitive cells.

12. An image sensor in accordance with claim 11, wherein said first interpolating circuit comprises a color interpolating block for calculating, when said color filter has a combination of patterns of two or three of Cy, Mg, Ye, W and G pixel data, the pixel data and R, G and B for the subject virtual pixel by using one of a combination of R=W−

Cy, B=W−

Ye and G=Ye+Cy−

W, a combination of R=Ye−

G, B=Cy−

G and G=G, and a combination of R=Ye−

G, B=Cy−

G and G=G.

13. An image sensor in accordance with claim 11, wherein said second interpolating circuit comprises a generating block for calculating, when said color filter has a combination of patterns of three or four of Cy, Mg, Ye, W and G pixel data, luminance data Y for the subject virtual pixel by using one of a combination of Y=Ye+Cy+W, a combination of Y=Ye+Cy+G, and a combination of Y=Mg+Ye+Cy+G.

14. An image sensor in accordance with claim 11, wherein said interpolating circuit further comprises a third interpolating circuit for calculating luminance data for a subject actually present photosensitive cell under consideration by using pixel data around said subject actually present photosensitive cell.

15. An image sensor in accordance with claim 10, wherein said band broadening circuit comprises:

a luminance interpolating circuit for interpolating luminance data for a subject actually present photosensitive cell under consideration or a subject virtual photosensitive cell under consideration by using luminance data output from said interpolating circuit;

a plane interpolating circuit for further broadening, by using the pixel data output from said luminance interpolating circuit and said interpolating circuit, a frequency band included in said pixel data, and generating an R, a G and a B plane data;

a matrix circuit for generating luminance data and chrominance data by using the R, G and B plane data; and

an aperture adjusting circuit for executing contour enhancement with the luminance data output from said matrix circuit.

16. An image sensor in accordance with claim 15, wherein said luminance interpolating circuit executes low-pass filter processing with the luminance data.

17. An image sensor in accordance with claim 11, wherein said second interpolating circuit comprises an adaptive interpolating circuit for calculating subject luminance data Y under consideration by using G pixel data and R and B pixel data around said subject luminance data Y or by taking account of adaptive processing based on a decision on a horizontal color boundary and a vertical color boundary.

18. An image sensor in accordance with claim 14, wherein said third interpolating circuit comprises a luminance calculating block for calculating, when said color filter has a W square lattice, CyYe full-checker pattern in which G is arranged in a square lattice while Cy and Ye are shifted from W by one half of the pixel pitch, luminance data W for a subject Cy or a subject Ye photosensitive cell under consideration by adding a mean of pixel data of one color of said full-checker pattern different from the other color of a position of said subject Cy or Ye photosensitive cell and the pixel data of said other.

19. An image sensor in accordance with claim 14, wherein said third interpolating circuit comprises a complementary-color adaptive interpolating circuit for generating subject luminance data W under consideration by adding, while taking account of adaptive processing based on a decision on a horizontal and a vertical color boundary, a mean value of the pixel data of a direction greater in correlation than the other direction.

20. An image sensor in accordance with claim 1, wherein said signal processing section comprises:

a band-by-band data generating circuit for calculating pixel data for the virtual photosensitive cells by using the pixel data output from the actually present photosensitive cells, three lines at a time, and calculating, for each frequency band, particular luminance data and particular chrominance data relating to said virtual photosensitive cell and said actually present photosensitive cells by using said pixel data calculated; and

a frequency raising circuit for further raising frequencies of signal components included in the pixel data output from said band-by-band data generating circuit.

21. An image sensor in accordance with claim 10, wherein said band broadening circuit comprises:

a calculating circuit for performing calculations with the pixel data interpolated in the virtual photosensitive cells by said interpolating circuit in accordance with faithful color reproduction and resolution in a horizontal and/or a vertical resolution to which importance is attached to thereby output component signals;

a pseudo-adding circuit for adding, with respect to a pseudo-frequency, the component signals output from said calculating circuit and respectively having importance attached to color reproducibility and resolution;

an overlap preventing circuit for preventing, when the component signals output from said calculating circuit and respectively having importance attached to resolution in the horizontal and vertical directions include a same frequency band, said same frequency band from overlapping;

a signal adjusting circuit for controlling an amplitude of the component signal output from said calculating circuit and having importance attached to color reproducibility; and

a signal converting circuit for generating R, G and B on the basis of the component signals output from said signal adjusting circuit and said overlap preventing circuit.

22. An image sensor in accordance with claim 20, wherein said frequency raising circuit comprises:

an item-by-item calculating circuit for executing, with the pixel data output from said band-by-band data generating circuit for the virtual photosensitive cells, calculations on the basis of faithful color reproduction and resolution in a horizontal and/or a vertical direction to which importance is attached to thereby output component signals;

a pseudo-adding circuit for adding, with respect to a pseudo-frequency, the component signals output from said item-by-item calculating circuit and respectively having importance attached to color reproducibility and resolution;

an overlap preventing circuit for preventing, when the component signals output from said item-by-item calculating circuit and respectively having importance attached to resolution in the horizontal and vertical directions include a same frequency band, said same frequency band from overlapping;

a signal adjusting circuit for.executing contour enhancement and amplitude control with the component signal output from said overlap preventing circuit and having importance attached to resolution, and executing amplitude control with the component signal having importance attached to color reproducibility; and

a signal converting circuit for generating R, G and B on the basis of the component signals output from said signal adjusting circuit.

23. An image sensor in accordance with claim 21, wherein said pseudo-adding circuit comprises:

a first adding circuit for receiving at one input a first component signal output from said calculating circuit and lying in a frequency band having importance attached to resolution, and receiving at the other input a second component signal lying in a frequency band lower than said frequency band and having importance attached to color reproducibility;

a filtering circuit for freeing an output of said first adding circuit and said first component signal from aliasing distortions; and

a second adding circuit for adding outputs of said filtering circuit.

24. An image sensor in accordance with claim 22, wherein said overlap preventing circuit comprises:

a filter for limiting a frequency band included in one signal output from said pseudo-adding circuit and identical with a frequency band included in the other signal also output from said pseudo-adding circuit; and

an adding circuit for adding an output of said filter and said other signal.

25. An image sensor in accordance with claim 15, wherein said aperture adjusting circuit comprises:

an enhancement data generating circuit for performing correlation detection and interpolation with only G of the virtual photosensitive cells to thereby output a component signal attaching importance to resolution as a contour enhancing signal; and

an overlap preventing circuit for preventing, when the contour enhancing signal output from said enhancement data generating circuit attaches importance to resolution in the horizontal and vertical directions and include a same frequency band, said same frequency band from overlapping.

26. An image sensor as claimed in claim 9, wherein said image pickup section comprises a plurality of image pickup devices each having a respective photosensitive array, said image sensor further comprising beam splitting optics intervening between said plurality of image pickup devices and focusing optics, which focuses the incident light, for splitting said incident light into at least two light paths to thereby project an image of a subject on photosensitive array of said plurality of image pickup devices;

wherein the photosensitive cells constituting the respective photosensitive array are bidimensionally arranged in rows and columns and have a center, as to a geometric shape of said respective photosensitive array, shifted by one half of the pixel pitch in a direction of rows and a direction of columns from a center of the photosensitive cells constituting the other photosensitive array; and

wherein said color filter comprises a plurality of color filters respectively positioned in front of the photosensitive array and using complementary colors that produce, when superposed, pixels of primary colors at the positions where said photosensitive cells are absent.

27. An image sensor in accordance with claim 26, wherein assuming that the photosensitive section comprises two photosensitive devices, said plurality of color filters form, when combined, a W square lattice, CyYe full-checker pattern in which W is arranged in a square lattice while Cy and Ye are arranged in a full-checker pattern, a G square lattice, CyYe full-checker pattern in which G is arranged in a square lattice, a partly overlapping full-checker pattern in which part of said W square lattice, CyYe full-checker pattern overlap or in which W is substituted for G, a vertical or a horizontal stripe pattern, or a four-color full-checker pattern in which G is positioned at a center of a CyYe full-checker pattern while G surrounding said CyYe full-checker pattern is arranged in a square lattice or in which W is substituted for G.

28. An image sensor in accordance with claim 26, wherein assuming that the photosensitive section comprise three photosensitive devices, said plurality of filters comprise a first color filter with a shifted pixel pattern, a second color filter identical in color with said first color filter, but shifted in pixel pattern from said first color filter by one half of the pixel pitch, and a third color filter identical in pixel pattern with, but different in color, from said second color filter, and wherein said second and third color filters are adhered to each other.

29. An image sensor in accordance with claim 26, wherein assuming that the photosensitive section comprises three photosensitive devices, two of three color filters each having a particular shifted pixel pattern both are shifted by one pixel pitch to thereby form a color filter having an overlapping spatial arrangement.

30. A signal processing method using photosensitive cells bidimensionally arranged beneath apertures that are formed in a screen member for causing light to be incident to said photosensitive cells via a plurality of color filters each having a particular spectral sensitivity characteristic, said apertures having a shape of a square lattice or a polygon and shifted in a vertical direction column by column or in a horizontal direction row by row or a square lattice rotated by 45°

or a polygon, said method handling said photosensitive cells as bidimensional pixels for transforming the incident light to image signals based on said pixels and processing said images signals to thereby output broadband image signals, said signal processing method comprising;

a digitizing step for converting signals read out of the photosensitive cells, two lines at a time, three lines at a time or three lines at a time with a line-by-line shift in accordance with a combination pattern of said plurality of filters to digital data;

a data storing step for storing pixel data produced by said digitizing step;

a virtual pixel interpolating step for reading out the pixel data in a same manner as the signals are read out of the photosensitive cells, and interpolating pixel data of any one of R, G and B in a position of a subject virtual photosensitive cell under consideration, which is void of a photosensitive cell due to a shifted arrangement of said photosensitive cells, on the basis of pixel data of complementary colors output from the photosensitive cells adjoining said subject virtual photosensitive cell; and

a band broadening step for interpolating pixel data of R, G or B in a position of a subject actually present photosensitive cell under consideration by using the pixel data interpolated by said virtual pixel interpolating step, and using resulting pixel data to broaden a frequency band of the pixel data at positions of the actually present photosensitive cells and the virtual photosensitive cells.

31. A method in accordance with claim 30, wherein said virtual pixel interpolating step comprises:

a first interpolating step for calculating, when said virtual pixel interpolating step reads out the pixel data, two lines at a time, the pixel data for a subject virtual photosensitive cell under consideration by performing subtraction at least once with pixel data output from two or three photosensitive cells adjoining said subject virtual photosensitive cell and including complementary colors; and

a second interpolating step for calculating luminance data for the subject virtual photosensitive cell by combining pixel data of the primary colors calculated by said first interpolating step for the virtual photosensitive cells.

32. A method in accordance with claim 31, wherein said first interpolating step calculates, when generating the pixel data by using a combination of patterns of two or three of Cy, Ye and W pixel data output from the photosensitive cells adjoining the subject virtual pixel, R, B and G for said subject virtual pixel by using R=W−

Cy, B=W−

Ye and G=Ye+Cy−

W, respectively.

33. A method in accordance with claim 31, wherein said second interpolating step adds all of Cy, Ye and W pixel data output from the photosensitive cells adjoining the subject virtual photosensitive cell to thereby generate luminance data for said subject virtual photosensitive cell.

34. A method in accordance with claim 31, wherein said first interpolating step generates for the virtual photosensitive cells a G square lattice, RB full-checker pattern in which a G square lattice and an RB full-checker shifted from said G square lattice by one half of the pixel pitch are combined, said second interpolating step generating, when generating luminance data from R pixel data and B pixel data of said G square lattice, RB full-checker pattern around G pixel data, luminance data for a subject actually present photosensitive cell under consideration from a half-value of one of said R pixel data and B pixel data used to generate the luminance data and surrounding pixel data produced by adding the other pixel data closest to said one pixel data and located at an equal distance and then dividing a resulting sum by a number of the pixel data added.

35. A method in accordance with claim 31, wherein assuming that the color filter has the W square lattice, CyYe full-checker pattern, said first interpolating step generates color pixel data for the virtual photosensitive cells, said virtual pixel interpolating step further comprising:

a third interpolating step for calculating, assuming that Cy or Ye of said color filter is used at a position of the actually present photosensitive cell and that luminance data is to be generated at the position of the photosensitive cell of one of Cy or Ye, the luminance data on the basis of four image data of the other color adjoining said photosensitive cell, adding said luminance data and pixel data of said photosensitive cell, and generating, based on a resulting sum, pixel data having importance attached to faithful color reproduction in a horizontal direction and/or a vertical direction and pixel data having importance attached to resolution;

the luminance data being calculated at either one of the virtual photosensitive cell and the actually present photosensitive cell.

36. A method in accordance with claim 31, wherein assuming that the color filter has the W square lattice, CyYe full-checker pattern, said first interpolating step generates color pixel data at positions of the virtual photosensitive cells, the virtual pixel interpolating step further comprising:

a fourth interpolating step for calculating, assuming that Cy or Ye is used at a position of a subject actually present photosensitive cell under consideration and that luminance data is to be generated at a position of one of Cy and Ye photosensitive cells, first correlation values in a horizontal/vertical direction with four pixel data of the other color adjoining said one photosensitive cell, producing a mean value of two photosensitive cells positioned in a direction of a first correlation value greater than a preselected value or producing, if said correlation values are smaller than said preselected value in both of the horizontal and vertical directions, a mean value of said four pixel data, adding said mean value and pixel data of said photosensitive cell, and generating, based on a resulting sum, pixel data having importance attached to faithful color reproduction in the horizontal and/or the vertical direction and pixel data having importance attached to resolution;

the luminance data being calculated at either one of the virtual photosensitive cell and the actually present photosensitive cell.

37. A method in accordance with claim 36 wherein said fourth interpolating step calculates the first correlation values in the horizontal/vertical direction on the basis of four pixel data adjoining the subject photosensitive cell, and produces a mean value of the two photosensitive cells positioned in the direction of the correlation value greater than said first preselected value, or

uses, if the first correlation values are smaller than the first preselected value in both of the horizontal and vertical directions, four W pixel data adjoining the subject photosensitive cell, adds absolute values of differences between the pixel data positioned in the horizontal direction and adds absolute values of differences between the pixel data positioned in the vertical direction to thereby produce second correlation values in the horizontal and vertical directions, respectively, and produces a mean value of two photosensitive cells positioned in a direction of the second correlation value greater than said second preselected value, or

produces a mean value of the four W pixel data if the correlation values are smaller than the second preselected value in both of the horizontal and vertical directions, and

adds the mean value and the pixel data of the subject photosensitive cell, and generates, based on a resulting sum, pixel data having importance attached to faithful color reproduction in the horizontal and/or the vertical direction and pixel data having importance attached to resolution.

38. A method in accordance with claim 30, wherein, assuming that the color filter has the four-color full-checker pattern, said virtual pixel interpolating step comprises:

a fifth interpolating step for interpolating, when the pixel data stored by said data storing step are read out, three lines at a time or three lines at a time with a line-by-line shift, R, G and B for the virtual photosensitive cell surrounded by the photosensitive cells of four colors;

a sixth interpolating step for interpolating pixel data produced by said fifth interpolating step in the photosensitive cells;

a low-frequency luminance calculating step for calculating luminance data for a subject virtual photosensitive cell under consideration positioned between Cy and Ye by adding two Cy and Ye pixel data arranged in the horizontal and vertical directions; and

a high-frequency luminance calculating step for generating, assuming that G pixel data at a G photosensitive cell is high-frequency luminance data, said high-frequency luminance data by adding a double of said G pixel data and pixel data output from an Mg photosensitive cell.

39. A method in accordance with claim 30, wherein said band broadening step comprises:

a luminance interpolating step for interpolating luminance data in positions of the photosensitive cells by using the luminance data produced by said virtual pixel interpolating step;

a plane interpolating step for broadening a frequency band of pixel data produced by said luminance interpolating step and said virtual pixel interpolating step, and generating R plane data, G plane data and B plane data;

a matrix step for generating luminance data and chrominance data on the basis of the R, G and B plane data; and

an aperture adjusting step for executing contour enhancement with the luminance data generated by said matrix step.

40. A method in accordance with claim 39, wherein said luminance interpolating step comprises:

a horizontal interpolating step for generating, based on the luminance data generated by said virtual pixel interpolating step, luminance data for the photosensitive cells in an order of the horizontal direction and the vertical direction, or vice versa, executing, when interpolating pixel data in positions of the photosensitive cells in the horizontal direction, a zeroth-stage horizontal step that multiplies each pixel data to be interpolated by a particular coefficient and a first-stage horizontal step that adds right data and left data adjoining said pixel data and multiplies a resulting sum by a coefficient, and repeating n times a horizontal step that performs multiplication with pixel data equally spaced from said pixel data to be interpolated in the horizontal direction with a coefficient in the same manner as in said first-stage horizontal step, and adding all of products output after said zeroth-stage and first-stage horizontal steps to thereby generate pixel data to interpolate; and

a vertical processing step for multiplying, when generating pixel data for a subject position under consideration by using the pixel data produced by said horizontal interpolating step, multiplying the pixel data positioned in the vertical direction by a corresponding coefficient and adding all of resulting products to thereby execute vertical low-pass processing.

41. A method in accordance with claim 31, wherein said second interpolating step produces, when calculating the luminance data, first correlation values in the horizontal and vertical directions and compares said first correlation values with a first preselected value, said second interpolating step comprising:

a horizontal luminance calculating step for adding, if a result of comparison indicates that a correlation holds in the horizontal direction, pixel data in the horizontal direction and dividing a resulting sum by a double of a number of pixel data added to thereby produce surrounding pixel data;

a vertical luminance calculating step for adding, if the result of comparison indicates that a correlation holds in the vertical direction, pixel data in the vertical direction and dividing a resulting sum by a double of a number of pixel data added to thereby produce surrounding pixel data; and

a mean luminance calculating step for adding, if correlation values in the horizontal and vertical directions resulting from the comparison both are smaller than the preselected value, the other pixel data closest to the subject and located at an equal distance, and dividing a resulting sum by a double of a number of pixels added to thereby produce surrounding pixel data;

a luminance data pattern being generated on the basis of a half-value data of the subject pixel data and said surrounding pixel data output by any one of said horizontal luminance calculating step, said vertical luminance calculating step, and said mean luminance calculating step.

42. A method in accordance with claim 41, wherein said second interpolating step compares, before calculating the luminance data, the first correlation values of the horizontal and vertical directions with the first preselected value, executes said horizontal luminance calculating step or said vertical luminance calculating step in accordance with a result of comparison, adds, assuming that one of the R pixel data and B pixel data is the luminance data to be generated, correlation values derived from the other pixel data horizontally aligning with each other at both sides of said luminance data to thereby produce second correlation values in the horizontal and vertical directions, and compares said second correlation values with a second preselected value, said second interpolating step comprising:

said horizontal luminance calculating step executed when a result of comparison indicates that a correlation holds in the horizontal direction;

said vertical luminance calculating step executed when the result of comparison indicates that a correlation holds in the vertical direction; and

said mean luminance calculating step executed when the result of comparison indicates that a correlation does not hold in the vertical or the horizontal direction;

a luminance data pattern being generated on the basis of a half-value data of the subject pixel data and the surrounding pixel data output by any one of said horizontal luminance calculating step, said vertical luminance calculating step, and said mean luminance calculating step.

43. A method in accordance with claim 42, wherein said second interpolating step adds G pixel data diagonally aligning with each other and calculates, if an absolute value of a difference between resulting sums is greater than a third preselected value, the surrounding pixel data in said mean luminance calculating step.

44. A method in accordance with claim 31, wherein said second interpolating step calculates, before calculating the luminance data, a first right oblique correlation value and a first left oblique correlation value with differences between the pixel data of a same color obliquely aligning with each other in two directions with the intermediary of the subject pixel data whose luminance is to be calculated, and compares said first right oblique correlation value and said first left oblique value with a fourth preselected value, said second interpolating step comprising:

a right oblique luminance calculating step for adding, if a result of comparison indicates that a correlation holds in a right oblique direction, the pixel data used to produce the first right oblique correlation value and dividing a resulting sum by a double of a number of pixel data added to thereby produce surrounding pixel data;

a left oblique luminance calculating step for adding, if the result of comparison indicates that a correlation holds in a left oblique direction, the pixel data used to produce the first left oblique correlation value and dividing a resulting sum by a double of a number of pixel data added to thereby produce surrounding pixel data; and

a mean luminance calculating step for adding, if the result of comparison indicates that the first right oblique correlation value and the first left oblique correlation value are smaller than the fourth preselected value, all of the pixel data of the same color used to calculate the correlation values and dividing a resulting sum by a double of a number of pixel data used to thereby produce surrounding pixel data;

a luminance data pattern being generated on the basis of half-value data of the subject pixel data and said surrounding pixel data output by any one of said right oblique luminance calculating step, said left oblique luminance calculating step, and said mean luminance calculating step.

45. A method in accordance with claim 44, wherein said second interpolating step compares, before calculating the luminance data, the first right oblique correlation value and the first left oblique correlation value with the fourth preselected value, and adds a half-value of the subject pixel data and the surrounding pixel data produced by said right oblique luminance calculating step or said left oblique luminance calculating step, and further calculates a second right oblique correlation value and a second left oblique correlation value with differences between the pixel data of a different color obliquely aligning with each other in two directions with the intermediary of said subject pixel data, and compares said second right oblique correlation value and said second left oblique value with a fifth preselected value, said second interpolating step further comprising:

a right oblique luminance calculating step for performing, if a result of comparison indicates that a correlation holds in a right oblique direction, a calculation with the pixel data of different color used to produce the second right oblique correlation value and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data;

a left oblique luminance calculating step for performing, if the result of comparison indicates that a correlation holds in a left oblique direction, a calculation with the pixel data used to produce the second left oblique correlation value and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby. produce surrounding pixel data; and

a mean luminance calculating step for performing, if the result of comparison indicates that the second right oblique correlation value and the second left oblique correlation value both are smaller than the fifth preselected value, a calculation with all of the pixel data of the different color used to produce the second correlation values and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data;

a luminance data pattern being generated by adding a half-value data of the subject pixel data and said surrounding pixel data output by any one of said right oblique luminance calculating step, said left oblique luminance calculating step and said mean luminance calculating step.

46. A method in accordance with claim 44, wherein said second interpolating step compares, before calculating the luminance data, the first right oblique correlation value and the first left oblique correlation value with the fourth preselected value, and adds a half-value of the subject pixel data and the surrounding pixel data produced by said right oblique luminance calculating step or said left oblique luminance calculating step, and further calculates a second right oblique correlation value and a second left oblique correlation value with differences between the pixel data of a different color obliquely aligning with each other in two directions with the intermediary of said subject pixel data, and compares said second right oblique correlation value and said second left oblique value with a fifth preselected value, said second interpolating step further comprising:

a right oblique luminance calculating step for performing, if a result of comparison indicates that a correlation holds in a right oblique direction, a calculation with the pixel data of different color used to produce the second right oblique correlation value and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data;

a left oblique luminance calculating step for performing, if the result of comparison indicates that a correlation holds in a left oblique direction, a calculation with the pixel data used to produce the second left oblique correlation value and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data; and

a mean luminance calculating step for performing, if the result of comparison indicates that the second right oblique correlation value and the second left oblique correlation value both are smaller than the fifth preselected value, a calculation with all of the pixel data of the different color used to produce the second correlation values and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data;

a luminance data pattern being generated by adding a half-value data of the subject pixel data and said surrounding pixel data output by any one of said right oblique luminance calculating step, said left oblique luminance calculating step and said mean luminance calculating step;

said second interpolating step rotating, before the calculation, positions of the pixels by 45°

, then calculating the first and second correlation values in the horizontal and vertical directions, then comparing said first and second correlation values with the preselected values to thereby calculate luminance data, and gain rotating the positions of said pixels to return said luminance data to original positions.

47. A method in accordance with claim 42, wherein the first correlation value in the horizontal direction and the second correlation value in the vertical direction are respectively represented by a sum of an absolute value of a difference between one pixel of different color horizontally aligning with the other pixel of different color with the intermediary of the subject pixel and the luminance data of said subject pixel and an absolute value of a difference between said other pixel data of different color and the luminance data of said subject pixel, and a sum of an absolute value of a difference between one pixel of different color vertically aligning with the other pixel of different color with the intermediary of said subject pixel and the luminance data of said subject pixel and an absolute value of a difference between said other pixel data of different color and the luminance data of said subject pixel.

48. A method in accordance with claim 45, wherein the second right oblique correlation value and the second left oblique correlation value are respectively represented by an absolute value of the pixel data of different color horizontally aligning with each other with the intermediary of the subject pixel and an absolute value of the pixel data of different color vertically aligning with each other with the intermediary of said subject pixel.

49. A method in accordance with claim 42, wherein said second interpolating step compares, before calculating the luminance data, the first right oblique correlation value and the first left oblique correlation value with the fourth preselected value, and adds a half-value of the subject pixel data and the surrounding pixel data produced by said right oblique luminance calculating step or said left oblique luminance calculating step, and further calculates a second right oblique correlation value and a second left oblique correlation value with differences between the pixel data of a different color obliquely aligning with each other in two directions with the intermediary of said subject pixel data, and compares said second right oblique correlation value and said second left oblique value with a fifth preselected value, said second interpolating step further comprising:

a right oblique luminance calculating step for performing, if a result of comparison indicates that a correlation holds in a right oblique direction, a calculation with the pixel data of different color used to produce the second right oblique correlation value and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data;

a left oblique luminance calculating step for performing, if the result of comparison indicates that a correlation holds in a left oblique direction, a calculation with the pixel data used to produce the second left oblique correlation value and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data; and

a mean luminance calculating step for performing, if the result of comparison indicates that the second right oblique correlation value and the second left oblique correlation value both are smaller than the fifth preselected value, a calculation with all of the pixel data of the different color used to produce the second correlation values and dividing a result of calculation by a double of a number of pixel data used for the calculation to thereby produce surrounding pixel data;

a luminance data pattern being generated by adding a half-value data of the subject pixel data and said surrounding pixel data output by any one of said right oblique luminance calculating step, said left oblique luminance calculating step and said mean luminance calculating step;

the second correlation values being calculated by use of the pixel data closer to the subject pixel than the pixel data used to calculate the first correlation values.

50. A method in accordance with claim 39, wherein said plane interpolating step uses the pixel data of the respective colors generated, in said first interpolating step, for the virtual photosensitive cells in the G square lattice and the RB full-checker pattern shifted from said G square by one half of the pixel pitch, and the luminance data generated in said second interpolating step, said plane interpolating step comprising:

a step assigned to plane interpolation of the G pixel data for adding a difference between a mean value of the actually generated G pixel data adjoining a subject pixel to be interpolated in the horizontal and/or the vertical direction and a mean value of the luminance data adjoining said subject pixel in the horizontal and/or the vertical direction and the luminance data corresponding to the position of said subject pixel;

a first R step assigned to plane interpolation of the R pixel data for adding the luminance data of the position of the subject pixel to a difference between a mean value of the actually generated R pixel data obliquely adjoining said subject pixel and a mean value of the luminance data adjoining said subject pixel in a same direction as said actually generated R pixel data;

a second R step assigned to the plane interpolation of the R pixel data for adding the luminance data for remaining R of the subject pixel to a difference between a mean value of the pixel data located an equal distance and produced by said first R step and a mean value of the luminance data located at said equal distance to thereby generate pixel data R for the remaining R;

a third R step assigned to the plane interpolation of the R pixel data for adding the luminance data for remaining R of the subject pixel to a difference between a mean value of the R pixel data closest to the subject pixel and produced by said first and second R steps and a mean value of the luminance data corresponding to the pixels used to produce said mean value to thereby generate pixel data for the remaining R of the subject pixel;

a first B step assigned to plane interpolation of the B pixel data for adding the luminance data of the position of the subject pixel to a difference between a mean value of the actually generated B pixel data obliquely adjoining said subject pixel and a mean value of the luminance data adjoining said subject pixel in a same direction as said actually generated B pixel data;

a second B step assigned to the plane interpolation of the B pixel data for adding the luminance data for remaining B of the subject pixel to a difference between a mean value of the pixel data located an equal distance and produced by said first B step and a mean value of the luminance data located at said equal distance to thereby generate B pixel data for the remaining B; and

a third B step assigned to the plane interpolation of the B pixel data for adding the luminance data for remaining B of the subject pixel to a difference between a mean value of the B pixel data closest to the subject pixel and produced by said first and second B steps and a mean value of the luminance data corresponding to the pixels used to produce said mean value to thereby generate pixel data for the remaining B of the subject pixel.

51. A method in accordance with claim 30, wherein said band broadening step comprises:

an absent data interpolating step for interpolating, by using the pixel data and the luminance data produced by said virtual pixel interpolating step, absence of pixel data and luminance data at positions of the photosensitive cells;

an item-by-item data generating step for generating, by using the pixel data and the luminance data produced by said absent data interpolating step and said virtual pixel interpolating step, a component signal having importance attached to faithful color reproducibility and a component signal having importance attached to resolution in the horizontal and/or the vertical direction; and

an R, G and B matrix step for generating R, G and B by using the luminance data and the chrominance data produced by said item-by-item data generating step.

52. A method in accordance with claim 51, wherein said item-by-item data generating step comprises:

a color reproducibility step for executing the calculation with the pixel data output from the photosensitive cells in accordance with the shifted arrangement, while attaching importance to faithful color reproducibility in the horizontal and/or the vertical direction; and

a resolution step for executing the calculation with the pixel data while attaching importance to resolution in the horizontal and/or the vertical direction.

53. A method in accordance with claim 52, wherein said item-by-item data generating method comprises:

a pseudo-adding step for adding the component signals attaching importance to color reproducibility and resolution, respectively, with respect to a pseudo-frequency; and

a band overlap preventing step for preventing a same frequency band from overlapping when importance is attached to resolution in the horizontal and vertical directions.

54. A method in accordance with claim 53, wherein said pseudo-adding step comprises:

a subtracting step for subtracting a second component signal attaching importance to color reproducibility and lying in a frequency band lower than a frequency band of a first component signal produced by said pixel data generating step, from said frequency band of said first component signal attaching importance to resolution;

a distortion preventing step for executing processing with an output of said subtracting step and said first component signal for obviating aliasing distortions; and

a second adding step for adding resulting outputs of said distortion preventing step.

55. A method in accordance with claim 53, wherein said band overlap preventing step comprises:

a band limiting step for limiting a frequency band of one of a horizontal and a vertical component signal identical with a frequency band of the other component signal; and

an adding step for adding an output of said band limiting step and the other component signal.

56. A method in accordance with claim 31, wherein optics projects an image of a scene at least to two photosensitive surfaces constituted by the photosensitive cells bidimensionally arranged, wherein said two photosensitive surfaces have respective geometric centers shifted from each other by one half of the pixel pitch in a direction of rows and a direction of columns, and wherein said second interpolating step uses pixel data produced by spatially superposing images of the scene incident to said two photosensitive surfaces via filters positioned in front of said photosensitive surfaces in one-to-one correspondence to said photosensitive surfaces for separating incident light into complementary colors or complementary colors and G.