Image processing apparatus, imaging device, image processing method, and program for reducing noise or false colors in an image

• US 9,699,429 B2
• Filed: 12/26/2012
• Issued: 07/04/2017
• Est. Priority Date: 03/27/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kaizu, Shun
• Primary Examiner(s): CHEN, CHIA WEI A
• Application Number:
  US14/384,835
• Publication Number:
  US 20150029358A1
• Time in Patent Office: 1,651 Days
• Field of Search: 348276-279
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 2209/045 using mosaic colour filter
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures
• CPC Class Codes: H04N 25/626 Reduction of noise due to residual charges remaining after image readout, e.g. to remove ghost images or afterimages

Abstract

There are provided an apparatus and a method which generate an RGB image having less color noise and fewer false colors by inputting an RGBW image. The apparatus has an image sensor having an RGBW array, and an image processing unit which performs image processing by inputting a sensor image formed of an RGBW pixel signal output from the image sensor. The image sensor has a periodic array of a unit composition formed of each RGBW pixel, and has an array in which composition ratios of each RGB pixel within the unit composition are adapted to be the same as each other. The image processing unit converts a pixel array of the sensor image formed of the RGBW pixel signal, and performs at least either array conversion processing for generating an RGB array image or signal processing for generating each RGB image signal in which all RGB pixel values are set for each pixel position of the sensor image.

Claims

1. An image processing apparatus comprising:

an image sensor that has an RGBW array including R, G and B pixels which are light receiving elements corresponding to wavelength lights of each RGB color and a W pixel which is a light receiving element receiving substantially all RGB wavelength light; and

an image processing unit that performs image processing by inputting a sensor image formed of an RGBW pixel signal output from the image sensor,

wherein the image sensor has a periodic array of a unit composition formed of each RGBW pixel, and has an array in which composition ratios of the R, G and B pixels within the unit composition are the same as each other and wherein each row and column of the periodic array are configured so that one W pixel is interposed between each of the R, G and B pixels and the R, G and B pixels are sequentially arranged in each row and column, and

wherein the image processing unit converts a pixel array of the sensor image formed of the RGBW pixel signal, and performs at least either array conversion processing for generating an RGB array image or signal processing for generating each RGB image signal in which all RGB pixel values are set for each pixel position of the sensor image.

2. The image processing apparatus according to claim 1,

wherein the image processing unit converts the pixel array of the sensor image formed of the RGBW pixel signal, and has an array conversion unit which generates the RGB array image formed of a Bayer array, and

wherein the array conversion unit has a W pixel interpolation unit which generates a whole W pixel image signal by setting the W pixel at an RGB pixel position of the sensor image;

an RGBW pixel generation unit which generates an RGBW corresponding low frequency signal which is a low frequency signal of each RGBW pixel signal at each pixel position of the sensor image; and

a correlation processing unit which inputs the whole W pixel image signal and the RGBW corresponding low frequency signal, and which performs array conversion for a composition pixel of the sensor image and generates an RGB image of the Bayer array by calculating a pixel value based on an estimation in which the W pixel and the R, G and B pixels have a positive correlation.

3. The image processing apparatus according to claim 2,

wherein the W pixel interpolation unit detects an edge direction by applying a pixel value of the W pixel of the sensor image, and determines a W pixel value at an interpolation pixel position by regarding a pixel present in the detected edge direction as a reference pixel.

4. The image processing apparatus according to claim 1,

wherein the image processing unit has a demosaic processing unit which performs demosaic processing on the sensor image and generates each RGB image signal in which all RGB pixel values are set for each pixel position of the sensor image, and

wherein the demosaic processing unit has a W pixel interpolation unit which generates a whole W pixel image signal by setting the W pixel at an RGB pixel position of the sensor image;

an RGBW pixel generation unit which generates an RGBW corresponding low frequency signal which is a low frequency signal of each RGBW pixel signal at each pixel position of the sensor image; and

a correlation processing unit which inputs the whole W pixel image signal and the RGBW corresponding low frequency signal, and which generates each RGB pixel signal in which all RGB pixel values are set for each pixel position of the sensor image, by calculating a pixel value based on an estimation in which the W pixel and the RGB pixel have a positive correlation.

5. The image processing apparatus according to claim 4,

wherein the W pixel interpolation unit detects an edge direction by applying a pixel value of the W pixel of the sensor image, and determines a W pixel value at an interpolation pixel position by regarding a pixel present in the detected edge direction as a reference pixel.

6. The image processing apparatus according to claim 4,

wherein the image processing unit has a linear matrix processing unit which performs pixel value conversion to which a preset matrix is applied, on each RGB image signal generated by the demosaic processing unit.

7. The image processing apparatus according to claim 1,

wherein the image processing unit has a demosaic processing unit which performs demosaic processing on the sensor image and generates each RGBW image signal in which all RGB and W pixel values are set for each pixel position of the sensor image, and

wherein the demosaic processing unit has a W pixel interpolation unit which generates a whole W pixel image signal by setting the W pixel at an RGB pixel position of the sensor image;

an RGBW pixel generation unit which generates an RGBW corresponding low frequency signal which is a low frequency signal of each RGBW pixel signal at each pixel position of the sensor image; and

a correlation processing unit which inputs the whole W pixel image signal and the RGBW corresponding low frequency signal, and which generates each RGB pixel signal in which all RGB pixel values are set for each pixel position of the sensor image, by calculating a pixel value based on an estimation in which the W pixel and the RGB pixel have a positive correlation.

8. The image processing apparatus according to claim 7,

wherein the image processing unit has a linear matrix processing unit which performs pixel value conversion to which a preset matrix is applied, on each RGBW image signal generated by the demosaic processing unit.

9. The image processing apparatus according to claim 1,

wherein the image sensor is configured to output an image having an added pixel value of the same color pixel values which are adjacent to each other in an oblique direction, as the sensor image,

wherein the image processing unit has an array conversion unit which converts a pixel array of the sensor image configured to have the added pixel value and formed of the RGBW pixel signal, and which generates an RGB array image formed of a Bayer array, and

wherein the array conversion unit has an RGBW pixel generation unit which generates an RGBW corresponding low frequency signal which is a low frequency signal of each RGBW pixel signal at each pixel position of the sensor image;

and a correlation processing unit which inputs the whole W pixel image signal output as the sensor image and the RGBW corresponding low frequency signal, and which performs array conversion on a composition pixel of the sensor image and generates an RGB image of the Bayer array, by calculating a pixel value based on an estimation in which the W pixel and the RGB pixel have a positive correlation.

10. The image processing apparatus according to claim 1,

wherein the image sensor is configured to output an image including a long-period exposure pixel and a short-period exposure pixel which are set according to a control of a control unit, as the sensor image,

wherein the image processing unit has an array conversion unit which generates a high dynamic range (HDR) image by applying the sensor image including the long-period exposure pixel and the short-period exposure pixel, and which generates an RGB array HDR image formed of a Bayer array by converting the pixel array, and

wherein the array conversion unit has a W pixel interpolation unit which generates a whole W pixel long-period exposure image in which a long-period exposure W pixel is set for all pixel positions, by applying a long-period exposure pixel signal of the sensor image, and which generates a whole W pixel short-period exposure image in which a short-period exposure W pixel is set for all pixel positions, by applying a short-period exposure pixel signal of the sensor image;

an RGBW pixel generation unit which generates a long-period exposure RGBW corresponding low frequency signal which is a low frequency signal of each long-period exposure RGBW pixel signal at each pixel position, by applying the long-period exposure pixel signal of the sensor image, and which generates a short-period exposure RGBW corresponding low frequency signal which is a low frequency signal of each short-period exposure RGBW pixel signal at each pixel position, by applying the short-period exposure pixel signal of the sensor image;

an HDR composition processing unit which generates a whole W pixel image signal as the high dynamic range (HDR) image and the RGBW corresponding low frequency signal by inputting the whole W pixel long-period exposure image, the whole W pixel short-period exposure image, the long-period exposure RGBW corresponding low frequency signal, and the short-period exposure RGBW corresponding low frequency signal; and

a correlation processing unit which inputs the whole W pixel image signal and the RGBW corresponding low frequency signal as the high dynamic range (HDR) image generated by the HDR composition processing unit, and which performs array conversion on a composition pixel of the sensor image and generates an RGB image of the Bayer array, by calculating a pixel value based on an estimation in which the W pixel and the RGB pixel have a positive correlation.

11. The image processing apparatus according to claim 10,

wherein the HDR composition processing unit calculates a composition pixel value of the high dynamic range (HDR) image through blend processing of the short-period exposure pixel signal and the long-period exposure pixel signal which are obtained after gain adjustment according to an exposure ratio.

12. The image processing apparatus according to claim 10,

wherein the W pixel interpolation unit detects an edge direction by applying the long-period exposure W pixel signal of the sensor image, and generates the whole W pixel long-period exposure image in which the long-period exposure W pixel is set for all pixel positions by determining a W pixel value at an interpolation pixel position by regarding a pixel present in the detected edge direction as a reference pixel, and

wherein the W pixel interpolation unit detects an edge direction by applying the short-period exposure W pixel signal of the sensor image, and generates the whole W pixel short-period exposure image in which the short-period exposure W pixel is set for all pixel positions by determining a W pixel value at an interpolation pixel position by regarding a pixel present in the detected edge direction as a reference pixel.

13. The image processing apparatus according to claim 1,

wherein in the image sensor, two rows of a long-period exposure pixel row and a short-period exposure pixel row are alternately set according to a control of a control unit,

wherein the image sensor is configured to output an image which is formed of a long-period exposure pixel signal having an added pixel value of the same color pixel values which are adjacent to each other in an oblique direction of the long-period exposure pixel row in a unit of two rows, and a short-period exposure pixel signal having an added pixel value of the same color pixel values which are adjacent to each other in an oblique direction of the short-period exposure pixel row in a unit of two rows, as the sensor image,

wherein the image processing unit has an array conversion unit which generates a high dynamic range (HDR) image by applying the sensor image including the long-period exposure pixel and the short-period exposure pixel, and which generates an RGB array HDR image formed of a Bayer array by converting the pixel array, and

wherein the array conversion unit has a W pixel interpolation unit which generates a whole W pixel long-period exposure image in which a long-period exposure W pixel is set for all pixel positions, by applying a long-period exposure pixel signal of the sensor image, and which generates a whole W pixel short-period exposure image in which a short-period exposure W pixel is set for all pixel positions, by applying a short-period exposure pixel signal of the sensor image;

an RGBW pixel generation unit which generates a long-period exposure RGBW corresponding low frequency signal which is a low frequency signal of each long-period exposure RGBW pixel signal at each pixel position, by applying the long-period exposure pixel signal of the sensor image, and which generates a short-period exposure RGBW corresponding low frequency signal which is a low frequency signal of each short-period exposure RGBW pixel signal at each pixel position, by applying the short-period exposure pixel signal of the sensor image;

an HDR composition processing unit which generates a whole W pixel image signal as the high dynamic range (HDR) image and the RGBW corresponding low frequency signal by inputting the whole W pixel long-period exposure image, the whole W pixel short-period exposure image, the long-period exposure RGBW corresponding low frequency signal, and the short-period exposure RGBW corresponding low frequency signal; and

a correlation processing unit which inputs the whole W pixel image signal and the RGBW corresponding low frequency signal as the high dynamic range (HDR) image generated by the HDR composition processing unit, and which performs array conversion on a composition pixel of the sensor image and generates an RGB image of the Bayer array, by calculating a pixel value based on an estimation in which the W pixel and the RGB pixel have a positive correlation.

14. The image processing apparatus according to claim 13,

wherein the HDR composition processing unit calculates a composition pixel value of the high dynamic range (HDR) image through blend processing of the short-period exposure pixel signal and the long-period exposure pixel signal which are obtained after gain adjustment according to an exposure ratio.

15. The image processing apparatus according to claim 13,

wherein the W pixel interpolation unit detects an edge direction by applying the long-period exposure W pixel signal of the sensor image, and generates the whole W pixel long-period exposure image in which the long-period exposure W pixel is set for all pixel positions by determining a W pixel value at an interpolation pixel position by regarding a pixel present in the detected edge direction as a reference pixel, and

wherein the W pixel interpolation unit detects an edge direction by applying the short-period exposure W pixel signal of the sensor image, and generates the whole W pixel short-period exposure image in which the short-period exposure W pixel is set for all pixel positions by determining a W pixel value at an interpolation pixel position by regarding a pixel present in the detected edge direction as a reference pixel.

16. An imaging device comprising:

an RGBW array including R, G and B pixels which are light receiving elements corresponding to wavelength lights of each RGB color and a W pixel which is a light receiving element receiving substantially all RGB wavelength light,

wherein the imaging device has a periodic array of a unit composition formed of each RGBW pixel, and has an array in which composition ratios of the R, G and B pixels within the unit composition are the same as each other and wherein each row and column of the periodic array are configured so that one W pixel is interposed between each of the R, G and B pixels and the R, G and B pixels are sequentially arranged in each row and column.

17. The imaging device according to claim 16,

wherein the imaging device has the periodic array in which 6×

6 pixels are the unit composition, and

wherein the composition ratio of the R, G, B and W pixels within the unit composition is 1;

1;

1;

3.

18. The imaging device according to claim 16,

wherein the imaging device has a composition in which each RGBW pixel is arranged in each column and each row.

19. An image processing method executed in an image processing apparatus,

wherein the image processing apparatus includes an image sensor that has a periodic array of a unit composition formed of each RGBW pixel including R, G and B pixels which are light receiving elements corresponding to wavelength lights of each RGB color and a W pixel which is a light receiving element receiving substantially all RGB wavelength light and wherein each row and column of the periodic array are configured so that one W pixel is interposed between each of the R, G and B pixels and the R, G and B pixels are sequentially arranged in each row and column, and an image processing unit that performs image processing on a sensor image which is an output of the image sensor having an array in which composition ratios of the R, G and B pixels within the unit composition are the same as each other, and

wherein the image processing unit converts a pixel array of the sensor image formed of the RGBW pixel signal, and performs at least either array conversion processing for generating an RGB array image or signal processing for generating each RGB image signal in which all RGB pixel values are set for each pixel position of the sensor image.

20. A non-transitory computer-readable storage device encoded with computer-executable instructions that, when executed by an image processing apparatus, perform image processing,

wherein the image processing apparatus includes an image sensor that has a periodic array of a unit composition formed of each RGBW pixel including R, G and B pixels which are light receiving elements corresponding to wavelength lights of each RGB color and a W pixel which is a light receiving element receiving substantially all RGB wavelength light and wherein each row and column of the periodic array are configured so that one W pixel is interposed between each of the R, G and B pixels and the R, G and B pixels are sequentially arranged in each row and column, and an image processing unit that performs image processing on a sensor image which is an output of the image sensor having an array in which composition ratios of the R, G and B pixels within the unit composition are the same as each other, and

wherein the image processing unit converts a pixel array of the sensor image formed of the RGBW pixel signal, and performs at least either array conversion processing for generating an RGB array image or signal processing for generating each RGB image signal in which all RGB pixel values are set for each pixel position of the sensor image.