IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

• US 20200137368A1
• Filed: 10/21/2019
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: TAOKA, Mineki
• Granted Patent: US 10,812,765 B2
• Time in Patent Office: Days
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/17 Colour separation based on photon absorption depth, e.g. full colour resolution obtained simultaneously at each pixel location
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

An image processing apparatus including a processor is provided. The processor inputs, from an imaging element in which first imaging pixels having a lower SNR and second imaging pixels having a higher SNR are arranged on a same layer, a first captured image by the first imaging pixels and a second captured image by the second imaging pixels when the first imaging pixels and the second imaging pixels perform imaging simultaneously, selects a target pixel from the first captured image, extracts, from the second captured image or an interpolated image of the second captured image, pixels having luminance values close to a luminance value of a pixel corresponding to the target pixel in the second captured image or interpolated image, selects pixels corresponding to the extracted pixels from the first captured image, and corrects a luminance value of the target pixel based on luminance values of the selected pixels.

Claims

1. An image processing apparatus comprising:

processing circuitry configured to,

receive a first captured image and a second captured image simultaneously captured via first imaging pixels and second imaging pixels, respectively, that are one of stacked or are arranged on a same layer of an imaging element, the second imaging pixels having a higher signal-to-noise ratio (SNR) than the first imaging pixels,

select a target pixel from the first captured image, the target pixel in the first captured image having a corresponding pixel in the second captured image or an interpolated image of the second captured image,

extract pixels from the second captured image or the interpolated image of the second captured image to generate extracted pixels such that first luminance values of the extracted pixels are close to a second luminance value of the corresponding pixel,

select pixels corresponding to the extracted pixels from the first captured image as selected pixels, and

correct a luminance value of the target pixel based on luminance values of the selected pixels.

2. The image processing apparatus of claim 1, wherein the first luminance values fall within a set luminance range of the second luminance value.

3. The image processing apparatus of claim 2, wherein the set luminance range is based on a magnitude of the second luminance value.

4. The image processing apparatus of claim 1, wherein the processing circuitry is configured to extract the pixels having the first luminance values by,

extracting the pixels having the first luminance values from a set number of pixels in a vertical direction and a horizontal direction around the pixel corresponding to the target pixel.

5. The image processing apparatus of claim 1, wherein the processing circuitry is configured to calculate an average value of the luminance values of the selected pixels, and

correct the luminance value of the target pixel by a value determined based on the luminance value of the target pixel, the average value, and a set gain.

6. The image processing apparatus of claim 1, wherein

the first imaging pixels are configured to perform imaging in a red wavelength band, a green wavelength band, or a blue wavelength band, and

the second imaging pixels are configured to perform imaging in a near-infrared wavelength band.

7. The image processing apparatus of claim 1, wherein

the first imaging pixels are configured to perform imaging in a green wavelength band or a blue wavelength band, and

the second imaging pixels are configured to perform imaging in a red wavelength band.

8. The image processing apparatus of claim 1, wherein the second imaging pixels are in an upper layer of the imaging element as compared to the first imaging pixels.

9. The image processing apparatus of claim 1, wherein the processing circuitry is further configured to select pixels corresponding to the extracted pixels from an interpolated image of the first captured image.

10. An imaging apparatus comprising:

the imaging element including the first imaging pixels and the second imaging pixels; and

the processing circuitry of claim 1.

11. An image processing method comprising:

receiving a first captured image and a second captured image captured simultaneously via first imaging pixels and second imaging pixels, respectively, stacked or are arranged on a same layer of an imaging element, the second imaging pixels having a higher signal-to-noise ratio (SNR) than the first imaging pixels;

selecting a target pixel from the first captured image, the target pixel in the first captured image having a corresponding pixel corresponding in the second captured image or an interpolated image of the second captured image;

extracting pixels from the second captured image or the interpolated image of the second captured image to generate extracted pixels such that first luminance values of the extracted pixels are close to a second luminance value of the corresponding pixel;

selecting pixels corresponding to the extracted pixels from the first captured image as selected pixels; and

correcting a luminance value of the target pixel based on luminance values of the selected pixels.

12. The image processing method of claim 11, wherein the first luminance values fall within a set luminance range of the second luminance value.

13. The image processing method of claim 12, wherein the set luminance range is based on a magnitude of the second luminance value.

14. The image processing method of claim 11, wherein the extracting comprises:

extracting the pixels having the first luminance values from a set number of pixels in a vertical direction and a horizontal direction around the pixel corresponding to the target pixel.

15. The image processing method of claim 11, wherein the correcting comprises:

calculating an average value of the luminance values of the selected pixels, and

correcting the luminance value of the target pixel by a value determined based on the luminance value of the target pixel, the average value, and a set gain.

16. The image processing method of claim 11, wherein

the first imaging pixels are configured to perform the imaging in a red wavelength band, a green wavelength band, or a blue wavelength band, and

the second imaging pixels are configured to perform the imaging in a near-infrared wavelength band.

17. The image processing method of claim 11, wherein

the first imaging pixels are configured to perform the imaging in a green wavelength band or a blue wavelength band, and

the second imaging pixels are configured to perform the imaging in a red wavelength band.

18. The image processing method of claim 11, wherein the second imaging pixels are in an upper layer of the imaging element as compared to the first imaging pixels.

19. The image processing method of claim 11, further comprising:

selecting pixels corresponding to the extracted pixels from an interpolated image of the first captured image.

20. A non-transitory computer readable medium storing image processing program that, when executed by a computing device, configures the computing device to:

receive a first captured image and a second captured image by simultaneously captured via first imaging pixels and second imaging pixels, respectively, that are one of stacked or are arranged on a same layer of an imaging element, the second imaging pixels having a higher signal-to-noise ratio (SNR) than the first imaging pixels;

select a target pixel from the first captured image, the target pixel in the first captured image having a corresponding pixel corresponding thereto in the second captured image or an interpolated image of the second captured image;

extract pixels from the second captured image or the interpolated image of the second captured image to generate extracted pixels such that the first luminance values of the extracted pixels are close to a second luminance value of the corresponding pixel;

select pixels corresponding to the extracted pixels from the first captured image as selected pixels; and

correct a luminance value of the target pixel based on luminance values of the selected pixels.