IMAGE DEMOSAICER AND METHOD

• US 20190318451A1
• Filed: 04/11/2018
• Published: 10/17/2019
• Est. Priority Date: 04/11/2018
• Status: Active Grant

• Current Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Original Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Inventors: Swami, Sarvesh, Duan, Yubo, Liu, Chengming
• Granted Patent: US 10,572,974 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

A method for demosaicing a raw image includes: (1) horizontally-interpolating a green channel formed of primary pixel-values B_g(x,y)_g to yield a horizontally-interpolated green channel that includes both B_g(x,y)_g and non-primary pixel-values I_gh(x,y)_r,b; (2) modifying each I_gh(x,y)_r,b, by horizontally-neighboring pixel-values, to yield a refined horizontally-interpolated green channel; (3) vertically-interpolating the green channel to yield a vertically-interpolated green channel that includes pixel-values I_gv(x,y)_r,b; (4) modifying each I_gv(x,y)_r,b by vertically-neighboring pixel-values, to yield a refined vertically-interpolated green channel; (5) generating a full-resolution green channel from the refined interpolated green channels and gradients thereof; (6) generating a full-resolution red channel by determining red pixel-values from a local-red mean value of neighboring pixel-values and the full-resolution green channel; (7) generating a full-resolution blue channel by determining pixel-values from a local-blue mean value of neighboring pixel-values and the full-resolution green channel; and (8) combining the full-resolution channels to yield a demosaiced raw image.

Claims

1. A method for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayer-pattern color-filter array (CFA) having a plurality of red color filters CF_r, green color filters CF_g, and blue color filters CF_b each covering a respective one of a plurality of pixel sub-arrays of the pixel array, the pixel array including a plurality of pixels p(x,y)_α

located at a respective one of a plurality of coordinates (x,y)_α

, and beneath a color filter CF_α

, and each generating a respective one of a plurality of primary pixel-values B_α

(x,y)_α

, where a is one of r, g, and b, the method comprising;

horizontally-interpolating a green channel, of the raw image, formed of primary pixel-values B_g(x,y)_g of the plurality of primary pixel-values B_α

(x,y)_α

, to yield a horizontally-interpolated green channel that includes both primary pixel-values B_g(x,y)_g and non-primary pixel-values I_gh(x,y)_r,b;

modifying each non-primary pixel-value I_gh(x,y)_r,b by a plurality of horizontally-neighboring non-primary pixel-values I_gh(x,y)_r,b, to yield a refined horizontally-interpolated green channel;

vertically-interpolating the green channel to yield a vertically-interpolated green channel that includes both primary pixel-values B_g(x,y)_g and non-primary pixel-values I_gv(x Y)_r,b;

modifying each non-primary pixel-value I_gv(x,y)_r,b by a plurality of vertically-neighboring non-primary pixel-values I_gv(x,y)_r,b, to yield a refined vertically-interpolated green channel;

generating a full-resolution green channel from the refined horizontally-interpolated green channel, refined vertically-interpolated green channel, a horizontal-gradient of the horizontally-interpolated green channel, and a vertical-gradient of the vertically-interpolated green channel;

generating a full-resolution red channel by determining a plurality of non-primary red pixel-values I_r(x_t, y_t)_g,b, each corresponding to a respective one of the plurality of coordinates (x_t, y_t)_g,b, from a local-red mean value of neighboring red primary pixel-values B_r(x,y)_r and the full-resolution green channel;

generating a full-resolution blue channel by determining a plurality of non-primary blue pixel-values I_b(x_t,y_t)_g,r, each corresponding to a respective one of the plurality of coordinates (x_t,y_t)_g,r, from a local-blue mean value of neighboring blue primary pixel-values B_b(x,y)_b and the full-resolution green channel; and

combining the full-resolution red channel, the full-resolution green channel, and the full-resolution blue channel to yield a demosaiced raw image.

2. The method of claim 1, both the horizontally-interpolating the green channel and the vertically-interpolating the green channel comprising linearly interpolating a logarithm of primary pixel-values B_g(x,y)_g.

3. The method of claim 1, each of the plurality of pixel sub-arrays having dimensions m×

n, the step of modifying each non-primary pixel-value I_gh(x,y)_r,b comprising;

for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between (i) a pair of horizontally-neighboring source pixel sub-arrays each beneath respective red color filters, of the CFA and horizontally collinear with the target pixel sub-array,

determining a red horizontal-supplement H_r(x,y) as a linear combination of B_r(x,y)_r, primary pixel-values B_r(x, y₁)_r of horizontally-adjacent pixels p(x, y₁) beneath the target pixel sub-array, and primary pixel-values B_r(x, y₂)_r of horizontally-collinear pixels p(x, y₂) beneath red color filters of the pair of horizontally-neighboring source pixel sub-arrays, where |y−

y₁|<

n and |y−

y₂|≥

n.

4. The method of claim 1, each of the plurality of pixel sub-arrays having dimensions m×

n, the step of modifying each non-primary pixel-value I_gv (x,y)_r,b, comprising;

for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between a pair of vertically-neighboring source pixel sub-arrays each beneath a respective red color filter of the CFA and vertically collinear with the target pixel sub-array,

determining a red vertical-supplement V_r(x,y) as a linear combination of B_r(x,y)_r, primary pixel-values B_r(x₁, y)_r of vertically-adjacent pixels p(x₁, y) beneath the target pixel sub-array, and primary pixel-values B_r(x₂, y)_r of vertically-collinear pixels p(x₂, y) beneath red color filters of the pair of vertically-neighboring source pixel sub-arrays, where |x−

x₁|<

m and |x−

x₂|≥

m.

5. The method of claim 1, further comprising determining the horizontal-gradient and the vertical-gradient by:

subtracting the raw image from the refined horizontally-interpolated green channel, and

subtracting the raw image from the refined vertical-interpolated green channel.

6. The method of claim 1, at least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel including:

binning the full-resolution green channel, and

binning the raw image.

7. The method of claim 6, the at least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel including:

computing a variance of the binned full-resolution green channel and a covariance of (a) the binned full-resolution green channel and (b) the binned raw image.

8. The method of claim 6,

binning of the full-resolution green channel yielding a binned green channel that includes a plurality of binned green pixel-values,

binning the raw image yielding a binned raw image that includes a plurality of binned red pixel-values based on the plurality of primary pixel-values B_r and a plurality of binned blue pixel-values based on the plurality of primary pixel-values B_b, the local-red mean value being an average of binned red pixel-values, the local-blue mean value being an average of binned blue pixel-values.

9. The method of claim 8,

each of the plurality of non-primary red pixel-values I_r(x_t, y_t)_g,b having a magnitude that increases in response to an increasing covariance of the average of binned red pixel-values and an average of binned green pixel-values;

and

each of the plurality of non-primary blue pixel-values I_b (x_t, y_t)_g,r having a magnitude that increases in response to an increasing covariance of the average of binned blue pixel-values and the average of binned green pixel-values.

10. The method of claim 8,

each of the plurality of non-primary red pixel-values I_r(x_t, y_t)_g,b having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values;

and

each of the plurality of non-primary blue pixel-values I_b(x_t,y_t)_g,r having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values.

11. An image demosaicer for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayer-pattern color-filter array (CFA) having a plurality of red color filters CF_r, green color filters CF_g, and blue color filters CF_b each covering a respective one of a plurality of m×

n pixel sub-arrays of the pixel array, the pixel array including a plurality of pixels p(x,y)_α

located at a respective one of a plurality of coordinates (x,y)_α

, and beneath a color filter CF_α

, and each generating a respective one of a plurality of primary pixel-values B_α

(x,y)_α

, where a is one of r, g, and b;

a memory storing non-transitory computer-readable instructions and adapted to store the raw image;

a microprocessor adapted to execute the instructions to;

horizontally-interpolate a green channel, of the raw image, formed of primary pixel-values B_g(x,y)_g of the plurality of primary pixel-values B_α

(x,y)_α

, to yield a horizontally-interpolated green channel that includes both primary pixel-values B_g(x,y)_g and non-primary pixel-values I_gh(x,y)_r,b;

modify each non-primary pixel-value I_gh(x,y)_r,b by a plurality of horizontally-neighboring non-primary pixel-values I_gh(x,y)_r,b, to yield a refined horizontally-interpolated green channel;

vertically-interpolate the green channel to yield a vertically-interpolated green channel that includes both primary pixel-values B_g(x,y)_g and non-primary pixel-values I_g(x,y)_r,b;

modify each non-primary pixel-value I_gv(x,y)_r,b by a plurality of vertically-neighboring non-primary pixel-values I_gv(x,y)_r,b, to yield a refined vertically-interpolated green channel;

generate a full-resolution green channel from the refined horizontally-interpolated green channel, refined vertically-interpolated green channel, a horizontal-gradient of the horizontally-interpolated green channel, and a vertical-gradient of the vertically-interpolated green channel;

generate a full-resolution red channel by determining a plurality of non-primary red pixel-values I_r (x_t, y_t)_g,b, each corresponding to a respective one of the plurality of coordinates (x_t, y_t)_g,b, from a local-red mean value of neighboring red primary pixel-values B_r(x,y)_r and the full-resolution green channel;

generate a full-resolution blue channel by determining a plurality of non-primary blue pixel-values I_b(x_t, y_t)_g,r, each corresponding to a respective one of the plurality of coordinates (x_t, y_t)_g,r, from a local-blue mean value of neighboring blue primary pixel-values B_b (x,y)_b and the full-resolution green channel; and

combine the full-resolution red channel, the full-resolution green channel, and the full-resolution blue channel to yield a demosaiced raw image.

12. The image demosaicer of claim 11, the microprocessor being further adapted to execute the instructions to, when both horizontally-interpolating the green channel and vertically-interpolating the green channel, linearly interpolate a logarithm of primary pixel-values B_g(x,y).

13. The image demosaicer of claim 11, each of the plurality of pixel sub-arrays having dimensions m×

n, when restoring resolution to the horizontally-interpolated green channel and for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between (i) a pair of horizontally-neighboring source pixel sub-arrays each beneath respective red color filters, of the CFA and horizontally collinear with the target pixel sub-array, the microprocessor being further adapted to execute the instructions to;

determine a red horizontal-supplement H_r(x,y) as a linear combination of B_r(x,y)_r, primary pixel-values B_r(x, y₁)_r of horizontally-adjacent pixels p(x, y₁) beneath the target pixel sub-array, and primary pixel-values B_r(x, y₂)_r of horizontally-collinear pixels p(x, y₂) beneath red color filters of the pair of horizontally-neighboring source pixel sub-arrays, where |y−

y₁|<

n and |y−

y₂|≥

n.

14. The image demosaicer of claim 11, each of the plurality of pixel sub-arrays having dimensions m×

n, when restoring resolution to the vertically-interpolated green channel and for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between (i) a pair of vertically-neighboring source pixel sub-arrays each beneath respective red color filters, of the CFA and vertically collinear with the target pixel sub-array, the microprocessor being further adapted to execute the instructions to;

determine a red vertical-supplement V_r(x,y) as a linear combination of B_r(x,y), primary pixel-values B_r(x₁, y) of vertically-adjacent pixels p(x₁, y) beneath the target pixel sub-array, and primary pixel-values B_r(x₂, y) of vertically-collinear pixels p (x₂, y) beneath red color filters of the pair of vertically-neighboring source pixel sub-arrays, where |x−

x₁|<

m and |x−

x₂|≥

m.

15. The image demosaicer of claim 11, the microprocessor being further adapted to execute the instructions to determine the horizontal-gradient and the vertical-gradient by:

subtracting the raw image from the refined horizontally-interpolated green channel; and

subtracting the raw image from the refined vertical-interpolated green channel.

16. The image demosaicer of claim 11, the microprocessor being further adapted to execute the instructions to, when at least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel:

bin the full-resolution green channel, and

bin the raw image.

17. The image demosaicer of claim 16, the microprocessor being further adapted to execute the instructions to, when generating a full-resolution red channel, compute a variance of the binned full-resolution green channel and a covariance of (a) the binned full-resolution green channel and (b) the binned raw image.

18. The image demosaicer of claim 16,

binning of the full-resolution green channel yielding a binned green channel that includes a plurality of binned green pixel-values,

binning the raw image yielding a binned raw image that includes a plurality of binned red pixel-values based on the plurality of primary pixel-values B_r and a plurality of binned blue pixel-values based on the plurality of primary pixel-values B_b, the local-red mean value being an average of binned red pixel-values, the local-blue mean value being an average of binned blue pixel-values.

19. The image demosaicer of claim 18,

each of the plurality of non-primary red pixel-values I_r(x_t, y_t)_g,b having a magnitude that increases in response to an increasing covariance of the average of binned red pixel-values and the average of binned green pixel-values;

and

each of the plurality of non-primary blue pixel-values I_b (x_t, y_t)_g,r having a magnitude that increases in response to an increasing covariance of the average of binned blue pixel-values and the average of binned green pixel-values.

20. The image demosaicer of claim 18,

each of the plurality of non-primary red pixel-values I_r(x_t, y_t)_g,b having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values;

and

each of the plurality of non-primary blue pixel-values I_b(x_t,y_t)_gx having a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values.