IMAGE DEMOSAICER AND METHOD

0Associated
Cases 
0Associated
Defendants 
0Accused
Products 
0Forward
Citations 
0
Petitions 
1
Assignment
First Claim
1. A method for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayerpattern colorfilter array (CFA) having a plurality of red color filters CF_{r}, green color filters CF_{g}, and blue color filters CF_{b }each covering a respective one of a plurality of pixel subarrays of the pixel array, the pixel array including a plurality of pixels p(x,y)_{α }
 located at a respective one of a plurality of coordinates (x,y)_{α}, and beneath a color filter CF_{α}, and each generating a respective one of a plurality of primary pixelvalues B_{α}(x,y)_{α}, where a is one of r, g, and b, the method comprising;
horizontallyinterpolating a green channel, of the raw image, formed of primary pixelvalues B_{g}(x,y)_{g }of the plurality of primary pixelvalues B_{α}(x,y)_{α}, to yield a horizontallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gh}(x,y)_{r,b};
modifying each nonprimary pixelvalue I_{gh}(x,y)_{r,b }by a plurality of horizontallyneighboring nonprimary pixelvalues I_{gh}(x,y)_{r,b}, to yield a refined horizontallyinterpolated green channel;
verticallyinterpolating the green channel to yield a verticallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gv}(x Y)_{r,b};
modifying each nonprimary pixelvalue I_{gv}(x,y)_{r,b }by a plurality of verticallyneighboring nonprimary pixelvalues I_{gv}(x,y)_{r,b}, to yield a refined verticallyinterpolated green channel;
generating a fullresolution green channel from the refined horizontallyinterpolated green channel, refined verticallyinterpolated green channel, a horizontalgradient of the horizontallyinterpolated green channel, and a verticalgradient of the verticallyinterpolated green channel;
generating a fullresolution red channel by determining a plurality of nonprimary red pixelvalues I_{r}(x_{t}, y_{t})_{g,b}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,b}, from a localred mean value of neighboring red primary pixelvalues B_{r}(x,y)_{r }and the fullresolution green channel;
generating a fullresolution blue channel by determining a plurality of nonprimary blue pixelvalues I_{b}(x_{t},y_{t})_{g,r}, each corresponding to a respective one of the plurality of coordinates (x_{t},y_{t})_{g,r}, from a localblue mean value of neighboring blue primary pixelvalues B_{b}(x,y)_{b }and the fullresolution green channel; and
combining the fullresolution red channel, the fullresolution green channel, and the fullresolution blue channel to yield a demosaiced raw image.
1 Assignment
0 Petitions
Accused Products
Abstract
A method for demosaicing a raw image includes: (1) horizontallyinterpolating a green channel formed of primary pixelvalues B_{g}(x,y)_{g }to yield a horizontallyinterpolated green channel that includes both B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gh}(x,y)_{r,b}; (2) modifying each I_{gh}(x,y)_{r,b}, by horizontallyneighboring pixelvalues, to yield a refined horizontallyinterpolated green channel; (3) verticallyinterpolating the green channel to yield a verticallyinterpolated green channel that includes pixelvalues I_{gv}(x,y)_{r,b}; (4) modifying each I_{gv}(x,y)_{r,b }by verticallyneighboring pixelvalues, to yield a refined verticallyinterpolated green channel; (5) generating a fullresolution green channel from the refined interpolated green channels and gradients thereof; (6) generating a fullresolution red channel by determining red pixelvalues from a localred mean value of neighboring pixelvalues and the fullresolution green channel; (7) generating a fullresolution blue channel by determining pixelvalues from a localblue mean value of neighboring pixelvalues and the fullresolution green channel; and (8) combining the fullresolution channels to yield a demosaiced raw image.
0 Citations
No References
No References
20 Claims
 1. A method for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayerpattern colorfilter array (CFA) having a plurality of red color filters CF_{r}, green color filters CF_{g}, and blue color filters CF_{b }each covering a respective one of a plurality of pixel subarrays of the pixel array, the pixel array including a plurality of pixels p(x,y)_{α }
 located at a respective one of a plurality of coordinates (x,y)_{α}, and beneath a color filter CF_{α}, and each generating a respective one of a plurality of primary pixelvalues B_{α}(x,y)_{α}, where a is one of r, g, and b, the method comprising;
horizontallyinterpolating a green channel, of the raw image, formed of primary pixelvalues B_{g}(x,y)_{g }of the plurality of primary pixelvalues B_{α}(x,y)_{α}, to yield a horizontallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gh}(x,y)_{r,b}; modifying each nonprimary pixelvalue I_{gh}(x,y)_{r,b }by a plurality of horizontallyneighboring nonprimary pixelvalues I_{gh}(x,y)_{r,b}, to yield a refined horizontallyinterpolated green channel; verticallyinterpolating the green channel to yield a verticallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gv}(x Y)_{r,b}; modifying each nonprimary pixelvalue I_{gv}(x,y)_{r,b }by a plurality of verticallyneighboring nonprimary pixelvalues I_{gv}(x,y)_{r,b}, to yield a refined verticallyinterpolated green channel; generating a fullresolution green channel from the refined horizontallyinterpolated green channel, refined verticallyinterpolated green channel, a horizontalgradient of the horizontallyinterpolated green channel, and a verticalgradient of the verticallyinterpolated green channel; generating a fullresolution red channel by determining a plurality of nonprimary red pixelvalues I_{r}(x_{t}, y_{t})_{g,b}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,b}, from a localred mean value of neighboring red primary pixelvalues B_{r}(x,y)_{r }and the fullresolution green channel; generating a fullresolution blue channel by determining a plurality of nonprimary blue pixelvalues I_{b}(x_{t},y_{t})_{g,r}, each corresponding to a respective one of the plurality of coordinates (x_{t},y_{t})_{g,r}, from a localblue mean value of neighboring blue primary pixelvalues B_{b}(x,y)_{b }and the fullresolution green channel; and combining the fullresolution red channel, the fullresolution green channel, and the fullresolution blue channel to yield a demosaiced raw image.  View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
 located at a respective one of a plurality of coordinates (x,y)_{α}, and beneath a color filter CF_{α}, and each generating a respective one of a plurality of primary pixelvalues B_{α}(x,y)_{α}, where a is one of r, g, and b, the method comprising;
 11. An image demosaicer for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayerpattern colorfilter array (CFA) having a plurality of red color filters CF_{r}, green color filters CF_{g}, and blue color filters CF_{b }each covering a respective one of a plurality of m×
 n pixel subarrays of the pixel array, the pixel array including a plurality of pixels p(x,y)_{α} located at a respective one of a plurality of coordinates (x,y)_{α}, and beneath a color filter CF_{α}, and each generating a respective one of a plurality of primary pixelvalues B_{α}(x,y)_{α}, where a is one of r, g, and b;
a memory storing nontransitory computerreadable instructions and adapted to store the raw image; a microprocessor adapted to execute the instructions to; horizontallyinterpolate a green channel, of the raw image, formed of primary pixelvalues B_{g}(x,y)_{g }of the plurality of primary pixelvalues B_{α}(x,y)_{α}, to yield a horizontallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gh}(x,y)_{r,b}; modify each nonprimary pixelvalue I_{gh}(x,y)_{r,b }by a plurality of horizontallyneighboring nonprimary pixelvalues I_{gh}(x,y)_{r,b}, to yield a refined horizontallyinterpolated green channel; verticallyinterpolate the green channel to yield a verticallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{g}(x,y)_{r,b}; modify each nonprimary pixelvalue I_{gv}(x,y)_{r,b }by a plurality of verticallyneighboring nonprimary pixelvalues I_{gv}(x,y)_{r,b}, to yield a refined verticallyinterpolated green channel; generate a fullresolution green channel from the refined horizontallyinterpolated green channel, refined verticallyinterpolated green channel, a horizontalgradient of the horizontallyinterpolated green channel, and a verticalgradient of the verticallyinterpolated green channel; generate a fullresolution red channel by determining a plurality of nonprimary red pixelvalues I_{r }(x_{t}, y_{t})_{g,b}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,b}, from a localred mean value of neighboring red primary pixelvalues B_{r}(x,y)_{r }and the fullresolution green channel; generate a fullresolution blue channel by determining a plurality of nonprimary blue pixelvalues I_{b}(x_{t}, y_{t})_{g,r}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,r}, from a localblue mean value of neighboring blue primary pixelvalues B_{b }(x,y)_{b }and the fullresolution green channel; and combine the fullresolution red channel, the fullresolution green channel, and the fullresolution blue channel to yield a demosaiced raw image.  View Dependent Claims (12, 13, 14, 15, 16, 17, 18, 19, 20)
 n pixel subarrays of the pixel array, the pixel array including a plurality of pixels p(x,y)_{α} located at a respective one of a plurality of coordinates (x,y)_{α}, and beneath a color filter CF_{α}, and each generating a respective one of a plurality of primary pixelvalues B_{α}(x,y)_{α}, where a is one of r, g, and b;
1 Specification
Many consumer electronics products include at least one camera. These products include tablet computers, mobile phones, and smart watches. In such products, and in digital still cameras themselves, images and video captured in lowlight environments suffer from low signaltonoise ratio, which limits image and video quality. Highdynamic range images also face this problem, for example, in lowluminosity regions of the image.
Embodiments disclosed herein ameliorate said image quality problems resulting from relatively low illumination. In one aspect, a method for demosaicing a raw image captured by an image sensor is disclosed. The image sensor includes a pixel array and a Bayerpattern colorfilter array (CFA) having a plurality of red color filters CF_{r}, green color filters CF_{g}, and blue color filters CF_{b }each covering a respective one of a plurality of pixel subarrays of the pixel array. The pixel array including a plurality of pixels p(x,y)_{α }located at a respective one of a plurality of coordinates (x,y)_{α}, and beneath a color filter CF_{r}, and each generating a respective one of a plurality of primary pixelvalues B_{α}(x,y)_{α}, where a is one of r, g, and b. The method includes the following steps denoted 18:
 (1) horizontallyinterpolating a green channel, of the raw image, formed of primary pixelvalues B_{g}(x,y)_{g }of the plurality of primary pixelvalues B_{α}(x,y)_{α}, to yield a horizontallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gh}(x,y)_{r,b};
 (2) modifying each nonprimary pixelvalue I_{gh}(x,y)_{r,b }by a plurality of horizontallyneighboring nonprimary pixelvalues I_{gh}(x,y)_{r,b}, to yield a refined horizontallyinterpolated green channel;
 (3) verticallyinterpolating the green channel to yield a verticallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gv}(x,y)_{r,b};
 (4) modifying each nonprimary pixelvalue I_{gv}(x,y)_{r,b }by a plurality of verticallyneighboring nonprimary pixelvalues I_{gv}(x,y)_{r,b}, to yield a refined verticallyinterpolated green channel;
 (5) generating a fullresolution green channel from the refined horizontallyinterpolated green channel, refined verticallyinterpolated green channel, a horizontalgradient of the horizontallyinterpolated green channel, and a verticalgradient of the verticallyinterpolated green channel;
 (6) generating a fullresolution red channel by determining a plurality of nonprimary red pixelvalues I_{r}(x_{t}, y_{t})_{g,b}, each corresponding to a respective one of the plurality of coordinates (x_{t},y_{t})_{g,b}, from a localred mean value of neighboring red primary pixelvalues B_{r}(x,y)_{b }and the fullresolution green channel;
 (7) generating a fullresolution blue channel by determining a plurality of nonprimary blue pixelvalues I_{b}(x_{t}, y_{t})_{g,r}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,r}, from a localblue mean value of neighboring blue primary pixelvalues B_{b}(x,y)_{b }and the fullresolution green channel; and
 (8) combining the fullresolution red channel, the fullresolution green channel, and the fullresolution blue channel to yield a demosaiced raw image.
In a second aspect, an image demosaicer for demosaicing a raw image includes a memory and a microprocessor. The raw image is captured by the image sensor of the aforementioned method. The memory stores nontransitory computerreadable instructions and is adapted to store the raw image. The microprocessor is adapted to execute the instructions to execute steps 18 of the aforementioned method.
Each pixel 132 has a respective pixel charge corresponding to a respective intensity of light from a scene 120 imaged onto pixel array 132A. Circuitry 138 converts each pixel charge to a respective one of a first plurality of pixelvalues of a raw image 190, which is stored in memory 104. Camera 100 may include a display 139 configured to display a demosaiced version of raw image 190 as a demosaiced image 198.
In scene 120, indoor lighting, not shown, illuminates the front of person 121 facing the camera while sunlight illuminates sunny scene 123. Hence, person 121 and sunny scene 123 have respective differing luminosities. Since the direct sunlight is typically significantly brighter than the indoor lighting, luminosity of sunny scene 123 far exceeds luminosity of person 121 such that scene 120 has a high dynamic range of luminosity. Standard digital imaging enables capture of scene 120 using a single exposure time optimized for either (but not both) luminosity of person 121 or sunny scene 123. Even when the exposure time is optimized for luminosity of person 121, the exposure time may be too long to capture of clear image of person 121, e.g., if person 121 is moving.
Image sensor 200 and CFA 240 are positioned with respect to an xy coordinate system 209. Coordinates (x,y) denote locations of each color filter and pixel therebeneath, where the origin of coordinate system 209 (x=y=0) may correspond to any pixel within pixel array 132A. Herein, all coordinates are expressed as (x,y) coordinates, that is, with the x value being listed first. Also, herein, the notation α(i,j) refers to a color filter at coordinates (i,j), where α denotes a transmission spectrum of a color filter and i and j are integers. For example, α is one of r, g, b, c, m, y, and k which represent, respectively, red, green, blue, cyan, magenta, yellow, and clear (panchromatic) color filters. A subscript on a coordinate pair indicates the type color filter, of CFA 240, covering a pixel 132 located at the coordinate. For example, pixel p(x,y)_{α} denotes a pixel 132, located at coordinate (x,y), that is covered by a color filter with a transmission spectrum denoted by a. The pixel p(x,y)_{α} generates a primary pixelvalue B_{α}(x,y).
CFA 240 includes a plurality of red (R), green (G), and blue (B) color filters arranged in a fourcell Bayer pattern. In a conventional Bayerpattern CFA aligned with a pixel array, each color filter is aligned with a respective pixel of the pixel array. A conventional Bayer pattern may be viewed as a “singlecell Bayer pattern.” In a CFA having a fourcell Bayer pattern, a fourpixel subarray of the pixel array may be beneath one color filter of the same color.
In CFA 240, each color filter may be aligned with a twobytwo pixel subarray of pixel array 132A. For example, pixel array 132A includes a pixel subarray 236 formed of four pixels 132 located at a respective one of four coordinates (5,5), (6,5), (5, 6), and (6,6) hereinafter coordinates C_{1}. CFA 240 includes a blue color filter aligned to pixels 132 that constitute pixel subarray 236. This blue color filter may be a single continuous optical element, or include a plurality of optical elements, such as four distinct optical elements arranged in a twobytwo array. Similarly, CFA 240 includes red and green color filters each aligned with a respective twobytwo pixel subarray of pixel array 132A.
Each primary pixelvalue B_{r }is generated by a respective pixel 132 beneath a red color filter of CFA 140. Each primary pixelvalue B_{y }is generated by a respective pixel 132 beneath a green color filter of CFA 140. Each primary pixelvalue B_{b }is generated by a respective pixel 132 beneath a blue color filter of CFA 140. For example, primary pixelvalues B_{r}, B_{g}, and B_{b }at respective coordinates (3,4), (3,6), and (5,6) are generated by respective pixels of image sensor 200 at the same respective coordinates (3,4), (3,6), and (5,6). Expressed more concisely, and introducing a notational convention used herein, primary pixelvalues B_{r}(3,4), B_{b}(3,6), and B_{g}(5,6) are generated by respective pixels p(3,4)_{r}, p(3,6)_{g}, and p(5,6)_{b }of image sensor 200.
Image demosaicer 700 includes a microprocessor 702 and a memory 704 that stores software 720 that includes computerreadable instructions. Microprocessor 702 may be a digital signal processor such as an image processor. Memory 704 may be transitory and/or nontransitory and may include one or both of volatile memory (e.g., SRAM, DRAM, VRAM, or any combination thereof) and nonvolatile memory (e.g., FLASH, ROM, magnetic media, optical media, or any combination thereof).
Memory 704 and microprocessor 702 may function as memory 104 and microprocessor 102, respectively, of camera 100,
Raw image 790 includes color channels 791, 792, and 793. Examples of color channels 791, 792, and 793 are, respectively, red channel 400, green channel 500, and blue channel 600. Software 720 includes one or more of the following software modules that when executed by microprocessor 702 produce respective data outputs: a greenchannel interpolator 722, a resolution restorer 723, a gradient filter 724, a combiner 727, a redchannel interpolator 728, a bluechannel interpolator 729, and a combiner 732.
Greenchannel interpolator 722 is configured to interpolate color channel 792 to yield a horizontallyinterpolated green channel 742H and verticallyinterpolated green channel 742V. Resolution restorer 723 produces a refined green channel 743H and a refined green channel 743V from, respectively, horizontallyinterpolated green channel 742H and verticallyinterpolated green channel 742V. Gradient filter 724 produces a gradient image 744H and a gradient image 744V from, respectively, horizontallyinterpolated green channel 742H and verticallyinterpolated green channel 742V. Gradient images 744H and 744V may be obtained by applying a gradient filter to channels 743H and 743V, respectively. Alternatively, gradient images 744H and 744V may be obtained by applying a gradient filter to channels 742H and 742V, respectively. Combiner 727 produces a fullresolution green channel 747 from refined green channels 743H and 743V. Redchannel interpolator 728 generates fullresolution red channel from fullresolution green channel 747 and color channel 791. Bluechannel interpolator 729 generates fullresolution blue channel from fullresolution green channel 747 and color channel 793. Combiner 732 combines fullresolution channels 747, 748, and 79 to form demosaiced image 798.
Luminance Interpolation
Interpolated channel 900 includes a plurality of primary pixelvalues B_{g}(x,y)_{g }and a plurality of green pixelvalues 901 (x,y)_{r}, and a plurality of green pixelvalues 903 (x,y)_{b}. Green pixelvalues 901 (x,y), and 903 (x,y)_{b }are interpolated from verticallyadjacent primary pixelvalues B_{g}(x,y)_{g}. For example, green pixelvalue 903(6,5)_{b }is interpolated from at least two primary pixelvalues B_{g }(x≠6,5)_{g}. Green pixelvalues 901 (x,y)_{r }and 903 (x,y)_{b }are each examples of nonprimary pixelvalues because they are part of interpolated green channel 900 at coordinates (x,y) beneath nongreen color filters.
Both interpolated channels 800 and 900 may be logarithmically interpolated from green channel 500. For example, the logarithms of interpolated pixelvalues 801, 803, 901, and 903 may be interpolated from logarithms of primary pixelvalues B_{g}(x,y)_{g }of green channel 500. Logarithmic interpolation enables accurate interpolation without requiring information about white balance and illumination associated with raw image 390. Interpolating logarithms of pixelvalues also more heavily weights smaller pixelvalues than does interpolating pixelvalues themselves, which contributes to accurate gradient detection unperturbed by highpixelvalue noise.
Refined green channel 1000 includes the following green pixelvalues: primary pixelvalues B_{g}(x,y)_{g }of raw image 390, a plurality of green pixelvalues 1001(x,y)_{r}, and a plurality of green pixelvalues 1003 (x,y)_{b}. Each green pixelvalue 1001 (x,y), may be equal to a sum of green pixelvalue 801(x,y)_{r }and a red horizontalsupplement H_{r}(x,y), as shown by equation 1. Each green pixelvalue 1003 (x,y)_{b }may be equal to a sum of green pixelvalue 803 (x,y)_{b }and a blue horizontalsupplement H_{b }(x,y), as shown by equation 2.
1001(x,y)_{r}=801(x,y)_{r}+H_{r}(x,y) (1)
1003(x,y)_{b}=803(x,y)_{b}+H_{b}(x,y) (2)
Refined green channel 1100 includes the following green pixelvalues: plurality of primary pixelvalues B_{g}(x,y)_{g }of raw image 390, a plurality of green pixelvalues 1101(x,y)_{r}, and a plurality of green pixelvalues 1103(x,y)_{b}. Each green pixelvalue 1101(x,y)_{r }may be equal to a sum of green pixelvalue 901(x,y)_{r }and a red verticalsupplement V_{r}(x,y), as shown by equation 3. Each green pixelvalue 1103(x,y)_{b }may be equal to a sum of green pixelvalue 903 (x,y)_{b }and a blue verticalsupplement V_{b}(x,y), as shown by equation 4.
1101(x,y)_{r}=901(x,y)_{r}+V_{r}(x,y) (3)
1103(x,y)_{b}=903(x,y)_{b}+V_{b}(x, (4)
Coordinate (x,y) of horizontalsupplements H(x,y) and verticalsupplements V (x,y) corresponds to an m×n pixel subarray of pixel array 132A, discussed above regarding image sensor 200 and CFA 240. For example,
As
 (a) B_{r}(x,y),
 (b) primary pixelvalues B_{r}(x, y_{1}) of horizontallyadjacent pixels p(x, y_{1}) beneath the target pixel subarray, and
 (c) primary pixelvalues B_{r}(x, y_{2}) of horizontallycollinear pixels p(x, y_{2}) beneath red color filters of the pair of neighboring source pixel subarrays.
Horizontal coordinates y_{1 }and y_{2 }may satisfyy−y_{1}<n and y−y_{2}≥n.
The following example of horizontalsupplement H_{r}(x,y) applies to coordinate (x_{1}, y_{1})_{r}, which corresponds to pixel p(x_{1}, y_{1})_{r }beneath a red color filter of CFA 240. Horizontalsupplement H_{r}(x_{1}, y_{1}), may be expressed as:
H_{r}(x_{1},y_{1})_{r}=a_{0}B_{r}(x_{1},y_{1})_{r}−a_{1}B_{r}(x_{1},y_{1}−4)r−a_{2}B_{r}(x_{1},y_{1}−3)_{r}−a_{3}B_{r}(x_{1},y_{1}+1)_{r}−a_{4}B_{r}(x_{1},y_{1}+4)_{r}−a_{5}B_{r}(x_{1},y_{1}+5)_{r}.
Green pixelvalue 1001(x_{1}, y_{1})_{r }may equal the sum of green pixelvalue 801(x_{1}, y_{1})_{r }and horizontalsupplement H_{r}(x_{1},y_{1})_{r}. Coefficients a_{05 }are, for example: a_{0}=4, a_{14}=1, a_{5}=0.
As
(a) B_{r}(x,y),
(b) primary pixelvalues B_{r}(x_{1}, y) of verticallyadjacent pixels p(x_{1}, y) beneath the target pixel subarray, and
(c) primary pixelvalues B_{r}(x_{2}, y) of verticallycollinear pixels p(x_{2}, y) beneath red color filters of the pair of neighboring source pixel subarrays.
Vertical coordinates x_{1 }and x_{2 }may satisfyx−x_{1}<m and x−x_{2}≥m.
The following example of verticalsupplement V_{r}(x,y) applies to coordinate (x_{1},y_{1})_{r}. Verticalsupplement V_{r}(x_{1},y_{1})_{r }may be expressed as:
V_{r}(x_{1},y_{1})_{r}=c_{0}B_{r}(x_{1},y_{1})_{r}−c_{1}B_{r}(x_{1}−5,y_{1})_{r}−c_{2}B_{r}(x_{1}−4,y_{1})_{r}−c_{3}B_{r}(x_{1}−1,y_{1})_{r}−c_{4}B_{r}(x_{1}+3,y_{1})_{r}−c_{5}B_{r}(x_{1}+4,y_{1})_{r }
Green pixelvalue 1101(x_{1}, y_{1})_{r }may equal the sum of green pixelvalue 901(x_{1}, y_{1})_{r }and verticalsupplement V_{r}(x_{1},y_{1})_{r}. Coefficients c_{05 }are, for example: c_{0}=4, c_{1}=0, and c_{25}=1.
Without departing from the scope hereof, Supplements H_{r}(x,y) and V_{r}(x,y) may include nonzero terms for pixelvalues B_{r}(x,y)_{r }further away from coordinate (x_{1}, y_{1}) than those indicated above.
As
 (a) B_{b }(x,y)
 (b) primary pixelvalues B_{b }(x, y_{1}) of horizontallyadjacent pixels p(x, y_{1}) beneath the target pixel subarray, and
 (c) primary pixelvalues B_{b }(x, y_{2}) of horizontallycollinear pixels p(x, y_{2}). Horizontal coordinates y_{1 }and y_{2 }may satisfyy−y_{1}<n and y−y_{2}≥n.
The following example of horizontalsupplement H_{b }(x,y) applies to coordinate (x_{2}, y_{2})_{b}, which corresponds to pixel p(x_{2}, y_{2})_{b }beneath a blue color filter of CFA 240. Horizontalsupplement H_{b }(x_{2}, y_{2})_{b }may be expressed as:
H_{b}(x_{2},y_{2})_{b}=a_{0}B_{b}(x_{2},y_{2})_{b}−a_{1}B_{b}(x_{2},y_{2}−5)_{b}−a_{2}B_{b}(x_{2},y_{2}−4)_{b}−a_{3}B_{b}(x_{2},y_{2}−1)_{b}−a_{4}B_{b}(x_{2},y_{2}+3)_{b}−a_{5}B_{b}(x_{2},y_{2}+4)_{b}.
Green pixelvalue 1001(x_{2}, y_{2})_{b }may equal the sum of green pixelvalue 803 (x_{2}, y_{2})_{b }and horizontalsupplement H_{b}(x_{2}, y_{2})_{b}. Coefficients a_{05 }are, for example: a_{0}=4, a_{14}=1, a_{5}=0.
As
 (a) B_{b}(x, y),
 (b) primary pixelvalues B_{b }(x_{1}, y) of verticallyadjacent pixels p(x_{1}, y) beneath the target pixel subarray, and
 (c) primary pixelvalues B_{b }(x_{2}, y) of verticallycollinear pixels p(x_{2}, y) beneath blue color filters of the pair of neighboring source pixel subarrays.
Vertical coordinates x_{1 }and x_{2 }satisfyx−x_{1}<m and x−x_{2}≥m.
The following example of verticalsupplement V_{b }(x,y) applies to coordinate (x_{2}, y_{2})_{b}. Verticalsupplement V_{b}(x_{2}, y_{2})_{b }may be expressed as:
V_{b}(x_{2},y_{2})_{b}=c_{0}B_{b}(x_{2},y_{2})_{b}−c_{1}B_{b}(x_{2}−5,y_{2})_{b}−c_{2}B_{b}(x_{2}−y_{2})_{b}−c_{3}B_{b}(x_{2}+1,y_{2})_{b}−c_{4}B_{b}(x_{2}+3,y_{2})_{b}−c_{5}B_{b}(x_{2}+4,y_{2})_{b}.
Green pixelvalue 1101(x_{2}, y_{2})_{b }may equal the sum of green pixelvalue 901(x_{2}, y_{2})_{b }and verticalsupplement V_{b }(x_{2}, y_{2})_{b}. Coefficients c_{05 }are, for example: c_{0}=4, c_{1}=0, and c_{25}=1.
Gradient images 1600 and 1700 may be spatiallyaveraged, e.g., via a convolution filter, over a group of pixels such that its resolution is less than resolutions of channel 1000, 1100, and raw image 390. For example, gradient images 1600 and 1700 may be averaged over a weighted or unweighted q_{x}×q_{y }kernel, where each of q_{x }and q_{y }exceed both m and n, where m×n is the size of the pixel subarray beneath a same color filter in image sensor 200. For example,
1600(x,y)=1001−B_{r}_{L}+B_{g}−B_{r}_{G}+1003−B_{b})_{L}+B_{g}−B_{b}_{G}, (5)
where each righthandside term is evaluated at coordinate (x,y), ·_{L }and ·_{G }denote, respectively, a local average within the kernel, e.g., kernel 1610, and a global average over the entire raw image 390. Equations 1 and 2 are example expressions of pixelvalues 1001 and 1003. Similarly, an average verticalgradient ∇_{v }of gradient image 1700 at coordinate (x,y) may be expressed by equation (6).
1700(x,y)=1101−B_{r}_{L}+B_{g}−B_{r}_{G}+1103−B_{b}_{L}+B_{g}−B_{b}_{G}. (6)
In equation (6), each righthandside term is evaluated at coordinate (x,y). Equations 3 and 4 are example expressions of pixelvalues 1101 and 1103.
Gradient filter 724 may also suppress noise in gradient images 744H and 744V. For example, gradient filter may reduce pixel values of gradient images 1600 and 1700 at coordinates (x,y) where gradient images 1600 and 1700 differ by less than a predetermined threshold. Such small differences denote noise, and hence do not contribute to accurate gradient values.
At pixel coordinates (x,y)_{g }corresponding to green color filters of CFA 240, fullresolution green channel 1800 has pixelvalues equal to primary pixelvalues B_{g}. For example, at coordinates (x,y)=(4,2), pixelvalue 1802(4,2)=B_{g}(4,2).
Chrominance Interpolation
Binned raw image 1900 includes a plurality of binned pixelvalues: binned red pixelvalues 1901, binned green pixelvalues 1902, and binned blue pixelvalues 1903. Each binned pixelvalue may be an average of pixelvalues of raw image 390 beneath a pixel subarray of pixel array 132A. For example: binned red pixelvalue 1901(1) is an average of red primary pixelvalues B_{r }at coordinates (3,3), (3,4), (4,3), and (4,4); binned green pixelvalue 1902(1) is an average of green primary pixelvalues B_{y }at coordinates (3,5), (3,6), (4,5), and (4,6); binned blue pixelvalue 1903(1) is an average of blue primary pixelvalues B_{b }at coordinates (5,5), (5,6), (6,5), and (6,6).
Binned green channel 2000 includes a plurality of binned green pixelvalues 2001, 2002, and 2003 generated by pixelsubarray, of pixel array 132A, located beneath red, green, and blue color filters of CFA 240, respectively. Each binned pixelvalue 20012003 may be an average of pixelvalues of fullresolution green channel 1800 beneath a pixel subarray of pixel array 132A. For example: binned pixelvalue 2001(1) is an average of pixelvalues 1802(3,3), 1802(4,4) and primary pixelvalues B_{g}(3,4) and B_{g}(4,3); binned green pixelvalue 2002(1) is an average of green primary pixelvalues B_{g}(3,5), B_{g}(3,6), and pixelvalues 1802(4,5) and 1802(4,6); and binned blue pixelvalue 2003(1) is an average of blue primary pixelvalues B_{b }at coordinates pixelvalues 1802(5,5), 1802(5,6) and primary pixelvalues B_{g}(6,5), and B_{g}(6,6).
The following examples illustrate computation of a blue pixelvalue at a coordinate beneath a red color filter and red pixelvalue at a coordinate beneath a blue color filter. The red color filter is located at (x,y)=(7,7); the blue color filter is located at (x,y)=(5,5). Pixelvalues at these coordinates are enclosed by a dashed box in
At pixel coordinate (x,y)=(5,5), demosaiced image 2100 includes red pixelvalue I_{r}(5,5), green pixelvalue I_{g}(5,5), blue primary pixelvalue B_{b}(5,5). Green pixelvalue I_{g}(5,5) equals pixelvalue 1802(5,5) of fullresolution green channel 1800,
Red pixelvalue I_{r}(5,5) may be computed by chrominance interpolation equations (21r1) through (21r8). Per equation (21r1), R_{avg }is an average of binned red pixelvalues 1901(14) of binned raw image 1900. Per equation (21r2), G_{avg }is an average of binned pixelvalues 2001(14) of binned green channel 2000, which correspond to pixels 132 beneath a green color filter of CFA 240. Per equation (21r3), RG_{avg }is an average of an addendbyaddend multiplication of R_{avg }and G_{avg}. Per equation (21r4), GG_{avg }is an average of an addendbyaddend multiplication of G_{avg }and G_{avg}. Equations (21r5) through (21r7) derive a weight based on a covariance and variance between R_{avg }and G_{avg}.
Redchannel interpolator 728 may generate red pixelvalues I_{r}(x,y) for all coordinates (x,y)_{α≠r }of demosaiced image 2100, where red pixelvalues I_{r}(x,y) comprise an example of fullresolution red channel 748.
At pixel coordinate (x,y)=(7,7), demosaiced image 2100 includes red primary pixelvalue B_{r}(7,7) of raw image 390, green pixelvalue I_{g}(7,7), and blue pixelvalue I_{b}(7,7). Green pixelvalue I_{g}(7,7) equals pixelvalue 1802(7,7) of fullresolution green channel 1800,
Blue pixelvalue I_{b}(7,7) may be computed by chrominance interpolation equations (21b1) through (21b8), e.g., by bluechannel interpolator 729 of image demosaicer 700. Per equation (21b1), B_{avg }is an average of binned blue pixelvalues 1903(14) of binned raw image 1900. Per equation (21b2), G_{avg }is an average of binned pixelvalues 2003(14) of binned green channel 2000, which correspond to pixels 132 beneath a green color filter of CFA 240. Per equation (21b3), BG_{avg }is an average of an addendbyaddend multiplication of B_{avg }and G_{avg}. Per equation (21b4), GG_{avg }is an average of an addendbyaddend multiplication of G_{avg }and G_{avg}. Equations (21b5) through (21b8) derive a weight based on a covariance and variance between B_{avg }and G_{avg}.
Bluechannel interpolator 729 may generate blue pixelvalues I_{b}(x,y) for all coordinates (x,y)_{α≠b }of demosaiced image 2100, where blue pixelvalues I_{b}(x,y) comprise an example of fullresolution blue channel 749.
In an embodiment, the weight of equations (21r8) and (21b8) are determined at least in part by applying a guided filter to fullresolution green channel 1800 and raw image 390, and/or images derived therefrom. The guided filter may be applied to fullresolution green channel 1800 and red channel 400 and blue channel 600 of raw image 390, for example, such that weights are derived from a correlation between channels 1800, 400, and 600. Use of a guided filter reduces color aliasing that may arise during chrominance interpolation. A guided filter is, for example, one described in K. He, J. Sun and X. Tang, “Guided Image Filtering,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 13971409, June 2013.
Step 2210H includes horizontallyinterpolating a green channel, of the raw image, formed of primary pixelvalues B_{g}(x,y)_{g }of the plurality of primary pixelvalues B_{α}(x,y)_{α}, to yield a horizontallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gh}(x,y)_{r,b}. In an example of step 2210H, greenchannel interpolator 722 interpolates green channel 500 to yield horizontallyinterpolated green channel 800. For example, greenchannel interpolator 722 may interpolate a logarithm of green channel 500 to yield a logarithm of channel 800, and also apply an inverse logarithmic function thereto to yield horizontallyinterpolated green channel 800
Step 2210V includes verticallyinterpolating a green channel, of the raw image, formed of primary pixelvalues B_{g}(x,y)_{g }of the plurality of primary pixelvalues B_{α}(x,y)_{α}, to yield a verticallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gv}(x,y)_{r,b }In an example of step 2210V, greenchannel interpolator 722 interpolates green channel 500 to yield verticallyinterpolated green channel 900. For example, greenchannel interpolator 722 may interpolate a logarithm of green channel 500 to yield a logarithm of channel 900, and also apply an inverse logarithmic function thereto to yield verticallyinterpolated green channel 900.
Step 2220H includes restoring resolution to the horizontallyinterpolated green channel by modifying each nonprimary pixelvalue I_{gh}(x,y)_{r,b }by a plurality of horizontallyneighboring nonprimary pixelvalues I_{gh}(x,y)_{r,b}, to yield a refined horizontallyinterpolated green channel. In an example of step 2220H, resolution restorer 723 generates refined horizontallyinterpolated green channel 1000 from horizontallyinterpolated green channel 800.
Step 2220V includes restoring resolution to the verticallyinterpolated green channel by modifying each nonprimary pixelvalue I_{gh}(x,y)_{r,b }by a plurality of verticallyneighboring nonprimary pixelvalues I_{gh}(x,y)_{r,b}, to yield a refined verticallyinterpolated green channel. In an example of step 2220V, resolution restorer 723 generates refined verticallyinterpolated green channel 1100 from verticallyinterpolated green channel 900.
Step 2230 includes generating a fullresolution green channel from the horizontallyinterpolated green channel, verticallyinterpolated green channel, a horizontalgradient of the horizontallyinterpolated green channel, and a verticalgradient of the verticallyinterpolated green channel. In an example of step 2230, combiner 727 generates fullresolution green channel 1800 from refined horizontallyinterpolated green channel 1000 and refined verticallyinterpolated green channel 1100.
Step 2240R includes generating a fullresolution red channel by determining a plurality of nonprimary red pixelvalues I_{r}(x_{t}, y_{t})_{g,b}, each corresponding to a respective one of the plurality of coordinates (x_{t},y_{t})_{g,b, }from a localred mean value of neighboring red primary pixelvalues B_{r}(x,y)_{r }and the fullresolution green channel. In an example of step 2240H, redchannel interpolator 728 generates red pixelvalues I_{r}(x,y) of demosaiced image 2100, for example, by executing one or more of equations (21r1) (21r8). As previously stated, an example of fullresolution red channel 748 includes red pixelvalues I_{r}(x,y).
Step 2240B includes generating a fullresolution blue channel by determining a plurality of nonprimary blue pixelvalues I_{b }(x_{t}, y_{t})_{g,r}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,r}, from a localblue mean value of neighboring blue primary pixelvalues B_{b}(x,y)_{b }and the fullresolution green channel. In an example of step 2240B, bluechannel interpolator 729 generates blue pixelvalues I_{b}(x,y) of demosaiced image 2100, for example, by executing one or more of equations (21b1) (21b8). As previously stated, an example of fullresolution blue channel 749 includes blue pixelvalues I_{b}(x,y).
Step 2250 includes combining the fullresolution red channel, the fullresolution green channel, and the fullresolution blue channel to yield a demosaiced raw image. In an example of step 2250, combiner 732 combines fullresolution red channel 748, fullresolution green channel 1800, and fullresolution blue channel 749 to yield demosaiced image 2100.
Combinations of Features
Features described above as well as those claimed below may be combined in various ways without departing from the scope hereof. The following examples illustrate some possible, nonlimiting combinations:
(A1) denotes a method for demosaicing a raw image captured by an image sensor. The image sensor includes a pixel array and a Bayerpattern colorfilter array (CFA) having a plurality of red color filters CF_{r}, green color filters CF_{g}, and blue color filters CF_{b }each covering a respective one of a plurality of pixel subarrays of the pixel array. The pixel array including a plurality of pixels p(x,y)_{α} located at a respective one of a plurality of coordinates (x,y)_{α}, and beneath a color filter CF_{α}, and each generating a respective one of a plurality of primary pixelvalues B_{α}(x,y)_{α}, where a is one of r, g, and b. Method (A1) includes the following steps denoted 18:
 (1) horizontallyinterpolating a green channel, of the raw image, formed of primary pixelvalues B_{g}(x,y)_{g }of the plurality of primary pixelvalues B_{α}(x,y)_{α}, to yield a horizontallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gh}(x,y)_{r,b};
 (2) modifying each nonprimary pixelvalue I_{gh}(x,y)_{r,b }by a plurality of horizontallyneighboring nonprimary pixelvalues I_{gh}(x,y)_{r,b}, to yield a refined horizontallyinterpolated green channel;
 (3) verticallyinterpolating the green channel to yield a verticallyinterpolated green channel that includes both primary pixelvalues B_{g}(x,y)_{g }and nonprimary pixelvalues I_{gv}(x,y)_{r,b};
 (4) modifying each nonprimary pixelvalue I_{gv }(x,y)_{r,b }by a plurality of verticallyneighboring nonprimary pixelvalues I_{gv}(x,y)_{r,b}, to yield a refined verticallyinterpolated green channel;
 (5) generating a fullresolution green channel from the refined horizontallyinterpolated green channel, refined verticallyinterpolated green channel, a horizontalgradient of the horizontallyinterpolated green channel, and a verticalgradient of the verticallyinterpolated green channel;
 (6) generating a fullresolution red channel by determining a plurality of nonprimary red pixelvalues I_{r}(x_{t}, y_{t})_{g,b}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,b}, from a localred mean value of neighboring red primary pixelvalues B_{r}(x,y)_{r }and the fullresolution green channel;
 (7) generating a fullresolution blue channel by determining a plurality of nonprimary blue pixelvalues I_{b}(x_{t}, y_{t})_{gx}, each corresponding to a respective one of the plurality of coordinates (x_{t}, y_{t})_{g,r}, from a localblue mean value of neighboring blue primary pixelvalues B_{b}(x,y)_{b }and the fullresolution green channel; and
 (8) combining the fullresolution red channel, the fullresolution green channel, and the fullresolution blue channel to yield a demosaiced raw image.
(A2) In any method denoted by (A1), both the horizontallyinterpolating the green channel and the verticallyinterpolating the green channel may include linearly interpolating a logarithm of primary pixelvalues B_{g}(x,y)_{g}.
(A3) In any method denoted by one of (A1) and (A2), in which each of the plurality of pixel subarrays having dimensions m×n, the step of modifying each nonprimary pixelvalue I_{gh}(x,y)_{r,b }may include, for each pixel p(x,y) beneath a red color filter of a target pixel subarray, of the plurality of pixel subarrays, located between (i) a pair of horizontallyneighboring source pixel subarrays each beneath respective red color filters, of the CFA and horizontally collinear with the target pixel subarray: determining a red horizontalsupplement H_{r}(x,y) as a linear combination of B_{r}(x,y)_{r}, primary pixelvalues B_{r}(x, y_{1})_{r }of horizontallyadjacent pixels p(x, y_{1}) beneath the target pixel subarray, and primary pixelvalues B_{r}(x, y_{2})_{r }of horizontallycollinear pixels p(x, y_{2}) beneath red color filters of the pair of horizontallyneighboring source pixel subarrays, where y−y_{1}<n and y−y_{2}≥n.
(A4) In any method denoted by one of (A1) through (A3), in which each of the plurality of pixel subarrays having dimensions m×n, the step of modifying each nonprimary pixelvalue I_{gv}(x,y)_{r,b }may include, for each pixel p(x,y) beneath a red color filter of a target pixel subarray, of the plurality of pixel subarrays, located between a pair of verticallyneighboring source pixel subarrays each beneath a respective red color filter of the CFA and vertically collinear with the target pixel subarray: determining a red verticalsupplement V_{r}(x,y) as a linear combination of B_{r}(x,y)_{r}, primary pixelvalues B_{r}(x_{1}, y)_{r }of verticallyadjacent pixels p(x_{1},y) beneath the target pixel subarray, and primary pixelvalues B_{r}(x_{2}, y)_{r }of verticallycollinear pixels p(x_{2}, y) beneath red color filters of the pair of verticallyneighboring source pixel subarrays, where x−x_{1}<m and x−x_{2}≥m.
(A5) Any method denoted by one of (A1) through (A4) may further include determining the horizontalgradient and the verticalgradient by: (a) subtracting the raw image from the refined horizontallyinterpolated green channel, and (b) subtracting the raw image from the refined verticalinterpolated green channel.
(A6) In any method denoted by one of (A1) through (A5), at least one of (i) generating the fullresolution red channel and (ii) generating the fullresolution blue channel may include binning the fullresolution green channel, and binning the raw image.
(A7) In any method denoted by (A6), the at least one of (i) generating the fullresolution red channel and (ii) generating the fullresolution blue channel may include: computing a variance of the binned fullresolution green channel and a covariance of (a) the binned fullresolution green channel and (b) the binned raw image.
(A8) In any method denoted by one of (A6) and (A7): (a) binning of the fullresolution green channel may yield a binned green channel that includes a plurality of binned green pixelvalues; and (b) binning the raw image may yield a binned raw image that includes a plurality of binned red pixelvalues based on the plurality of primary pixelvalues B_{r }and a plurality of binned blue pixelvalues based on the plurality of primary pixelvalues B_{b}, the localred mean value being an average of binned red pixelvalues, the localblue mean value being an average of binned blue pixelvalues.
(A9) In any method denoted by (A8), each of the plurality of nonprimary red pixelvalues I_{r}(x_{t}, y_{t})_{g,b }may have a magnitude that increases in response to an increasing covariance of the average of binned red pixelvalues and an average of binned green pixelvalues; and each of the plurality of nonprimary blue pixelvalues I_{b}(x_{t},y_{t})_{g,r }may have a magnitude that increases in response to an increasing covariance of the average of binned blue pixelvalues and the average of binned green pixelvalues.
(A10) In any method denoted by one of (A8) and (A9), each of the plurality of nonprimary red pixelvalues I_{r}(x_{t}, y_{t})_{g,b }may have a magnitude that decreases in response to an increasing variance of the average of binned green pixelvalues; and each of the plurality of nonprimary blue pixelvalues I_{b}(x_{t},y_{t})_{g,r }may have a magnitude that decreases in response to an increasing variance of the average of binned green pixelvalues.
An image demosaicer (B1) for demosaicing a raw image includes a memory and a microprocessor. The raw image is captured by the image sensor of the aforementioned method. The memory stores nontransitory computerreadable instructions and is adapted to store the raw image. The microprocessor is adapted to execute the instructions to execute any of the methods denoted by (A1) through (A10).
Changes may be made in the above image demosaicer and demosaicing methods without departing from the scope hereof. It should thus be noted that the matter contained in the above description or shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. The following claims are intended to cover all generic and specific features described herein, as well as all statements of the scope of the present image demosaicer and demosaicing methods, which, as a matter of language, might be said to fall therebetween.