SURFACE MESH MATCHING FOR LENS ROLL-OFF CORRECTION

• US 20090043524A1
• Filed: 08/07/2007
• Published: 02/12/2009
• Est. Priority Date: 08/07/2007
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Li, Hsiang-Tsun, Li, Jingqiang, Zhong, Xin, Hung, Szepo Robert
• Granted Patent: US 8,023,758 B2
• Time in Patent Office: Days
• US Class Current: 702/86
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 5/006 Geometric correction correction of chromatic aberrations adapted to be used in an image pickup device containing an electronic image sensor H04N23/10; detecting, correcting, reducing or removing artefacts resulting only from the lens unit, e.g. flare, shading, vignetting or "cos4" H04N25/61
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

A two-dimensional (2D) mesh is applied over a distortion surface to approximate a lens roll-off distortion pattern. The process to apply the 2D mesh distributes a plurality of grid points among the distortion pattern and sub-samples the distortion pattern to derive corrected digital gains at each grid location. Non-grid pixels underlying grid blocks having a grid point at each corner are adjusted based on the approximation of the lens roll-off for the grid points of the grid block. In one example, bilinear interpolation is used. The techniques universally correct lens roll-off distortion irregardless of the distortion pattern shape or type. The technique may also correct for green channel imbalance.

Claims

1. An apparatus comprising:

a processing unit to implement a set of operations to apply a two-dimensional (2D) mesh with a plurality of grid blocks over a distortion surface to approximate a lens roll-off distortion, and to adjust non-grid pixels (NGPs) in the distortion surface underlying a corresponding grid block based on the approximation for the corresponding grid block to correct for the lens roll-off distortion; and

a memory coupled to the processing unit.

2. The apparatus of claim 1, wherein the set of operations which apply the 2D mesh includes a subset of operations to distribute a plurality of grid points among the distortion surface and determine a digital gain correction value to boost a brightness level for each grid point to a brightness level of a bright spot in the distortion surface, each grid block having a set of four grid points (A,B,C and D) located at each corner of the grid block so that adjacent grid blocks share a pair of grid points.

3. The apparatus of claim 2, wherein the set of operations to adjust includes a subset of operations to approximate the lens roll-off distortion of any one pixel in the distortion surface between the set of grid points.

4. The apparatus of claim 3, wherein the subset of operations to approximate include a subset of operations to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=(d2*a+d1*b)/2^M;

V=(d2*c+d1*d)/2^M; and

ICF=(U*e2+V*e1)/2^N

where

d2=d1−

2^M

e2=e1−

2^N

and where M is the number of grid points in a horizontal direction of the 2D mesh;

N is the number of grid points in a vertical direction of the 2D mesh;

2^m is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

2ⁿ is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

e1 is a distance calculated from a NGP to an upper horizontal plane, of the corresponding grid block, having the horizontally aligned grid points A and B;

d1 is calculated from a location of the NGP to a leading vertical plane, of the corresponding grid block, having the vertically aligned grid points A and C; and

a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively.

5. The apparatus of claim 4, wherein the each grid point has associated therewith a digital gain correction value for each of a red (R) color channel, green1 (Gr) color channel, green2 (Gb) color channel, and a blue (B) color channel.

6. The apparatus of claim 3, wherein the subset of operations to approximate include a subset of operations to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=a+d1*(b−

a)/BW;

V=c+d1*(d−

c)/BW; and

ICF=U+(V−

U)*e1/BH,

where BW is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

BH is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

e1 is a distance calculated from a NGP to an upper horizontal plane, of the corresponding grid block, having the horizontally aligned grid points A and B;

d1 is calculated from a location of the NGP to a leading vertical plane, of the corresponding grid block, having the vertically aligned grid points A and C; and

a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively.

7. The apparatus of claim 6, wherein the 2D mesh has a total of W grid blocks in a horizontal direction and a horizontal index j and a vertical index are computed as

j=x/2^M, and

i=y/2^N;

and the digital gain correction values are defined as

a=LUT[i*W+j],

b=LUT[i*W+j+1],

c=LUT[(i+1)*W+j] and

d=LUT[(i+1)*W+j+1],

where LUT[ ] is a location in a mesh lookup table having associated therewith a respective one digital gain correction value a, b, c or d for the grid points A, B, C and D, respectively;

M is the number of grid points in a horizontal direction of the 2D mesh; and

N is the number of grid points in a vertical direction of the 2D mesh.

8. The apparatus of claim 3, wherein the subset of operations to approximate include a subset of operations to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=a+accumulate(Delta_Table[Yblock,Xblock]);

V=c+accumulate(Delta_Table[Yblock+1,Xblock]); and

ICF=U+(V−

U)*accumulate(Delta_Height),

where

$\mathrm{delta\_table}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}\right]=\left(\frac{\mathrm{CF}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}+1\right]-\mathrm{CF}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}\right]}{\mathrm{Block\_width}}\right)$

and where the Block_width is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

Block_Height is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

Delta_Table[Yblock, Xblock] represents a correction factor (CF) change slope on a top boundary edge of a block index (Yblock, Xblock);

the top boundary edge is a horizontal line between two grid points, the grid point A to the grid point B;

the Delta_Height equals 1/Block_Height;

the accumulate(Delta_Height) is a function for calculating a summation of N Delta_Heights;

N is the number of grid points in a vertical direction of the 2D mesh; and

the a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively

9. The apparatus of claim 1, wherein the processing unit is a portion of a cellular phone, wireless device, wireless communications device, a video game console, a personal digital assistant (PDA), a laptop computer, or an audio/video-enabled device.

10. An integrated circuit comprising:

a processing unit to implement a set of operations to apply a two-dimensional (2D) mesh with a plurality of grid blocks over a distortion surface to approximate a lens roll-off distortion, and to adjust non-grid pixels (NGPs) in the distortion surface underlying a corresponding grid block based on the approximation for the corresponding grid block to correct for the lens roll-off distortion; and

a memory coupled to the processing unit.

11. The integrated circuit of claim 10, wherein the set of operations which apply the 2D mesh includes a subset of operations to distribute a plurality of grid points among the distortion surface and determine a digital gain correction value to boost a brightness level for each grid point to a brightness level of a bright spot in the distortion surface, each grid block having a set of four grid points (A,B,C and D) located at each corner of the grid block to that adjacent grid blocks share a pair of grid points.

12. The integrated circuit of claim 11, wherein the set of operations to adjust further includes a subset of operations to approximate the lens roll-off distortion of any one pixel in the distortion surface between the set of grid points.

13. The integrated circuit of claim 12, wherein the subset of operations to approximate include a subset of operations to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=(d2*a+d1*b)/2^M;

V=(d2*c+d1*d)/2^M; and

ICF=(U*e2+V*e1)/2^N

where

d2=d1−

2^M

e2=e1−

2^N

and where M is the number of grid points in a horizontal direction of the 2D mesh;

N is the number of grid points in a vertical direction of the 2D mesh;

2^m is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

2ⁿ is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

e1 is a distance calculated from a NGP to an upper horizontal plane, of the corresponding grid block, having the horizontally aligned grid points A and B;

d1 is calculated from a location of the NGP to a leading vertical plane, of the corresponding grid block, having the vertically aligned grid points A and C; and

a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively.

14. The integrated circuit of claim 13, wherein the each grid point has associated therewith a digital gain correction value for each of a red (R) color channel, green1 (Gr) color channel, green2 (Gb) color channel, and a blue (B) color channel.

15. The integrated circuit of claim 12, wherein the subset of operations to approximate include a subset of operations to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=a+d1*(b−

a)/BW;

V=c+d1*(d−

c)/BW; and

ICF=U+(V−

U)*e1/BH,

where BW is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

BH is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

e1 is a distance calculated from a NGP to an upper horizontal plane, of the corresponding grid block, having the horizontally aligned grid points A and B;

d1 is calculated from a location of the NGP to a leading vertical plane, of the corresponding grid block, having the vertically aligned grid points A and C; and

a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively.

16. The integrated circuit of claim 15, wherein the 2D mesh has a total of W grid blocks in a horizontal direction and a horizontal index j and a vertical index are computed as

j=x/2^M, and

i=y/2^N;

and the digital gain correction values are defined by

a=LUT[i*W+j],

b=LUT[i*W+j+1],

c=LUT[(i+1)*W+j] and

d=LUT[(i+1)*W+j+1],

where LUT[ ] is a location in a mesh lookup table having associated therewith a respective one digital gain correction value a, b, c or d for the grid points A, B, C and D, respectively;

M is the number of grid points in a horizontal direction of the 2D mesh; and

N is the number of grid points in a vertical direction of the 2D mesh.

17. The integrated circuit of claim 12, wherein the subset of operations to approximate include a set of operations to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=a+accumulate(Delta_Table[Yblock,Xblock]);

V=c+accumulate(Delta_Table[Yblock+1,Xblock]); and

ICF=U+(V−

U)*accumulate(Delta_Height)

where

$\mathrm{delta\_table}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}\right]=\left(\frac{\mathrm{CF}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}+1\right]-\mathrm{CF}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}\right]}{\mathrm{Block\_width}}\right)$

and where the Block_width is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

Block_Height is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

Delta_Table[Yblock, Xblock] represents a correction factor (CF) change slope on a top boundary edge of a block index (Yblock, Xblock);

the top boundary edge is a horizontal line between two grid points, the grid point A to the grid point B;

the Delta_Height equals 1/Block_Height;

the accumulate(Delta_Height) is a function for calculating a summation of N Delta_Heights;

N is the number of grid points in a vertical direction of the 2D mesh; and

the a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively

18. The integrated circuit of claim 10, wherein the processing unit is a portion of a cellular phone, wireless device, wireless communications device, a video game console, a personal digital assistant (PDA), a laptop computer, or an audio/video-enabled device.

19. A computer program product including a computer readable medium having instructions for causing a computer to:

apply a two-dimensional (2D) mesh with a plurality of grid blocks over a distortion surface to approximate a lens roll-off distortion; and

adjust non-grid pixels (NGPs) in the distortion surface underlying a corresponding grid block based on the approximation for the corresponding grid block to correct for the lens roll-off distortion.

20. The computer program product of claim 19, wherein the instructions to apply the 2D mesh includes instructions to:

distribute a plurality of grid points among the distortion surface; and

determine a digital gain correction value to boost a brightness level for each grid point to a brightness level of a bright spot in the distortion surface;

wherein each grid block has a set of four grid points (A, B, C and D) located at each corner of the grid block and adjacent grid blocks share a pair of grid points.

21. The computer program product of claim 20, wherein the instructions to adjust further includes instructions to approximate the lens roll-off distortion of any one pixel in the distortion surface between the set of grid points.

22. The computer program product of claim 21, wherein the instructions to approximate include instructions to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=(d2*a+d1*b)/2^M;

V=(d2*c+d1*d)/2^M; and

ICF=(U*e2+V*e1)/2^N

where

d2=d1−

2^M

e2=e1−

2^N

and where M is the number of grid points in a horizontal direction of the 2D mesh;

N is the number of grid points in a vertical direction of the 2D mesh;

2^m is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

2ⁿ is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

e1 is a distance calculated from a NGP to an upper horizontal plane, of the corresponding grid block, having the horizontally aligned grid points A and B;

d1 is calculated from a location of the NGP to a leading vertical plane, of the corresponding grid block, having the vertically aligned grid points A and C; and

a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively.

23. The computer program product of claim 22, wherein the each grid point has associated therewith a digital gain correction value for each of a red (R) color channel, green1 (Gr) color channel, green2 (Gb) color channel, and a blue (B) color channel.

24. The computer program product of claim 21, wherein the instructions to approximate include instructions to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=a+d1*(b−

a)/BW;

V=c+d1*(d−

c)/BW; and

ICF=U+(V−

U)*e1/BH,

where BW is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

BH is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

e1 is a distance calculated from a NGP to an upper horizontal plane, of the corresponding grid block, having the horizontally aligned grid points A and B;

d1 is calculated from a location of the NGP to a leading vertical plane, of the corresponding grid block, having the vertically aligned grid points A and C; and

a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively.

25. The computer program product of claim 24, wherein the 2D mesh has a total of W grid blocks in a horizontal direction and a horizontal index j and a vertical index are computed as

j=x/2^M, and

i=y/2^N;

and the digital gain correction values are defined by

a=LUT[i*W+j],

b=LUT[i*W+j+1],

c=LUT[(i+1)*W+j] and

d=LUT[(i+1)*W+j+1],

where LUT[ ] is a location in a mesh lookup table having associated therewith a respective one digital gain correction value a, b, c or d for the grid points A, B, C and D, respectively;

M is the number of grid points in a horizontal direction of the 2D mesh; and

N is the number of grid points in a vertical direction of the 2D mesh.

26. The computer program product of claim 21, wherein the instructions to approximate include instructions to perform bilinear interpolation to calculate a set of interpolation factors (U,V) and an interpolated correction factor (ICF) for a respective one non-grid pixel (NGP) in the corresponding grid block defined by

U=a+accumulate(Delta_Table[Yblock,Xblock]);

V=c+accumulate(Delta_Table[Yblock+1,Xblock]); and

ICF=U+(V−

U)*accumulate(Delta_Height),

where

$\mathrm{delta\_table}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}\right]=\left(\frac{\mathrm{CF}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}+1\right]-\mathrm{CF}<br><br>\left[\mathrm{Yblock},\mathrm{Xblock}\right]}{\mathrm{Block\_width}}\right)$

and where the Block_width is a distance between horizontally aligned grid points A and B and grid points C and D of the corresponding grid block;

Block_Height is a distance between vertically aligned grid points A and C and grid points B and D of the corresponding grid block;

Delta_Table[Yblock, Xblock] represents a correction factor (CF) change slope on a top boundary edge of a block index (Yblock, Xblock);

the top boundary edge is a horizontal line between two grid points, the grid point A to the grid point B;

the Delta_Height equals 1/Block_Height;

the accumulate(Delta_Height) is a function for calculating a summation of N Delta_Heights;

N is the number of grid points in a vertical direction of the 2D mesh; and

the a, b, c and d are the digital gain correction values for the grid points A, B, C and D, respectively.

27. The computer program product of claim 21, wherein the instructions to adjust simultaneously adjust for green channel imbalance.

28. A device comprising:

applying means for applying a two-dimensional (2D) mesh with a plurality of grid blocks over a distortion surface to approximate a lens roll-off distortion; and

adjusting means for adjusting non-grid pixels (NGPs) in the distortion surface underlying a corresponding grid block based on the approximation for the corresponding grid block to correct for the lens roll-off distortion.

29. The device of claim 28, wherein the applying means includes distributing means for distributing a plurality of grid points among the distortion surface;

and determining means for determining a digital gain correction value to boost a brightness level for each grid point to a brightness level of a bright spot in the distortion surface;

wherein each grid block has a set of four grid points (A, B, C and D) located at each corner of the grid block and adjacent grid blocks share a pair of grid points.

30. The device of claim 29, wherein the adjusting means includes approximating means for approximating the lens roll-off distortion of any one pixel in the distortion surface between the set of grid points.

31. The device of claim 30, wherein the approximating means simultaneously adjusting for green channel imbalance.

32. A processor comprising:

applying means for applying a two-dimensional (2D) mesh with a plurality of grid blocks over a distortion surface to approximate a lens roll-off distortion; and

adjusting means for adjusting non-grid pixels (NGPs) in the distortion surface underlying a corresponding grid block based on the approximation for the corresponding grid block to correct for the lens roll-off distortion.

33. The processor of claim 32, wherein the applying means includes distributing means for distributing a plurality of grid points among the distortion surface;

and determining means for determining a digital gain correction value to boost a brightness level for each grid point to a brightness level of a bright spot in the distortion surface;

wherein each grid block has a set of four grid points (A, B, C and D) located at each corner of the grid block and adjacent grid blocks share a pair of grid points.

34. The processor of claim 33, wherein the adjusting means further includes approximating means for approximating the lens roll-off distortion of any one pixel in the distortion surface between the set of grid points.

35. The processor of claim 34, wherein the approximating means simultaneously adjusting for green channel imbalance.

36. A method comprising:

applying a two-dimensional (2D) mesh with a plurality of grid blocks over a distortion surface to approximate a lens roll-off distortion; and

adjusting non-grid pixels (NGPs) in the distortion surface underlying a corresponding grid block based on the approximation for the corresponding grid block to correct for the lens roll-off distortion.

37. The method of claim 36, wherein the applying includes distributing a plurality of grid points among the distortion surface;

and determining a digital gain correction value to boost a brightness level for each grid point to a brightness level of a bright spot in the distortion surface;

wherein each grid block has a set of four grid points (A, B, C and D) located at each corner of the grid block and adjacent grid blocks share a pair of grid points.

38. The method of claim 37, wherein the adjusting includes approximating the lens roll-off distortion of any one pixel in the distortion surface between the set of grid points.

39. The method of claim 38, wherein simultaneously with the approximating, further comprising adjusting for green channel imbalance.