Lens shading calibration for cameras

• US 9,451,187 B2
• Filed: 12/31/2012
• Issued: 09/20/2016
• Est. Priority Date: 12/31/2012
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Cohen, Noy, Motta, Ricardo J.
• Primary Examiner(s): Tran, Nhan T
• Assistant Examiner(s): NGUYEN, CHAN T H
• Application Number:
  US13/732,211
• Publication Number:
  US 20140184813A1
• Time in Patent Office: 1,359 Days
• Field of Search: 348/251, 348/187
• US Class Current: 1/1
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

Embodiments of the present invention are directed to methods and systems for performing automatic lens shading correction using module-specific calibrations. According to one aspect of the invention, a method is provided that is performed over three main stages. During a first stage, radial symmetric component data is determined that is common to camera modules of the type to be calibrated. During the second stage, the actual measurement of the shading profile of one or more specific camera modules is performed. In the third and final stage is the extrapolation stage, the base measurement surfaces of a camera module type determined in the first stage are extrapolated and modified with the module-specific Bezier correction surface and calibration data of the second stage. The output surfaces of this third and final stage are used to correct for the shading profile in the camera module, depending on the light-source estimation.

Claims

1. A method for performing lens shading correction, the method comprising:

generating a plurality of base surface measurements of a camera module type;

receiving a shading profile specific to each camera module of the camera module type, the shading profile comprising a radial component and a non-radial component;

calculating a module-specific calibration surface based on a corresponding shading profile; and

creating a module-specific corrected surface measurement for each illuminant of the plurality of illuminants based on the plurality of base surface measurements of the camera module type and the module-specific calibration surface,

wherein, each camera module of the plurality of camera modules is operable to be used in a mobile computing device,

further wherein the plurality of base surface measurements are generated based on a plurality of spectral scans and a plurality of images under a plurality of simulated conditions based on a target illuminant and a calibration illuminant.

2. The method according to claim 1, further comprising:

capturing a scene with a first camera module of the plurality of camera modules;

determining an illuminant in the scene captured by the first camera module; and

applying the module-specific corrected surface measurement corresponding to the determined illuminant to image data corresponding to the scene.

3. The method according to claim 1, wherein the generating the base surface measurement is performed offline and independently of the measuring a shading profile specific to each camera module of the camera module type.

4. The method according to claim 1, wherein the base surface measurement is stored in a memory, the memory being accessible to a camera driver executing in a processor and configured to interact with the camera module.

5. The method according to claim 1, wherein the radial component of the shading profile comprises at least one of:

a plurality of relative illumination characteristics of a lens comprised in the camera module;

a profile of a chief ray angle of a sensor comprised in the camera module;

a spectral response of an infrared filter comprised in the camera module.

6. The method according to claim 1, wherein the generating the base surface measurement further comprises at least one of:

obtaining a plurality of camera modules of the camera module type;

determining a calibration illumination data;

determining a target illumination data for the plurality of illuminants;

generating the plurality of spectral scans;

generating a configuration for each camera module based on a spectral scan;

generating the plurality of images under the plurality of simulated conditions, the plurality of simulated conditions being based on the target illumination data and the calibration illumination data; and

averaging the plurality of images of the plurality of camera modules.

7. The method according to claim 6, wherein the generating the plurality of spectral scans of the camera module comprises scanning the plurality of camera modules with a monochromator.

8. The method according to claim 6, wherein the plurality of images are stored as a raw image format (RAW) file.

9. The method according to claim 6, wherein the plurality of spectral scans comprises a spectral scan of a calibration light source used to measure the plurality of camera modules during a factory-floor calibration.

10. The method according to claim 9, wherein the plurality of spectral scans comprises a spectral scan of each illuminant of the plurality of illuminants.

11. The method according to claim 10, wherein the creating a module-specific corrected surface measurement comprises:

receiving the module-specific calibration surface;

receiving a base surface measurement data corresponding to the calibration illuminant;

receiving the base surface measurement data corresponding to a target illuminant; and

calculating the module-specific corrected surface measurement for the target illuminant based on the module-specific calibration surface, the base surface measurement data corresponding to the calibration illuminant, and the base surface measurement data corresponding to the target illuminant.

12. The method according to claim 11, wherein the module-specific calibration surface comprises a Bezier shading correction surface measurement, further wherein measuring a shading profile specific to each camera module of the camera module type comprises generating the Bezier shading correction surface for the camera module under the calibration illuminant.

13. The method according to claim 12, wherein the measuring a shading profile specific to each camera module is performed during a factory-floor calibration of the camera module.

14. The method according to claim 12, wherein the generating the Bezier shading correction surface comprises receiving, as input, an image of a uniform flat-field scene captured with the camera module, the image of the uniform flat-field scene comprising a plurality of pixels, each pixel of the plurality of pixels having a pixel value for each of a plurality of color channels.

15. The method according to claim 14, wherein the shading profile of a camera module comprises an intensity gradient for each of the plurality of color channels.

16. The method according to claim 14, wherein the generating the Bezier shading correction surface comprises:

receiving the shading profile of the camera module;

creating a plurality of correction surfaces based on the shading profile, each correction surface of the plurality of correction surfaces corresponding to a color channel of the plurality of color channels; and

calculating a Bezier-coefficients representation for the plurality of correction surfaces.

17. The method according to claim 16, wherein the creating a plurality of correction surfaces comprises:

finding a maximal value in each color channel of the plurality of color channels; and

for each pixel of a plurality of pixels, dividing the maximal value by the color value of the pixel.

18. The method according to claim 16, wherein the generating the Bezier shading correction surface further comprises correcting for an imbalance in green channels by multiplying a lower green channel of the plurality of color channels by a ratio of a higher green channel of the plurality of color channels and the lower green channel.

19. The method according to claim 16, wherein the calculating a Bezier-coefficients representation for the plurality of correction surfaces comprises:

dividing each correction surface into a plurality of sub-divisions;

performing a least-square optimization in each of the plurality of sub-divisions; and

determining a set of Bezier coefficients from the least-square optimizations.

20. The method according to claim 19, wherein the calculating the module-specific corrected surface measurement for the target illuminant is performed for each of the set of Bezier coefficients.

21. The method according to claim 19, wherein the calculating the module-specific corrected surface measurement for the target illuminant comprises:

generating a symmetric shading profile that matches a plurality of spatial locations corresponding to the set of Bezier coefficients;

calculating a plurality of falloff factors corresponding to the set of Bezier coefficients; and

multiplying each Bezier coefficient of the set of Bezier coefficients by a corresponding falloff factor of the plurality of falloff factors.

22. The method according to claim 21, wherein the calculating a plurality of falloff factors corresponding to the set of Bezier coefficients comprises calculating a squared radius at a spatial location of each Bezier coefficient of the set of Bezier coefficients relative to a center of the image.

23. The method according to claim 1, wherein the creating a module-specific corrected surface measurement for each illuminant of the plurality of illuminants is performed in a driver of the camera module.

24. The method according to claim 1, wherein the creating a module-specific corrected surface measurement for each illuminant of the plurality of illuminants is performed during a boot process of the mobile computing device in which the camera module is comprised.

25. The method according to claim 1, wherein the module-specific corrected surface measurements for the plurality of camera modules are stored in a memory of the mobile computing device, the memory being accessible to the driver of the camera module.

26. The method according to claim 1, wherein the module-specific calibration surface is stored in a one-time programmable memory in the mobile computing device.

27. The method according to claim 1, wherein the shading profile is measured during factory floor calibration of a camera module of the camera module type.

28. The method according to claim 1, wherein the radial component is substantially rotationally-symmetric.

29. The method according to claim 1, wherein the radial component is dependent on the type of illumination of the plurality of illuminants.

30. The method according to claim 1, wherein the non-radial component is substantially independent of the type of illumination of the plurality of illuminants.

31. The method according to claim 1, wherein the non-radial component of the shading profile comprises a horizontal/diagonal intensity gradient that is different for each color channel of a plurality of color channels.

32. An image capture device, said device comprising:

an image capture module comprising a lens, an image sensor and an image signal processor, the image capture module being configured to generate an image of a scene comprising a set of image data;

a memory, communicatively coupled to the image capture module, and configured to store the set of image data and a plurality of programmed instructions; and

a processor, communicatively coupled to the memory, and configured to execute the plurality of programmed instructions and to perform a lens shading correction method on the set of image data, the method comprising;

i) referencing a plurality of base surface measurements corresponding to a type of the image capture module and a shading profile specific to the image capture module to create a plurality of module-specific corrected surface measurements;

ii) determining an illuminant present in the scene;

iii) selecting a module-specific corrected surface measurement from the plurality of module-specific corrected surface measurements to correspond to the illuminant; and

iv) applying the selected module-specific corrected surface measurement to the image data,

wherein the plurality of base surface measurements are generated from a spectral scan of the image capture module corresponding to the type of the image capture module and a plurality of images generated under a plurality of simulated conditions based on a target illumination data and a calibration illumination data,

further wherein the shading profile comprises a radial component and a non-radial component.

33. The device according to claim 32, wherein the device comprises a mobile telephone.

34. The device according to claim 32, wherein the image data is preprocessed to subtract a black level in the image.

35. The device according to claim 32, wherein the image data is preprocessed to eliminate sensor artifacts from the image.

36. The device according to claim 32, wherein the image data comprises a full resolution version of the image and a sub-sampled version of the image.

37. A non-transitory computer readable medium containing program instructions embodied therein for causing a camera module in an image capture device to perform lens shading correction, the program instructions comprising:

instructions to reference, from a memory in an image capture device, a plurality of base surface measurements based on a plurality of spectral scans of a camera module type generated under a plurality of illuminants, and a plurality of images generated under conditions simulated to correspond to the plurality of illuminants;

instructions to reference, from the memory, a shading profile specific to the camera module of the camera module type to calculate a module-specific calibration surface, the shading profile comprising a radial component and a non-radial component;

instructions to create a module-specific corrected surface measurement for each illuminant of the plurality of illuminants based on the plurality of base surface measurements of and the module-specific calibration surface;

instructions to determine an illuminant in a scene captured by the camera module; and

instructions to apply the module-specific corrected surface measurement corresponding to the determined illuminant to an image data corresponding to the scene,

wherein, the instructions to reference the plurality of base surface measurements comprises instructions to generate a spectral scan with a monochromator of a camera module corresponding to the camera module type.

38. The device according to claim 37, wherein the image capture device is comprised in a mobile telephone.