System and method for raw image processing using conversion matrix interpolated from predetermined camera characterization matrices

• US 7,835,569 B2
• Filed: 10/13/2006
• Issued: 11/16/2010
• Est. Priority Date: 10/13/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Marcu, Gabriel G.
• Primary Examiner(s): Couso; Jose L
• Application Number:
  US11/549,627
• Publication Number:
  US 20080089580A1
• Time in Patent Office: 1,495 Days
• Field of Search: 382/162, 382/164, 382/167, 358/518, 358/519, 345/603, 345/604
• US Class Current: 382/162
• CPC Class Codes: H04N 1/603 controlled by characteristics of the picture signal generator or the picture reproducer
• CPC Class Codes: H04N 1/6086 by scene illuminant, i.e. conditions at the time of picture capture, e.g. flash, optical filter used, evening, cloud, daylight, artificial lighting, white point measurement, colour temperature
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values

Abstract

An automated RAW image processing method and system are disclosed. A computer receives a RAW image and metadata from a camera and interpolates an interpolated image from the RAW image so that the interpolated image is in a first color space. Predefined characterization matrices associated with the camera are obtained. Each of the characteristic matrices is associated with a different illuminant. A conversion matrix is interpolated from the characterization matrices and a white balance of the RAW image. The interpolated image is converted with the conversion matrix into a second color space. A resulting image in a rendered color space is produced from the converted image, and the resulting image is made available to an application program.

Claims

1. An automated RAW image processing method implementable by a processing device, the method comprising:

interpolating an interpolated image from a RAW image, the interpolated image being in a first color space;

obtaining a plurality of characterization matrices associated with information about a camera used to capture the RAW image, each of the characteristic matrices associated with a different illuminant;

interpolating with one or more processing units of a processing device a conversion matrix from the characterization matrices and a white balance of the RAW image;

converting with the one or more processing units the interpolated image with the conversion matrix into a second color space;

producing with the one or more processing units a resulting image in a rendered color space from the converted image; and

making the resulting image available to an application program.

2. The method of claim 1, wherein the processing device comprises a computer having a central processing unit to implement the automated RAW image processing method.

3. The method of claim 1, wherein the processing device comprises a computer having a graphics processing unit to implement the automated RAW image processing method.

4. The method of claim 1, wherein the processing device comprises a camera having a processing unit to implement the automated RAW image processing method.

5. The method of claim 1, wherein the processing device comprises an operating system automatically implementing the automated RAW image processing method without user intervention.

6. The method of claim 1, wherein each of the characteristic matrices is derived by:

obtaining a reference image of a scene under an illuminant in a first color space using a camera and a predefined white balance;

measuring color values of the scene under the illuminant;

fitting sensed color values of the reference image to the measured color values;

deriving the characterization matrix for the camera based on the fitting;

normalizing the characterization matrix, and

associating the characterization matrix with the camera and the illuminant.

7. The method of claim 6, further comprising repeating the acts of obtaining, measuring, fitting, deriving, and associating by using the same camera and one or more additional illuminants.

8. The method of claim 6, further comprising repeating the acts of obtaining, fitting, deriving, and associating by using the same illuminant and one or more additional cameras.

9. The method of claim 1, wherein the first color space comprises an unrendered RGB color space, wherein the second color space comprises an XYZ color space, and wherein the rendered color space comprises a rendered RGB color space.

10. The method of claim 1, wherein the information about the camera used to capture the RAW image comprises a white balance of the RAW image and an indication of a camera used to capture the RAW image.

11. The method of claim 10, wherein the indication is selected from the group consisting of a model, a manufacturer, a type of imaging sensor, and a camera specific category.

12. The method of claim 10, wherein the act of obtaining the plurality of characterization matrices associated with the information about the camera information comprises obtaining the characterization matrices associated with the indication of the camera used to capture the RAW image.

13. The method of claim 1, wherein the act of interpolating the conversion matrix that estimates the illuminant of the RAW image from the characterization matrices comprises:

selecting a first of the characteristic matrices associated with a first of the illuminants; and

selecting a second of the characteristic matrices associated with a second of the illuminants.

14. The method of claim 13, wherein selecting the first and second characteristic matrices comprises selecting the first and second characteristics matrices associated with the illuminants having distant correlated color temperatures.

15. The method of claim 1, wherein the act of interpolating the conversion matrix from the characterization matrices and the white balance of the RAW image comprises:

using a bi-dimensional convergence process to calculate white points from the white balance of the RAW image, the characteristics matrices, and

weighting factors for the characteristics matrices;

converging the white points within a threshold; and

calculating variables for the conversion matrix using the weighting factors associated with the converged white points.

16. The method of claim 1, further comprising subtracting black compensation variables for color channels of the interpolated image before converting the interpolated image with the conversion matrix.

17. The method of claim 16, further comprising calculating the black compensation variables by averaging black compensation variables for the characteristic matrices.

18. The method of claim 16, further comprising interpolating the black compensation variables from black compensation variables associated with the characterization matrices used to interpolate the conversion matrix.

19. The method of claim 16, wherein the black compensation variables are derived by:

obtaining a reference image of a scene under an illuminant using a camera;

measuring color values of the scene under the illuminant;

fitting sensed color values of the reference image to the measured color values;

deriving the characterization matrix and the black compensation variables for the camera based on the fitting; and

associating the characterization matrix and the black compensation variables with the camera and the illuminant.

20. The method of claim 19, further comprising repeating the acts of obtaining, measuring, fitting, deriving, and associating by using the same camera and one or more additional illuminants.

21. The method of claim 19, further comprising repeating the acts of obtaining, fitting, deriving, and associating by using the same illuminant and one or more additional cameras.

22. An automated RAW image processing method implementable by a processing device, the method comprising:

obtaining a reference image of a scene under an illuminant using a camera;

measuring color values of the scene under the illuminant;

fitting with one or more processing units of a processing device sensed color values of the reference image to the measured color values;

deriving with the one or more processing units a characterization matrix for the camera based on the fitting;

associating with the one or more processing units the characterization matrix with the camera and the illuminant; and

repeating the acts of obtaining, measuring, fitting, deriving, and associating by using the same camera and one or more additional illuminants.

23. The method of claim 22, further comprising repeating the acts of obtaining, fitting, deriving, and associating by using the same illuminant and one or more additional cameras.

24. The method of claim 22, further comprising:

obtaining a RAW image and metadata;

interpolating an interpolated image from the RAW image, the interpolated image being in a first color space;

obtaining a plurality of characterization matrices using the metadata, each of the characteristic matrices associated with a different illuminant;

interpolating a conversion matrix from the characterization matrices;

converting the interpolated image with the conversion matrix into a second color space;

producing a resulting image in a rendered color space from the converted image; and

making the resulting image available to an application program.

25. A processing device comprising an operating system performing an automated RAW image processing method, the method comprising:

interpolating an interpolated image from a RAW image, the interpolated image being in a first color space;

obtaining a plurality of characterization matrices associated with information about a camera used to capture the RAW image, each of the characteristic matrices associated with a different illuminant;

interpolating a conversion matrix from the characterization matrices and a white balance of the RAW image;

converting the interpolated image with the conversion matrix into a second color space;

producing a resulting image in a rendered color space from the converted image; and

making the resulting image available to an application program.

26. A program storage device, readable by a programmable processing device, comprising instructions stored on the program storage device for causing the programmable processing device to perform an automated RAW image processing method, the method comprising:

interpolating an interpolated image from a RAW image, the interpolated image being in a first color space;

obtaining a plurality of characterization matrices associated with information about a camera used to capture the RAW image, each of the characteristic matrices associated with a different illuminant;

interpolating a conversion matrix from the characterization matrices and a white balance of the RAW image;

converting the interpolated image with the conversion matrix into a second color space;

producing a resulting image in a rendered color space from the converted image; and

making the resulting image available to an application program.

27. A method for processing a RAW image captured by a camera, the method comprising:

interpolating an interpolated image from a RAW image, the interpolated image being in a first color space;

obtaining a plurality of characterization datum associated with the camera,

interpolating a conversion tool by using a processing device, the processing device being independent of the camera, wherein interpolating the conversion tool comprises employing at least some of the characterization datum and some of the data associated with the RAW image; and

converting with the processing device the interpolated image with the conversion tool into a second color space.

28. The method of claim 27, further comprising producing a resulting image in a rendered color space from the converted image.

29. The method of claim 28, further comprising making the resulting image available to an application program running on the processing device.

30. The method of claim 27, wherein the data associated with the RAW image comprises white balance information.

31. The method of claim 30, further comprising producing a resulting image in a rendered color space from the converted image.

32. The method of claim 31, further comprising making the resulting image available to an application program running on the processing device.

44. The method of claim 27, wherein the processing device comprises a computer, and wherein the at least one processor comprises a central processing unit.

45. The method of claim 27, wherein the processing device comprises a computer, and wherein the at least one processor comprises a graphics processing unit.

46. The method of claim 27, wherein the characterization datum associated with the camera comprises a plurality of conversion matrices, each of the characteristic matrices associated with a different illuminant.

33. A system, comprising:

an imaging device having an imaging sensor and a first interface module, the imaging sensor coupled to the first interface module;

a processing device having at least one processor, a memory for storing characterization data associated with the imaging device, a second interface module, and an operating system adapted to run on the at least one processor, the second interface module configured to receive data associated with a RAW image from the first interface module, the RAW image based upon data captured by the imaging sensor; and

a software program associated with the operating system and adapted to run on the at least one processor, the software program configured to;

interpolate an interpolated image from the RAW image, the interpolated image being in a first color space,

interpolate a conversion tool by employing at least some of the characterization data and some the data associated with the RAW image, and

convert the interpolated image into a second color space by using the conversion tool.

47. The system of claim 33, wherein the processing device comprises a computer, and wherein the at least one processor comprises a central processing unit.

48. The system of claim 33, wherein the processing device comprises a computer, and wherein the at least one processor comprises a graphics processing unit.

49. The system of claim 33, wherein the imaging device comprises a camera.

50. The system of claim 33, wherein the software program automatically processes the RAW image without user intervention.

51. The system of claim 33, wherein the software program is configured to produce a resulting image in a rendered color space from the converted image.

52. The system of claim 51, wherein the software program is configured to make the resulting image available to an application program running on the processing device.

53. The system of claim 33, wherein the data associated with the RAW image comprises white balance information.

54. The system of claim 33, wherein the data associated with the RAW image comprises an indication of the imaging device used to capture the RAW image.

55. The system of claim 33, wherein the characterization data associated with the imaging device comprises a plurality of conversion matrices, each of the characteristic matrices associated with a different illuminant.

34. A processing device, comprising:

at least one processor;

a memory for storing data associated with a camera;

a computer interface module for receiving data associated with a RAW image,

an operating system adapted to run on the at least one processor;

a software program associated with said operating system and adapted to run on the at least one processor, the software program configured to;

interpolate an interpolated image from the data associated with the RAW image, the interpolated image being in a first color space,

interpolate a conversion tool by employing at least some of the data associated with the camera and some of the data associated with the RAW image, and

convert the interpolated image into a second color space by using the conversion tool.

35. The device of claim 34, wherein the processing device comprises a computer, and wherein the at least one processor comprises a central processing unit.

36. The device of claim 34, wherein the processing device comprises a computer, and wherein the at least one processor comprises a graphics processing unit.

37. The device of claim 34, wherein the processing device comprises a camera.

38. The device of claim 34, wherein the software program automatically processes the RAW image without user intervention.

39. The device of claim 34, wherein the software program is configured to produce a resulting image in a rendered color space from the converted image.

40. The device of claim 39, wherein the software program is configured to make the resulting image available to an application program running on the processing device.

41. The device of claim 34, wherein the data associated with the RAW image comprises white balance information.

42. The device of claim 34, wherein the data associated with the RAW image comprises an indication of the camera used to capture the RAW image.

43. The device of claim 34, wherein the data associated with the camera comprises a plurality of conversion matrices, each of the characteristic matrices associated with a different illuminant.