System and Method for RAW Image Processing

• US 20080088857A1
• Filed: 06/01/2007
• Published: 04/17/2008
• Est. Priority Date: 10/13/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hayward, David, Zimmer, Mark, Marcu, Gabriel G.
• Granted Patent: US 7,773,127 B2
• Time in Patent Office: Days
• US Class Current: 358/1.6
• CPC Class Codes: H04N 1/56 Processing of colour picture signals H04N1/52 takes precedence
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

An automated RAW image processing method and system are disclosed. A RAW image and metadata related to the RAW image are obtained from a digital camera or other source. The RAW image and the related metadata are automatically processed using an Operating System service of a processing device to produce a resulting image in an absolute color space. The resulting image is then made available to an application program executing on the processing device through an application program interface with the Operating System service.

Claims

1. An automated RAW image processing method implementable by a processing device, the method comprising:

receiving a representation of a RAW image;

producing a resulting image in a color space by automatically pre-processing the RAW image with an operating system service of the processing device;

providing an interface to the operation system service; and

making the resulting image available via the interface to an application program executable on the processing device.

2. The method of claim 1, wherein the processing device comprises a computer.

3. The method of claim 1, comprising initially receiving the RAW image communicated from an imaging device to the processing device.

4. The method of claim 3, wherein the imaging device comprises a digital camera.

5. The method of claim 1, wherein the act of receiving the representation of the RAW image comprises receiving the RAW image from a kernel layer of the processing device.

6. The method of claim 1, wherein the act of providing an interface to the operating system service comprises providing an application programming interface having access to the operating system service to providing the resulting image to any of a plurality of application programs executable on the computer.

7. The method of claim 1, wherein the act of automatically pre-processing the RAW image comprises:

receiving metadata associated with the RAW image; and

using information from the metadata to obtain one or more algorithms stored on the processing device for pre-processing RAW images.

8. The method of claim 1, wherein the act of automatically pre-processing comprises producing a first image in a first unrendered color space by automatically performing one or more first pre-processing operations on the RAW image.

9. The method of claim 8, wherein the act of automatically performing one or more first pre-processing operations on the RAW image comprises performing one or more of:

subtracting black values from the RAW image, performing highlight recovery on the RAW image, eliminating stuck pixels from the RAW image, adjusting an auto-exposure of the RAW image, demosaicing the RAW image, and performing a chroma blur operation on the RAW image.

10. The method of claim 8, wherein the first unrendered color space comprises a camera RGB color space.

11. The method of claim 8, wherein the act of automatically pre-processing comprises producing a second image in a second unrendered color space by automatically performing one or more second pre-processing operations on the first image.

12. The method of claim 11, wherein the act of automatically performing one or more second pre-processing operations on the first image comprises applying a conversion matrix to convert RGB tristimulus values of an unrendered RGB color space in the first image into XYZ tristimulus values in the second image.

13. The method of claim 11, wherein the conversion matrix is derived from a plurality of characterization matrices associated with a camera identifiable by metadata associated with the RAW image.

14. The method of claim 8, wherein the act of automatically pre-processing comprises producing the resulting image in the rendered color space by automatically performing one or more third pre-processing operations on the second image.

15. The method of claim 14, wherein the act of automatically performing the one or more third pre-processing operations on the second image comprises using a color adaptation transform to adapt XYZ tristimulus values in the second image into RGB tristimulus values of a rendered RGB color space in the resulting image.

16. The method of claim 14, wherein the act of automatically performing the one or more third pre-processing operations on the second image comprises:

determining camera information from metadata associated with the RAW image;

obtaining a stored tone curve associated with the determined camera information; and

applying the obtained tone curve to each color channel of the second image to produce the resulting image.

17. The method of claim 1, wherein the act of automatically pre-processing comprises subtracting black offset from the RAW image.

18. The method of claim 17, wherein the act of subtracting black offset from the RAW image comprises:

determining black offset information from metadata associated with the RAW image; and

subtracting one or more black offset values from each photo-site of the RAW image based on the determined information.

19. The method of claim 1, wherein the act of automatically pre-processing comprises performing highlight recovery on the RAW image.

20. The method of claim 19, wherein the act of performing highlight recovery on the RAW image comprises:

receiving metadata associated with the RAW image;

scaling each color channel of the RAW image by a scaling factor based on a white balance indicated by the metadata;

calculating saturation values at which each color channel will become saturated using correlation information indicated by the metadata, the correlation information correlating saturation values to over-exposed images that have been examined for a camera identifiable by the metadata;

scaling the saturation values by the scaling factor;

analyzing each color channel of each pixel for saturation above the associated saturation value;

leaving the color channels unchanged for a given pixel if none of its color channels are above the associated saturation values;

replacing a given pixel by a neutral gray pixel if all of its color channels are above the associated saturation values;

replacing those saturated color channels with estimated values if at least one but less than all of the color channels for a given pixel are above the associated saturation values.

21. The method of claim 1, wherein the act of automatically pre-processing comprises reducing noise in the RAW image.

22. The method of claim 21, wherein the act of reducing noise in the RAW image comprises:

determining camera information, ISO setting, and exposure time from metadata associated with the RAW image;

obtaining a predetermined noise profile for the camera information stored in memory;

calculating an estimated amount of noise for the RAW image using the ISO setting and exposure time in the predetermined noise profile; and

pre-processing the RAW image for noise based on the estimated value.

23. The method of claim 1, wherein the act of automatically pre-processing comprises reducing stuck pixels from the RAW image.

24. The method of claim 23, wherein the act of reducing stuck pixels from the RAW image comprises:

determining camera information, ISO setting, and exposure time from metadata associated with the RAW image;

obtaining a predetermined stuck pixel profile for the camera information stored in memory;

calculating an estimated amount of stuck pixels for the RAW image using the ISO setting and exposure time in the predetermined stuck pixel profile; and

pre-processing the RAW image for stuck pixels based on the estimated value.

25. The method of claim 1, wherein the act of automatically pre-processing comprises adjusting an auto-exposure of the RAW image.

26. The method of claim 25, wherein the act of adjusting an auto-exposure of the RAW image comprises:

computing an average luminance of the RAW image;

determining camera information and exposure of the RAW image from metadata associated with the RAW image;

determining luminance variables from predetermined information stored in memory based on the camera information, the average luminance, and the exposure; and

adjusting the luminance of the RAW image by the determined luminance variables.

27. The method of claim 1, wherein the act of automatically pre-processing comprises demosaicing the RAW image.

28. The method of claim 1, wherein the act of automatically pre-processing comprises performing a chroma blur operation on the RAW image.

29. The method of claim 28, wherein the act of performing a chroma blur operation on the RAW image comprises:

creating from the RAW image an interpolated image having red, green, and blue values at each pixel location;

determining first luminance values for each pixel location of the interpolated image; and

creating from the RAW image a half-size image having red, green, and blue values;

blurring the half-size image with a blur operation;

resizing the blurred half-size image into a blurred full size image;

determining second luminance values for each pixel location of the blurred full size image; and

scaling at each pixel location the first luminance values in the interpolated image to match the second luminance values of the blurred full size image.

30. The method of claim 29, wherein the blur operation comprises one or more of a Gaussian blur, a selective blur, a bilateral filter, a median filter, and a box filter.

31. A operating system executable on a processing device to perform an automated RAW image processing method according to claim 1.

32. A program storage device, readable by a programmable processing device, comprising instructions stored thereon for causing the programmable processing device to perform an automated RAW image processing method according to claim 1.

33. An operating system executable on a computer, the operating system comprising:

an operating system service receiving a RAW image and automatically pre-processing the RAW image; and

an application programming interface having access to the operating system service and providing the pre-processed RAW image from the operating system service to an application program executable on the computer.

34. A operating system service provided by an operating system executable on a computer, wherein said operating system service—

receives as input a RAW image and associated metadata,

determines based on the metadata one or more pre-processing algorithms stored on the computer,

automatically preprocesses the RAW image with the one or more determined algorithms to produce a resulting image, and

makes the resulting image available to an application program executable on the computer.

35. A processing device, comprising:

at least one processor;

a memory storing algorithms for pre-processing RAW images;

a device interface module for receiving data associated with a RAW image from an imaging device; and

an operating system executable on the at least one processor and configured to;

receive a RAW image and associated metadata obtained from an imaging device via the device interface module;

determine one or more of the algorithms based on the metadata;

automatically pre-process the RAW image with the one or more determined algorithms to produce a resulting image in a rendered color space; and

make the resulting image available to an application program executable on the processing device.