System and Method for RAW Image Processing

• US 20100271505A1
• Filed: 07/07/2010
• Published: 10/28/2010
• Est. Priority Date: 10/13/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hayward, David, Zimmer, Mark, Marcu, Gabriel G.
• Granted Patent: US 8,493,473 B2
• Time in Patent Office: Days
• US Class Current: 348/231.200
• CPC Class Codes: H04N 1/56 Processing of colour picture signals H04N1/52 takes precedence
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/85 for matrixing
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

An automated RAW image processing method and system are disclosed. A RAW image and metadata related to the RAW image are obtained from a digital camera or other source. The RAW image and the related metadata are automatically processed using an Operating System service of a processing device to produce a resulting image in an absolute color space. The resulting image is then made available to an application program executing on the processing device through an application program interface with the Operating System service.

Claims

1. An automated RAW image processing method, comprising:

storing pre-processing information for pre-processing RAW images with a processing device, the pre-processing information being derived from processing at least one subject RAW image generated using at least one defined condition;

receiving a representation of a target RAW image and metadata, the target RAW image generated using a target condition, the metadata indicative of the target condition;

identifying, based on the metadata, the pre-processing information having the at least one defined condition corresponding to the target condition;

automatically pre-processing the target RAW image into a resulting image in a color space, the automatic pre-processing using an operating system service of the processing device and the identified pre-processing information; and

making the resulting image available to at least one application program executable on the processing device.

2. The method of claim 1, wherein receiving the representation comprises receiving the target RAW image from a kernel layer of the processing device.

3. The method of claim 1, wherein receiving the representation comprises receiving the target RAW image from an imaging device operatively coupled to the processing device.

4. The method of claim 3, wherein the processing device comprises a computer, and wherein the imaging device comprises a digital camera.

5. The method of claim 1, wherein making the resulting image available comprises providing an application programming interface having access to the operating system service, the application programming interface providing the resulting image to any of a plurality of the at least one application programs executable on the processing device.

6. The method of claim 1, wherein automatically pre-processing comprises interpolating photo-site pixels of the target RAW image into interpolated pixels, each of the photo-site pixels indicative of a single color, each of the interpolated pixels having color channels of a color space.

7. The method of claim 1, wherein identifying and automatically pre-processing comprises:

identifying an imaging device in the pre-processing information based on the metadata; and

converting unrendered color space pixel values from the target RAW image into XYZ tristimulus pixel values by applying a conversion matrix, the conversion matrix being derived from a plurality of characterization matrices associated with the identified imaging device.

8. The method of claim 1, wherein identifying and automatically pre-processing comprises:

identifying a tone curve in the pre-processing information based on the metadata; and

applying the identified tone curve to each color channel of pixels from the target RAW image.

9. The method of claim 1, wherein identifying and automatically pre-processing comprises:

identifying black offset information in the pre-processing information based on the metadata; and

subtracting one or more black offset values from pixels from the target RAW image based on the black offset information.

10. The method of claim 1, wherein identifying and automatically pre-processing comprises:

identifying white balance information in the pre-processing information based on the metadata; and

calculating saturation values at which each color channel of pixels from the target RAW image saturates, each color channel being scaled based on the white balance information; and

modifying any of the pixels saturated above the calculated saturation value associated therewith.

11. The method of claim 1, wherein identifying and automatically pre-processing comprises:

identifying a stuck pixel profile in the pre-processing information based on the metadata;

calculating a stuck pixel estimate for the target RAW image using the identified stuck pixel profile; and

processing for stuck pixels from the target RAW image based on the stuck pixel estimate.

12. The method of claim 1, wherein identifying and automatically pre-processing comprises:

identifying a noise profile in the pre-processing information based on the metadata;

calculating a noise estimate for the target RAW image using the identified noise profile; and

processing the target RAW image based on the noise estimate.

13. The method of claim 1, wherein identifying and automatically pre-processing comprises:

identifying a luminance variable in the pre-processing information based on the metadata; and

adjusting luminance of the target RAW image using the identified luminance variable.

14. The method of claim 1, wherein automatically pre-processing comprises:

creating a blurred reduced-size image from the target RAW image,

resizing the blurred reduced-size image into a blurred full-size image,

determining first luminance values for each first pixel location of an interpolated image interpolated from the target RAW image,

determining second luminance values for each second pixel location of the blurred full-size image, and

scaling the first luminance values in the interpolated image to match the second luminance values of the blurred full-size image.

15. The method of claim 1, wherein automatically pre-processing comprises creating a half-size image interpolated into the color space from the target RAW image.

16. The method of claim 1, wherein the pre-processing information comprise one or more of:

black-offset information, saturation information, a noise profile, a stuck pixel profile, luminance information, first information for interpolating the RAW image, second information for converting from one color space to another, and third information for adjusting tone.

17. The method of claim 1, wherein the pre-processing information is derived from processing the at least one subject RAW image to produce at least one predetermined result in a subject resulting image, the at least one predetermined result comprising one or more of:

a contrast level, a saturation level, an amount of pixels affected by stuck pixels, an amount of pixels affected by noise, a luminance level, a color level, a sharpness level, and a tone level in the subject resulting image.

18. The method of claim 1, wherein the at least one derived and target conditions include one or more of:

an indication of an imaging device used to generate the subject or target RAW image, a shutter speed, an aperture, a white balance, an exposure compensation, a metering setting, an ISO setting, a black-offset value, an attribute of the imaging device, and an attribute of the target RAW image.

19. The method of claim 1, wherein the at least one application is selected from the group consisting of a word processing application, a photo application, an address book application, and an e-mail application.

20. The method of claim 1, wherein the act of automatically pre-processing comprises accessing a graphics processing unit of the processing device.

21. A non-transitory program storage device, readable by a programmable processing device, comprising instructions stored thereon for causing the programmable processing device to perform an automated RAW image processing method according to claim 1.

22. A processing device, comprising:

memory storing pre-processing information for pre-processing target RAW images, the pre-processing information being derived from processing at least one subject RAW image generated using at least one defined condition; and

at least one processor communicatively coupled to the memory, the at least one processor having at least one application program and an operating system service executable thereon, the at least one processor configured to;

obtain a representation of a target RAW image and metadata associated therewith, the target RAW image generated using a target condition, the metadata indicative of the target condition,

identify, based on the metadata, the pre-processing information having the at least one defined condition corresponding to the target condition,

automatically pre-process the target RAW image into a resulting image in a color space, the automatic pre-processing using the operating system service and the identified pre-processing information, and

make the resulting image available to the at least one application program.