System and method for RAW image processing

• US 7,773,127 B2
• Filed: 06/01/2007
• Issued: 08/10/2010
• Est. Priority Date: 10/13/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hayward, David, Zimmer, Mark, Marcu, Gabriel G.
• Primary Examiner(s): WHIPKEY, JASON T
• Application Number:
  US11/756,906
• Publication Number:
  US 20080088857A1
• Time in Patent Office: 1,166 Days
• Field of Search: 348/223.1, 348/224.1, 348/225.1, 348/246, 348/247, 348/231.6
• US Class Current: 348/231.6
• CPC Class Codes: H04N 1/56 Processing of colour picture signals H04N1/52 takes precedence
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

An automated RAW image processing method and system are disclosed. A RAW image and metadata related to the RAW image are obtained from a digital camera or other source. The RAW image and the related metadata are automatically processed using an Operating System service of a processing device to produce a resulting image in an absolute color space. The resulting image is then made available to an application program executing on the processing device through an application program interface with the Operating System service.

Claims

1. An automated RAW image processing method implementable by a processing device, the method comprising:

receiving a representation of a target RAW image and metadata, the target RAW image having photo-site pixels each indicative of a single color value, the metadata indicative of an imaging device or a capture condition associated with the target RAW image;

obtaining, based on the metadata, predetermined information for RAW image processing, the predetermined information being derived from processing at least one subject RAW image obtained with the imaging device or the capture condition;

demosaicing the target RAW image to produce a resulting image defined in a color space, wherein the demosaicing comprises automatically pre-processing the target RAW image based on the predetermined information using an operating system service of the processing device;

and

making the resulting image available to an application program executable on the processing device.

2. The method of claim 1, wherein the processing device comprises a computer, and wherein the imaging device comprises a digital camera.

3. The method of claim 1, wherein the act of receiving the representation of the target RAW image comprises receiving the target RAW image from a kernel layer of the processing device.

4. The method of claim 1, wherein the act of making the resulting image available comprises providing an application programming interface having access to the operating system service to provide the resulting image to any of a plurality of application programs executable on the processing device.

5. The method of claim 1, wherein the act of automatically pre-processing comprises producing a first image in a first unrendered color space, wherein the first unrendered color space comprises a unrendered RGB color space.

6. The method of claim 5, wherein the act of automatically pre-processing comprises producing a second image in a second unrendered color space from the first image by applying a conversion matrix to convert RGB tristimulus values of the unrendered RGB color space in the first image into XYZ tristimulus values in the second image.

7. The method of claim 6, wherein the conversion matrix is derived from a plurality of characterization matrices associated with the predetermined information.

8. The method of claim 6, wherein the act of automatically pre-processing comprises producing the resulting image in a rendered color space from the second image.

9. The method of claim 8, wherein the act of producing the resulting image in the rendered color space comprises using a color adaptation transform to adapt the XYZ tristimulus values in the second image into RGB tristimulus values of a rendered RGB color space in the resulting image.

10. The method of claim 8, wherein the act of producing the resulting image comprises:

obtaining a stored tone curve associated with the predetermined information; and

applying the obtained tone curve to each color channel of the second image to produce the resulting image.

11. The method of claim 1, wherein demosaicing the target RAW image to produce the resulting image comprises creating a half-size image interpolated into the color space from the target RAW image.

12. The method of claim 1, wherein the predetermined information comprise one or more of:

black-offset information, saturation information, an amount of noise, an amount of stuck pixels, luminance information, information for interpolating the RAW image, information for converting from one color space to another, and information for adjusting tone.

13. The method of claim 1, wherein the predetermined information is derived from processing the at least one subject RAW image to produce at least one predetermined result, the at least one predetermined result comprising one or more of:

a contrast level, a saturation level, an amount of pixels affected by stuck pixels, an amount of pixels affected by noise, a luminance level, a color level, a sharpness level, and a tone level in the resulting image.

14. The method of claim 1, wherein the capture conditions include one or more of:

a shutter speed, an aperture, a white balance, an exposure compensation, a metering setting, an ISO setting, a black-offset value, an attribute of the imaging device, and an attribute of the RAW image.

15. The method of claim 1, wherein the application is selected from the group consisting of a word processing application, a photo application, an address book application, and an e-mail application.

16. The method of claim 1, wherein the act of automatically pre-processing comprises accessing a graphics processing unit of the processing device.

17. The method of claim 1, wherein the act of automatically pre-processing comprises:

obtaining black offset information from the predetermined information; and

subtracting one or more black offset values from each photo-site pixel of the target RAW image based on the black offset information.

18. The method of claim 1, wherein the act of automatically pre-processing comprises:

scaling each color channel of the target RAW image based on a white balance indicated by the metadata;

calculating saturation values at which each color channel saturates based on correlation information obtained from the predetermined information, the correlation information correlating saturation values to images;

analyzing each color channel of each pixel for saturation above the calculated saturation value associated therewith; and

modifying any of the pixels of the target RAW image based on the analysis.

19. The method of claim 1, wherein the act of automatically pre-processing comprises:

obtaining a stuck pixel profile from the predetermined information,

calculating an estimated value of stuck pixels in the target RAW image using the capture condition and the stuck pixel profile, and

processing the target RAW image for stuck pixels based on the estimated value.

20. The method of claim 1, wherein the act of automatically pre-processing comprises:

obtaining a noise profile from the predetermined information;

calculating an estimated amount of noise for the target RAW image using the capture condition and the noise profile; and

pre-processing the target RAW image based on the estimated amount of noise.

21. The method of claim 1, wherein the act of automatically pre-processing comprises:

computing an average luminance of the target RAW image;

obtaining luminance variables from the predetermined information based on the imaging device, the average luminance, and the capture condition; and

adjusting luminance of the target RAW image by the determined luminance variables.

22. The method of claim 1, wherein the act of automatically pre-processing comprises:

determining first luminance values for each first pixel location of an interpolated image interpolated from the target RAW image,

creating a blurred reduced-size image from the target RAW image,

resizing the blurred reduced-size image into a blurred full-size image,

determining second luminance values for each second pixel location of the blurred full-size image, and

scaling the first luminance values in the interpolated image to match the second luminance values of the blurred full-size image.

23. A non-transitory program storage device, readable by a programmable processing device, comprising instructions stored thereon for causing the programmable processing device to perform an automated RAW image processing method according to claim 1.

24. A processing device, comprising:

at least one processor;

a memory storing predetermined information for RAW image processing, the predetermined information associated with imaging devices or capture conditions used to obtain subject RAW images and derived from processing the subject RAW images;

a device interface module for receiving data associated with a RAW image from an imaging device; and

an operating system executable on the at least one processor and configured to;

receive a target RAW image and associated metadata obtained from an imaging device via the device interface module, the target RAW image having photo-site pixels each indicative of a single color value;

determine the predetermined information based on the metadata;

automatically pre-process the target RAW image with the determined predetermined information to demosaic the target RAW image and produce a resulting image defined in a color space; and

make the resulting image available to an application program executable on the processing device.

25. An automated RAW image processing method implementable by a processing device, the method comprising:

receiving a representation of a RAW image and metadata associated therewith, the metadata indicative of a white balance;

automatically pre-processing the RAW image with an operating system service of the processing device to produce a resulting image in a color space, the automatic pre-processing including;

scaling each color channel of the RAW image based on the white balance,

calculating saturation values at which each color channel saturates based on correlation information correlating saturation values to images,

analyzing each color channel of each pixel for saturation above the calculated saturation value associated therewith, and

modifying any of the pixels of the RAW image based on the analysis; and

making the resulting image available to an application program executable on the processing device.

26. The method of claim 25, comprising initially receiving the RAW image communicated from a digital camera to the processing device, wherein the processing device comprises a computer.

27. The method of claim 25, wherein the act of making the resulting image available comprises providing an application programming interface having access to the operating system service to provide the resulting image to any of a plurality of application programs executable on the processing device.

28. The method of claim 25, wherein the act of automatically pre-processing comprises performing one or more of:

subtracting black offset from the RAW image, reducing noise from the RAW image, reducing stuck pixels from the RAW image, adjusting luminance in the RAW image, demosaicing the RAW image, and blurring portion of the RAW image.

29. The method of claim 25, wherein scaling comprises scaling each color channel by a scaling factor based on the white balance;

and wherein calculating saturation values comprises scaling the saturation values by the scaling factor.

30. The method of claim 25, wherein the act of modifying comprises:

leaving the color channels unchanged for a given one of the pixels if none of its color channels are above the associated saturation values;

replacing a given one of the pixels by a neutral pixel if all of its color channels are above the associated saturation values; and

replacing those saturated color channels with estimated values if at least one but less than all of the color channels for a given one of the pixels are above the associated saturation values.

31. A non-transitory program storage device, readable by a programmable processing device, comprising instructions stored thereon for causing the programmable processing device to perform an automated RAW image processing method, the method comprising:

receiving a representation of a RAW image and metadata associated therewith, the metadata indicative of a white balance and correlation information;

automatically pre-processing the RAW image with an operating system service of the processing device to produce a resulting image in a color space, the pre-processing including;

scaling each color channel of the RAW image based on the white balance,

calculating saturation values at which each color channel saturates based on correlation information correlating saturation values to images,

analyzing each color channel of each pixel for saturation above the calculated saturation value associated therewith, and

modifying any of the pixels of the RAW image based on the analysis; and

making the resulting image available to an application program executable on the processing device.

32. The device of claim 31, comprising initially receiving the RAW image from a digital camera, wherein the processing device comprises a computer.

33. The device of claim 31, wherein the act of making the resulting image available comprises providing an application programming interface having access to the operating system service to provide the resulting image to any of a plurality of application programs executable on the processing device.

34. The device of claim 31, wherein the act of automatically pre-processing further comprises performing one or more of:

subtracting black offset from the RAW image, reducing noise from the RAW image, reducing stuck pixels from the RAW image, adjusting luminance in the RAW image, demosaicing the RAW image, and blurring portion of the RAW image.

35. The device of claim 31, wherein scaling comprises scaling each color channel by a scaling factor based on the white balance;

and wherein calculating saturation values comprises scaling the saturation values by the scaling factor.

36. The device of claim 31, wherein the act of modifying comprises:

leaving the color channels unchanged for a given one of the pixels if none of its color channels are above the associated saturation values;

replacing a given one of the pixels by a neutral pixel if all of its color channels are above the associated saturation values; and

replacing those saturated color channels with estimated values if at least one but less than all of the color channels for a given one of the pixels are above the associated saturation values.

37. A processing device, comprising:

at least one processor;

a memory storing algorithms for pre-processing RAW images;

a device interface module for receiving data associated with a RAW image from an imaging device; and

an operating system executable on the at least one processor and configured to;

receive a RAW image and associated metadata obtained from an imaging device via the device interface module, the metadata indicative of a white balance;

determine one or more of the algorithms based on the metadata;

automatically pre-process the RAW image with the one or more determined algorithms to produce a resulting image in a color space, the one or more algorithms scaling each color channel of the RAW image based on the white balance, calculating saturation values at which each color channel saturates based on correlation information correlating saturation values to images, analyzing each color channel of each pixel for saturation above the calculated saturation value associated therewith, and modifying any of the pixels of the RAW image based on the analysis; and

make the resulting image available to an application program executable on the processing device.

38. The device of claim 37, wherein the imaging device comprises a digital camera, and wherein the processing device comprises a computer.

39. The device of claim 37, wherein the operating system comprises an application programming interface providing the resulting image to any of a plurality of application programs executable on the processing device.

40. The device of claim 37, wherein the one or more algorithms perform one or more of:

subtracting black offset from the RAW image, reducing noise from the RAW image, reducing stuck pixels from the RAW image, adjusting luminance in the RAW image, demosaicing the RAW image, and blurring portion of the RAW image.

41. The device of claim 37, wherein to scale each color channel, the one or more algorithms use a scaling factor based on the white balance;

and wherein to calculate saturation values, the one or more algorithms scale the saturation values by the scaling factor.

42. The device of claim 37, wherein to modify any of the pixels of the RAW image, the one or more algorithms are configured to:

leave the color channels unchanged for a given one of the pixels if none of its color channels are above the associated saturation values;

replace a given one of the pixels by a neutral pixel if all of its color channels are above the associated saturation values; and

replace those saturated color channels with estimated values if at least one but less than all of the color channels for a given one of the pixels are above the associated saturation values.

43. An automated RAW image processing method implementable by a processing device, the method comprising:

receiving a representation of a RAW image and metadata associated with the RAW image, the metadata indicative of an imaging device and a capture condition;

automatically pre-processing the RAW image with an operating system service of the processing device to produce a resulting image in a color space, the pre-processing including;

obtaining a stuck pixel profile stored in memory based on the imaging device,

calculating an estimated value of stuck pixels in the RAW image using the capture condition and the stuck pixel profile, and

processing the RAW image for stuck pixels based on the estimated value; and

making the resulting image available to an application program executable on the processing device.

44. The method of claim 43, wherein the capture condition is indicative of an ISO setting and an exposure.

45. The method of claim 43, comprising initially receiving the RAW image from a digital camera, wherein the processing device comprises a computer.

46. The method of claim 43, wherein the act of making the resulting image available comprises providing an application programming interface having access to the operating system service to provide the resulting image to any of a plurality of application programs executable on the processing device.

47. The method of claim 43, wherein the act of automatically pre-processing comprises performing one or more of:

subtracting black offset from the RAW image, adjusting saturation in the RAW image, reducing noise from the RAW image, adjusting luminance in the RAW image, demosaicing the RAW image, and blurring portion of the RAW image.

48. A non-transitory program storage device, readable by a programmable processing device, comprising instructions stored thereon for causing the programmable processing device to perform an automated RAW image processing method, the method comprising:

receiving a representation of a RAW image and metadata associated with the RAW image, the metadata indicative of an imaging device and a capture condition;

automatically pre-processing the RAW image with an operating system service of the processing device to produce a resulting image in a color space, the pre-processing including;

obtaining a stuck pixel profile stored in memory based on the imaging device,

calculating an estimated value of stuck pixels in the RAW image using the capture condition and the stuck pixel profile, and

processing the RAW image for stuck pixels based on the estimated value; and

making the resulting image available to an application program executable on the processing device.

49. The method of claim 48, wherein the capture condition is indicative of an ISO setting and an exposure.

50. The method of claim 48, comprising initially receiving the RAW image from a digital camera, wherein the processing device comprises a computer.

51. The method of claim 48, wherein the act of making the resulting image available comprises providing an application programming interface having access to the operating system service to provide the resulting image to any of a plurality of application programs executable on the processing device.

52. The method of claim 48, wherein the act of automatically pre-processing comprises performing one or more of:

subtracting black offset from the RAW image, adjusting saturation in the RAW image, reducing noise from the RAW image, adjusting luminance in the RAW image, demosaicing the RAW image, and blurring portion of the RAW image.

53. A processing device, comprising:

at least one processor;

a memory storing algorithms for pre-processing RAW images;

a device interface module for receiving data associated with a RAW image from an imaging device; and

an operating system executable on the at least one processor and configured to;

receive a RAW image and associated metadata obtained from an imaging device via the device interface module, the metadata indicative of the imaging device and a capture condition;

determine one or more of the algorithms based on the metadata;

automatically pre-process the RAW image with the one or more determined algorithms to produce a resulting image in a color space, the one or more algorithms obtaining a stuck pixel profile stored in memory based on the imaging device, calculating an estimated value of stuck pixels in the RAW image using the capture condition and the stuck pixel profile, and processing the RAW image for stuck pixels based on the estimated value; and

make the resulting image available to an application program executable on the processing device.

54. The method of claim 53, wherein the capture condition is indicative of an ISO setting and an exposure.

55. The device of claim 53, wherein the imaging device comprises a digital camera, and wherein the processing device comprises a computer.

56. The device of claim 53, wherein the operating system comprises an application programming interface providing the resulting image to any of a plurality of application programs executable on the processing device.

57. The device of claim 53, wherein the one or more algorithms perform one or more of:

subtracting black offset from the RAW image, adjusting saturation in the RAW image, reducing noise from the RAW image, adjusting luminance in the RAW image, demosaicing the RAW image, and blurring portion of the RAW image.

58. An automated RAW image processing method implementable by a processing device, the method comprising:

receiving a representation of a RAW image;

automatically pre-processing the RAW image with an operating system service of the processing device to produce a resulting image in a color space, the pre-processing including;

determining first luminance values for each first pixel location of an interpolated image interpolated from the RAW image,

creating a blurred reduced-size image from the RAW image,

resizing the blurred reduced-size image into a blurred full-size image,

determining second luminance values for each second pixel location of the blurred full-size image, and

scaling the first luminance values in the interpolated image to match the second luminance values of the blurred full-size image; and

making the resulting image available to an application program executable on the processing device.

59. The method of claim 58, comprising initially receiving the RAW image from a digital camera, wherein the processing device comprises a computer.

60. The method of claim 58, wherein the act of making the resulting image available comprises providing an application programming interface having access to the operating system service to provide the resulting image to any of a plurality of application programs executable on the processing device.

61. The method of claim 58, wherein the act of automatically pre-processing comprises performing one or more of:

subtracting black offset from the RAW image, adjusting saturation in the RAW image, reducing noise from the RAW image, reducing stuck pixels from the RAW image, adjusting luminance in the RAW image, and demosaicing the RAW image.

62. The method of claim 58, wherein creating the interpolated image comprises creating the interpolated image having red, green, and blue values at each first pixel location, and wherein creating the blurred reduced-size image comprises:

creating from the RAW image a half-size image having red, green, and blue values at each pixel location, and

blurring the half-size image.

63. A non-transitory program storage device, readable by a programmable processing device, comprising instructions stored thereon for causing the programmable processing device to perform an automated RAW image processing method, the method comprising:

receiving a representation of a RAW image;

automatically pre-processing the RAW image with an operating system service of the processing device to produce a resulting image in a color space, the pre-processing including;

determining first luminance values for each first pixel location of an interpolated image interpolated from the RAW image,

creating a blurred reduced-size image from the RAW image,

resizing the blurred reduced-size image into a blurred full-size image,

determining second luminance values for each second pixel location of the blurred full-size image, and

scaling the first luminance values in the interpolated image to match the second luminance values of the blurred full-size image; and

making the resulting image available to an application program executable on the processing device.

64. The device of claim 63, comprising initially receiving the RAW image from a digital camera, wherein the processing device comprises a computer.

65. The device of claim 63, wherein the act of making the resulting image available comprises providing an application programming interface having access to the operating system service to provide the resulting image to any of a plurality of application programs executable on the processing device.

66. The device of claim 63, wherein the act of automatically pre-processing comprises performing one or more of:

subtracting black offset from the RAW image, adjusting saturation in the RAW image, reducing noise from the RAW image, reducing stuck pixels from the RAW image, adjusting luminance in the RAW image, and demosaicing the RAW image.

67. The device of claim 63, wherein creating the interpolated image comprises creating the interpolated image having red, green, and blue values at each first pixel location, and wherein creating the blurred reduced-size image comprises:

creating from the RAW image a half-size image having red, green, and blue values at each pixel location, and

blurring the half-size image.

68. A processing device, comprising:

at least one processor;

a memory storing algorithms for pre-processing RAW images;

a device interface module for receiving data associated with a RAW image from an imaging device; and

an operating system executable on the at least one processor and configured to;

receive a RAW image and associated metadata obtained from an imaging device via the device interface module;

determine one or more of the algorithms based on the metadata;

automatically pre-process the RAW image with the one or more determined algorithms to produce a resulting image in a color space, the one or more algorithms determining first luminance values for each first pixel location of an interpolated image interpolated from the RAW image, creating a blurred reduced-size image from the RAW image, resizing the blurred reduced-size image into a blurred full-size image, determining second luminance values for each second pixel location of the blurred full-size image, and scaling the first luminance values in the interpolated image to match the second luminance values of the blurred full-size image; and

make the resulting image available to an application program executable on the processing device.

69. The device of claim 68, wherein the imaging device comprises a digital camera, and wherein the processing device comprises a computer.

70. The device of claim 68, wherein the operating system comprises an application programming interface providing the resulting image to any of a plurality of application programs executable on the processing device.

71. The device of claim 68, wherein the one or more algorithms perform one or more of:

subtracting black offset from the RAW image, adjusting saturation in the RAW image, reducing noise from the RAW image, reducing stuck pixels from the RAW image, adjusting luminance in the RAW image, and demosaicing the RAW image.

72. The device of claim 68, wherein to creating the interpolated image, the operating system creates the interpolated image having red, green, and blue values at each first pixel location, and wherein to create the blurred reduced-size image, the one or more algorithms create from the RAW image a half-size image having red, green, and blue values at each pixel location and blur the half-size image.