Systems and methods for raw image processing

• US 8,872,946 B2
• Filed: 05/31/2012
• Issued: 10/28/2014
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Lin, Sheng, Lim, Suk Hwan, Silverstein, D. Amnon, Hayward, David, Butler, Simon Wolfenden, Petolino, Joseph Anthony Jr., Bratt, Joseph P.
• Primary Examiner(s): SELBY, GEVELL V
• Application Number:
  US13/485,056
• Publication Number:
  US 20130321677A1
• Time in Patent Office: 880 Days
• Field of Search: 348/222.1, 348/241
• US Class Current: 348/241
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 25/63 applied to dark current

Abstract

Systems and methods for processing raw image data are provided. One example of such a system may include memory to store image data in raw format from a digital imaging device and an image signal processor to process the image data. The image signal processor may include data conversion logic and a raw image processing pipeline. The data conversion logic may convert the image data into a signed format to preserve negative noise from the digital imaging device. The raw image processing pipeline may at least partly process the image data in the signed format. The raw image processing pipeline may also include, among other things, black level compensation logic, fixed pattern noise reduction logic, temporal filtering logic, defective pixel correction logic, spatial noise filtering logic, lens shading correction logic, and highlight recovery logic.

Claims

1. An image signal processing system comprising:

data conversion logic configured to convert unsigned input image data deriving from a digital sensor into signed input image data to preserve negative noise from the sensor; and

a raw image processing pipeline configured to process the signed input image data into processed signed image data;

wherein;

the data conversion logic is configured to scale the signed input image data by a programmable scale value;

or

the data conversion logic is configured to right-bit-shift the signed input image data to scale the signed input image data by a programmable or non-programmable scale value;

or

the data conversion logic is configured to offset the signed input image data by subtracting an offset value to set a zero-bias in the signed input image data;

or

the data conversion logic comprises a function of input direct memory access logic;

or

the raw image processing pipeline is configured to process the signed input image data into the signed processed image data by overlapping tiles of a larger image frame captured by the sensor;

or

any combination thereof.

2. The image signal processing system of claim 1, wherein the data conversion logic is configured to convert the processed signed image data into unsigned output image data when the output image data from the raw image processing pipeline is output to memory.

3. The image signal processing system of claim 1, comprising logic in the data conversion logic, the raw image processing pipeline, or elsewhere in the image signal processing system, or a combination thereof, configured to write non-overlapping segments of the overlapping tiles of the larger image frame to memory as a single frame.

4. The image signal processing system of claim 1, wherein the unsigned input image data comprises raw image data from a digital camera which has been uncompressed.

5. An electronic device comprising:

memory configured to store image data in raw format deriving from a digital imaging device; and

an image signal processor comprising;

data conversion logic configured to convert the stored image data from an unsigned into a signed format to preserve negative noise from the digital imaging device; and

a raw image processing pipeline configured to process the image data in at least in part in the signed format, wherein the raw image processing pipeline comprises;

black level compensation logic configured to perform black level compensation on the image data;

fixed pattern noise reduction logic configured to reduce fixed pattern noise in the image data;

temporal filtering logic configured to temporally filter the image data;

defective pixel correction logic configured to detect and correct defective pixel noise in the image data;

spatial noise filtering logic configured to spatially filter noise in the image data;

lens shading correction configured to correct for lens shading in the image data; and

highlight recovery logic configured to recover some image data lost to clipping or near clipping by image processing from any of the above-described logic blocks.

6. The electronic device of claim 5, wherein the logic of the raw image processing pipeline is configured to process the image data in the recited order.

7. The electronic device of claim 5, comprising noise statistics logic configured to ascertain noise standard deviation in relation to brightness of the image data.

8. The electronic device of claim 5, wherein the noise statistics logic is configured to employ common logic with the defective pixel correction logic.

9. The electronic device of claim 5, wherein the noise statistics logic is configured to process image data before the spatial noise filtering logic.

10. The electronic device of claim 5, wherein the noise statistics logic is configured to process image data after the temporal filtering logic.

11. The electronic device of claim 5, comprising a plurality of noise statistics logical blocks configured to ascertain noise standard deviations in relation to brightness of the image data at various stages of processing through the raw image processing pipeline.

12. The electronic device of claim 5, wherein the raw image processing logic is configured to receive at least the following formats:

raw8, raw10, raw12, raw14, and raw 16.

13. The electronic device of claim 5, wherein the raw image processing logic is configured to output processed raw image data and reprocess the processed raw image data.

14. The electronic device of claim 5, wherein the imaging device comprises a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via an input/output port, or some combination thereof.

15. The electronic device of claim 5, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

16. A method for processing image data comprising:

receiving image data of raw format into a raw processing pipeline of an image signal processor;

performing black level compensation on the image data;

reducing fixed pattern noise occurring in the image data;

applying a temporal filter to the image data;

detecting and correcting defective pixels of the image data;

collecting noise statistics relating a noise standard deviation to intensities of the image data;

applying a spatial noise filter to the image data;

applying lens shading correction to the image data;

applying a white balance gain to the image data;

recovering at least some highlight information from the image data which was clipped or nearly clipped by previous image processing; and

at least partially correcting for chromatic aberration in the image data.

17. The method of claim 16, wherein the method is performed in the recited order.

18. The method of claim 16, wherein the noise statistics are collected immediately before the spatial noise filter is applied.

19. The method of claim 16, wherein the noise statistics are collected after temporal filtering.

20. The method of claim 16, wherein the noise statistics are collected a plurality of different times immediately after noise reduction processing is applied to the image data.

21. The method of claim 16, wherein detecting and correcting defective pixels occurs before the spatial noise filter is applied.