SYSTEMS AND METHODS FOR RGB IMAGE PROCESSING

• US 20150296193A1
• Filed: 05/31/2012
• Published: 10/15/2015
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Lin, Sheng, Lim, Suk Hwan, Silverstein, D. Amnon, Hordley, Steven David, Bratt, Joseph P.
• Granted Patent: US 9,332,239 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 5/009 Global, i.e. based on properties of the image as a whole applied in cameras using an electronic image sensor H04N23/80, H04N23/70
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration
• CPC Class Codes: H04N 25/67 applied to fixed-pattern noise, e.g. non-uniformity of response
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Systems and methods for processing image data in RGB format are provided. In one example, an electronic device includes memory to store image data in raw or RGB format, or both, and an RGB image processing pipeline to process the image data. Specifically, the RGB image processing pipeline may process the image data regardless of whether the image data is of raw or RGB format. The RGB image processing pipeline may include receiving logic to receive the image data in raw or RGB format and demosaicing logic to, when the receiving logic receives the image data in raw format, convert the image data into RGB format. The logic may include local tone mapping logic configured to apply spatially varying tone curves to the image data, a color correction matrix configured to correct color in the image data, and gamma logic configured to transform the image data into gamma space.

Claims

1. An image signal processing system comprising:

data conversion logic configured to convert unsigned input image data deriving from a digital image sensor into signed input image data to preserve negative noise from the sensor; and

an RGB-format image processing pipeline configured to process the signed input image data into processed signed RGB output image data.

2. The image signal processing system of claim 1, wherein the input image data is in raw or RGB format and the RGB-format image processing pipeline is configured to receive both raw and RGB-format signed input image data.

3. The image signal processing system of claim 1, wherein the data conversion logic is configured to scale the signed input image data by a programmable scale value.

4. The image signal processing system of claim 3, wherein, when the signed input image data comprises RGB-format input image data, the data conversion logic is configured to apply the programmable scale value equally to all three color components of the signed input image data.

5. The image signal processing system of claim 1, wherein the data conversion logic is configured to right-bit-shift the signed input image data to scale the signed input image data by a programmable or non-programmable scale value.

6. The image signal processing system of claim 1, wherein the data conversion logic is configured to offset the signed input image data by subtracting an offset value to set a zero-bias in the signed input image data.

7. The image signal processing system of claim 6, wherein, when the signed input image data comprises RGB-format input image data, the data conversion logic is configured to apply the offset value equally to all three color components of the signed input image data.

8. The image signal processing system of claim 1, wherein the data conversion logic is configured to convert the signed RGB output image data into unsigned RGB output image data when the output of the RGB-format image processing pipeline is output to memory.

9. The image signal processing system of claim 1, wherein the RGB-format image processing pipeline is configured to process the signed input image data in overlapping tiles of a larger image frame.

10. The image signal processing system of claim 9, comprising logic in the data conversion logic, the RGB-format image processing pipeline, or elsewhere in the image signal processing system, or a combination thereof, configured to write non-overlapping segments of the overlapping tiles of the larger image frame to memory as a single frame

11. An electronic device comprising:

memory configured to store image data in raw or RGB format, or both;

an RGB image processing pipeline configured to process the image data regardless of whether the image data is of raw or RGB format, wherein the RGB image processing pipeline comprises;

receiving logic configured to receive the image data in raw or RGB format;

demosaicing logic configured to, when the receiving logic receives the image data in raw format, convert the image data into RGB format;

local tone mapping logic configured to apply spatially varying tone curves to the image data;

a color correction matrix configured to correct color in the image data; and

gamma logic configured to transform the image data into gamma space.

12. The electronic device of claim 11, wherein the logic of the RGB image processing pipeline is configured to process the image data in the recited order.

13. The electronic device of claim 11, comprising an imaging device comprising an image sensor configured to obtain the image data, wherein the RGB image processing logic comprises a color lookup table configured to adjust color of the image data to cause colors from image data obtained by different sensors to generally appear to be the same regardless of sensor manufacturer.

14. The electronic device of claim 13, wherein the lookup table comprises a three-dimensional color lookup table.

15. The electronic device of claim 14, wherein the three-dimensional color lookup table is configured to interpolate color-correction values using tetrahedral interpolation.

16. The electronic device of claim 11, wherein the RGB image processing logic is configured to receive at least the following formats:

raw8, raw10, raw12, rawl4, rawl6, RGB565, RGB888, and RGB16.

17. The electronic device of claim 11, wherein the RGB image processing logic is configured to output processed RGB image data and reprocess the processed RGB image data.

18. The electronic device of claim 11, wherein the RGB image processing logic comprises color space conversion logic to convert the image data into a YCC format.

19. The electronic device of claim 11, comprising a raw-format image processing pipeline configured to perform initial raw processing on the raw image data and provide the processed raw image data to the memory or to the RGB image processing pipeline.

20. The electronic device of claim 11, comprising an imaging device, wherein the imaging device comprises a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via an input/output port, or some combination thereof.

21. The electronic device of claim 11, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

22. An image signal processing system comprising:

raw image processing logic configured to process raw image data deriving from an image sensor, wherein the raw image data processing logic comprises at least highlight recovery logic and raw scaling logic; and

RGB image processing logic configured to receive the processed raw image data and perform RGB image processing operations in RGB format, wherein the RGB image data processing logic comprises at least demosaicing logic and local tone mapping logic.

23. The image signal processing logic of claim 22, wherein the local tone mapping logic is separated from the highlight recovery logic substantially only by image processing blocks consisting of the raw scaling logic and the demosaicing logic.

24. The image signal processing logic of claim 22, wherein at least one logical component of the raw image processing logic and the RGB image processing logic are configured to operate on signed image data.

25. A method for processing image data comprising:

receiving image data of raw or RGB format into a pipeline of an image signal processor;

when the image data is of raw format, converting the image data into RGB format image data using demosaicing logic;

applying a first gain, offset, or clamp, or a combination thereof, to the image data;

applying an RGB color correction matrix to the image data;

passing the image data through a 3D color lookup table;

applying a second gain, offset, or clamp, or a combination thereof, to the image data; and

transforming the image data into gamma space.

26. The method of claim 25, wherein the method is performed in the recited order.

27. The method of claim 26, comprising obtaining an RGB histogram using the image data immediately after the second gain, offset, clamp, or combination thereof is applied to the image data or immediately after the image data is transformed into gamma space.

28. The method of claim 25, wherein the image data is transformed into gamma space before the second gain, offset, or clamp, or combination thereof is applied and before the image data is passed though the 3D color lookup table.

29. The method of claim 25, comprising applying a spatially varying local tone map and a spatially varying color correction matrix to the image data:

before the first offset, gain, or clamp, or combination thereof, is applied;

or

before the RGB color correction matrix is applied;

or

before the first offset, gain, or clamp, or combination thereof, is applied and

before the RGB color correction matrix is applied.