SYSTEM AND METHOD FOR PROCESSING IMAGE DATA USING AN IMAGE SIGNAL PROCESSOR HAVING BACK-END PROCESSING LOGIC

• US 20120081385A1
• Filed: 09/30/2010
• Published: 04/05/2012
• Est. Priority Date: 09/30/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Côté, Guy, Frederiksen, Jeffrey E.
• Granted Patent: US 8,786,625 B2
• Time in Patent Office: Days
• US Class Current: 345/589
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/74 Circuitry for scanning or addressing the pixel array

Abstract

Disclosed embodiments provide for a an image signal processing system that includes back-end pixel processing unit that receives pixel data after being processed by at least one of a front-end pixel processing unit and a pixel processing pipeline. In certain embodiments, the back-end processing unit receives luma/chroma image data and may be configured to apply face detection operations, local tone mapping, bright, contrast, color adjustments, as well as scaling. Further, the back-end processing unit may also include a back-end statistics unit that may collect frequency statistics. The frequency statistics may be provided to an encoder and may be used to determine quantization parameters that are to be applied to an image frame.

Claims

1. An image signal processing system comprising:

an image processing pipeline configured to receive a frame of raw image data comprising pixels acquired using a digital image sensor and to process the raw image frame by applying demosaicing to produce a corresponding frame of full color RGB image data and subsequently convert the RGB image data into a corresponding frame of luma and chroma image data; and

a back-end processing unit configured to receive the frame of luma and chroma image data, and to process the luma and chroma image frame using at least one of feature detection logic, tone mapping logic, and scaling logic.

2. The image signal processing system of claim 1, wherein the feature detection logic comprises face detection logic configured to detect locations within the received frame containing facial features, and to provide the detected locations as a set of facial detection data.

3. The image signal processing system of claim 2, wherein the face detection logic is configured to detect the locations of the facial features using an object detection algorithm.

4. The image signal processing system of claim 1, wherein the feature detection logic is configured to detect locations within the received frame that correspond to corners of objects in order to determine global between the received frame and at least one prior or subsequent frame.

5. The image signal processing system of claim 2, comprising a front-end processing unit coupled upstream from the image processing pipeline and having a front-end statistics collection engine configured process the raw image frame to determine at least one of auto-white balance statistics, auto-exposure statistics, auto-focus statistics, and flicker detection statistics prior to the raw image frame being processed by the image processing pipeline and the back-end processing unit.

6. The image signal processing system of claim 5, wherein the facial detection data is fed back to the front-end statistics collection engine, and wherein the front-end statistics collection engine is configured to position one or more statistics collection windows for determining auto-white balance statistics using the facial detection feedback data.

7. The image signal processing system of claim of claim 1, wherein the digital image sensor comprises a Bayer color filter array, and wherein the raw image data comprises Bayer image data.

8. A method for processing image data using back-end processing logic of an image signal processing (ISP) system comprising:

receiving an image frame comprising luma and chroma image data, the image frame having been previously processed by an image processing pipeline of the ISP system;

applying local tone mapping to the image frame using tone mapping logic; and

using scaling logic to scale the image frame to at least one desired output resolution after applying local tone mapping.

9. The method of claim 8, wherein applying local tone mapping to the image frame comprises dividing the image frame into sections based on local features within the image frame and processing each of the sections by:

identifying a total available dynamic range of the image frame;

identifying a total available output dynamic range;

determining the actual dynamic range of luma values in a current section of the image frame;

expanding the actual dynamic range of the current section by mapping the actual dynamic range to the total available dynamic range if the actual dynamic range is less than the total available dynamic range; and

mapping the expanded actual dynamic range to the total available output dynamic range.

10. The method of claim 9, wherein applying local tone mapping comprises using local tone mapping operators.

11. The method of claim 10, wherein local tone mapping is applied in at least one of an RGB linear color space, a Lab color space, or some combination thereof.

12. The method of claim 8, comprising applying at least one of brightness, contrast, and color (BCC) adjustments to the image frame using BCC adjustment logic after applying local tone mapping but before applying scaling using the scaling logic.

13. The method of claim 8, wherein the at least one desired output resolution comprises a first output resolution and a second output resolution, wherein the first output resolution corresponds to a desired viewing resolution of a display device configured to display the image frame, and wherein the second output resolution corresponds to a preview resolution that is lower than the desired viewing resolution.

14. An image signal processing system comprising:

a back-end pixel processing unit configured to receive a frame of image data, wherein the back-end pixel processing unit comprises;

face detection logic configured to receive and process the image frame and to output set of facial detection data indicative of the location of facial features within the image frame;

tone mapping logic configured to applying tone mapping to the image frame;

scaling logic configured to scale the image frame to one or more desired output resolutions; and

a back-end statistics processing unit configured to process the scaled image data to collect frequency statistics for the scaled image data; and

encoder/decoder logic configured to apply compression to the scaled image data based at least partially upon the frequency statistics.

15. The image signal processing system of claim 14, wherein the back-end statistics processes the scaled image data as a plurality of macroblocks, and wherein collecting the frequency statistics comprises determining a frequency parameter indicative of the frequency content for each macroblock.

16. The image signal processing system of claim 15, wherein the frequency parameter is determined for each macroblock using at least one of a discrete cosine transform, a fast Fourier transform, or wavelet compression, or some combination thereof.

17. The image signal processing system of claim 15, wherein, for each macroblock, the encoder/decoder logic modulates at least one quantization parameter based upon the frequency parameter associated with the macroblock.

18. The image signal processing system of claim 14, wherein the facial detection data is provided to at least one of the back-end statistics processing unit and the encoder/decoder logic, and wherein the encoder/decoder logic is configured to reduce quantization for macroblocks in the image frame that correspond to the locations of facial features, as indicated by the facial detection data.

19. The image signal processing system of claim 14, wherein the back-end pixel processing unit comprises selection logic configured to receive both the image frame and at least one scaled image frame from the scaling unit, and wherein the selection logic is configured select one of either the image frame or the scaled image frame and to provide the selected image frame to the face detection logic.

20. The image signal processing system of claim 14, wherein the encoder comprises an H.264/JPEG encoder.

21. An electronic device comprising:

a digital image sensor;

a sensor interface configured to communicate with the digital image sensor;

a memory device;

a display device configured to display a visual representation of an image scene corresponding to raw image data acquired by the digital image sensor; and

an image signal processing sub-system comprising;

an image processing pipeline configured to receive a frame of raw image data comprising pixels acquired using the digital image sensor, demosaic the raw image frame to produce a corresponding full color RGB image frame, and convert full color RGB image data into a corresponding luma and chroma image frame; and

a back-end processing unit configured to receive and process the luma and chroma image frame using at least one of face detection logic, local tone mapping logic, and scaling logic, and to collect image statistics using a back-end statistics engine;

wherein the image statistics include frequency statistics which may be used to determine quantization parameters for encoding the luma and chroma image frame using an encoder.

22. The electronic device of claim 21, wherein processing the luma and chroma image frame using the local tone mapping logic comprises first converting the luma and chroma frame from the luma and chroma color space into an at least approximately linear RGB color space, and applying local tone mapping to the approximately linear RGB color space.

23. The electronic device of claim 22, wherein converting the luma and chroma data into the at least approximately linear RGB color space comprises applying an approximated transform based upon a power function.

24. The electronic device of claim 23, wherein if the luma and chroma data comprises luma and chroma data in a YCC 4:

2;

2 or YCC 4;

2;

0 sub-sampled data format, the luma and chroma data is first upconverted to a YCC 4;

4;

4 full data format before being converted into the at least approximately linear RGB color space.

25. The electronic device of claim 22, wherein converting the luma and chroma data into the at least approximately linear RGB color space comprises converting the luma and chroma data to an sRGB color space, and then converting the sRGB data to an sRGB_linear color space.

26. The electronic device of claim 21, wherein the digital image sensor comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via an input/output port, or some combination thereof.

27. The electronic device of claim 21, wherein the sensor interface comprises a Standard Mobile Imaging Architecture (SMIA) interface, a serial imaging interface, a parallel imaging interface, or some combination thereof.

28. The electronic device of claim 21, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

29. The electronic device of claim 21, wherein the image statistics comprise motion statistics, and wherein the back-end processing unit is configured to perform scene detection based upon the motion statistics.