Dynamic Global Tone Mapping with Integrated 3D Color Look-up Table

• US 20170358063A1
• Filed: 06/13/2016
• Published: 12/14/2017
• Est. Priority Date: 06/13/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Chen, Ying, Chawla, Sumit, Edwards, Elizabeth Torres, Chandu, Kartheek, Ding, Bo
• Granted Patent: US 9,959,604 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20008 Globally adaptive
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 1/6005 with luminance or chrominance signals, e.g. LC1C2, HSL or YUV
• CPC Class Codes: H04N 1/6008 with primary colour signals, e.g. RGB or CMY(K)
• CPC Class Codes: H04N 1/6019 using look-up tables H04N1/6025 takes precedence
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 25/6153 for colour signals
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

The processing of RGB image data can be optimized by performing optimization operations on the image data when it is converted into the YCbCr color space. First, a raw RGB color space is converted into a YCbCr color space, and raw RGB image data is converted into YCbCr image data using the YCbCr color space. For each Y-layer of the YCbCr image data, a 2D LUT is generated. The YCbCr image data is converted into optimized CbCr image data using the 2D LUTs, and optimized YCbCr image data is generated by blending CbCr image data corresponding to multiple Y-layers. The optimized YCbCr image data is converted into sRGB image data, and a tone curve is applied to the sRGB image data to produce optimized sRGB image data.

Claims

1. A camera system, comprising:

an image sensor configured to convert light incident upon the image sensor into raw RGB image data;

an image signal processor (“

ISP”

) configured to;

access a raw RGB color space corresponding to the raw RGB image data;

convert the raw RGB color space into a YCbCr reference color space;

convert the raw RGB image data into YCbCr image data using the YCbCr reference color space;

generate, for each of a plurality of Y-layers of the YCbCr image data, a 2D color look-up table (“

LUT”

);

convert the YCbCr image data into optimized CbCr image data using the 2D LUTs;

generate optimized YCbCr image data by blending, for at least two Y-layers, the corresponding CbCr image data; and

converting the optimized YCbCr image data into tone-mapped optimized sRGB image data;

a memory configured to store the tone-mapped optimized sRGB image data.

2. The camera system of claim 1, wherein converting the raw RGB color space into a YCbCr reference color space comprises:

transforming the raw RGB color space into an enhanced RGB color space using a linear transformation.

3. The camera system of claim 2, wherein converting the raw RGB color space into a YCbCr reference color space further comprises:

converting the enhanced RGB color space into a non-linear RGB color space using a non-linear conversion model.

4. The camera system of claim 3, wherein converting the raw RGB color space into a YCbCr reference color space further comprises:

transforming the non-linear RGB color space into the YCbCr referenced color space in a xvYCC color space using a transformation matrix.

5. The camera system of claim 1, wherein generating a 2D color LUT comprises:

accessing a set of reference grid points; and

estimating a transformation matrix for each reference grid point for different Y-layers.

6. The camera system of claim 5, wherein converting the YCbCr image data into optimized CbCr image data comprises:

determining optimized CbCr points for each reference grid point for the different P-layers using the estimated transformation matrices; and

transforming the YCbCr image data into optimized CbCr image data using the optimized CbCr points.

7. The camera system of claim 6, wherein transforming the YCbCr image data into optimized CbCr image data using the optimized CbCr points comprises performing bilinear interpolation on the optimized CbCr points for different Y-layers.

8. The camera system of claim 1, wherein generating a 2D color LUT comprises:

accessing a target YCbCr color space;

determining an optimized chroma gain map for different Y-layers based on the YCbCr reference color space and the target YCbCr color space; and

determining an optimized hue shift map for different Y-layers.

9. The camera system of claim 8, wherein converting the YCbCr image data into optimized CbCr image data comprises:

converting the YCbCr image data into optimized CbCr image data using the optimized chroma gain maps and the optimized hue shift maps.

10. The camera system of claim 1, wherein converting the optimized YCbCr image data into tone-mapped optimized sRGB image data comprises:

converting the optimized YCbCr image data into sRGB image data; and

applying a tone curve to the sRGB image data to produce tone-mapped optimized sRGB image data.

11. A method, comprising:

converting, by a camera, light incident upon an image sensor of the camera into raw RGB image data;

accessing, by an image signal processor (“

ISP”

) of the camera, a raw RGB color space corresponding to the raw RGB image data;

converting, by the ISP of the camera, the raw RGB color space into a YCbCr reference color space;

converting, by the ISP of the camera, the raw RGB image data into YCbCr image data using the YCbCr reference color space;

generating, by the ISP of the camera, for each of a plurality of Y-layers of the YCbCr image data, a 2D color look-up table (“

LUT”

);

converting, by the ISP of the camera, the YCbCr image data into optimized CbCr image data using the 2D LUTs;

generating, by the ISP of the camera, optimized YCbCr image data by blending, for at least two Y-layers, the corresponding CbCr image data;

converting, by the ISP of the camera, the optimized YCbCr image data into tone-mapped optimized sRGB image data; and

storing, by a memory of the camera, the tone-mapped optimized sRGB image data.

12. The method of claim 11, wherein converting the raw RGB color space into a YCbCr reference color space comprises:

transforming the raw RGB color space into an enhanced RGB color space using a linear transformation.

13. The method of claim 12, wherein converting the raw RGB color space into a YCbCr reference color space further comprises:

converting the enhanced RGB color space into a non-linear RGB color space using a non-linear conversion model.

14. The method of claim 13, wherein converting the raw RGB color space into a YCbCr reference color space further comprises:

transforming the non-linear RGB color space into the YCbCr referenced color space in a xvYCC color space using a transformation matrix.

15. The method of claim 11, wherein generating a 2D color LUT comprises:

accessing a set of reference grid points; and

estimating a transformation matrix for each reference grid point for different Y-layers.

16. The method of claim 15, wherein converting the YCbCr image data into optimized CbCr image data comprises:

determining optimized CbCr points for each reference grid point for the different P-layers using the estimated transformation matrices; and

transforming the YCbCr image data into optimized CbCr image data using the optimized CbCr points.

17. The method of claim 16, wherein transforming the YCbCr image data into optimized CbCr image data using the optimized CbCr points comprises performing bilinear interpolation on the optimized CbCr points for different Y-layers.

18. The method of claim 11, wherein generating a 2D color LUT comprises:

accessing a target YCbCr color space;

determining an optimized chroma gain map for different Y-layers based on the YCbCr reference color space and the target YCbCr color space; and

determining an optimized hue shift map for different Y-layers.

19. The method of claim 18, wherein converting the YCbCr image data into optimized CbCr image data comprises:

converting the YCbCr image data into optimized CbCr image data using the optimized chroma gain maps and the optimized hue shift maps.

20. The method of claim 11, wherein converting the optimized YCbCr image data into tone-mapped optimized sRGB image data comprises:

converting the optimized YCbCr image data into sRGB image data; and

applying a tone curve to the sRGB image data to produce tone-mapped optimized sRGB image data.