Highlight recovery in images

• US 10,070,109 B2
• Filed: 06/30/2016
• Issued: 09/04/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lim, Suk Hwan, Cao, Frederic, Tajbakhsh, Touraj
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US15/198,354
• Publication Number:
  US 20180007332A1
• Time in Patent Office: 796 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 9/643 Hue control means, e.g. flesh tone control
• CPC Class Codes: H04N 9/68 for controlling the amplitude of colour signals, e.g. automatic chroma control circuits H04N9/71, H04N9/73 take precedence; camera processing pipelines for controlling the colour saturation of colour signals H04N23/86

Abstract

Embodiments of the present disclosure relate to highlight recovery of a high-resolution image using a single low-resolution image captured at a lower exposure. An example apparatus includes a hue target circuit that receives an input image at a high-resolution including at least one pixel with a clipped color channel. For example, the input image is a Blue sky image with a pixel having clipped Blue channel. The hue target circuit also receives a set of candidate hue maps having a pixel resolution lower than the high-resolution of the input image. The hue target circuit generates a target hue value for the at least one pixel using the pixel information of the set of candidate hue maps. The apparatus also includes a hue recovery circuit that generates a recovered version of the input image by adjusting hue information of the clipped color channel based on the generated target hue.

Claims

1. An apparatus, comprising:

a hue target circuit configured to;

receive a first image capturing a scene at a first resolution comprising a pixel with hue information having a clipped color channel;

receive a plurality of candidate hue maps at a second resolution lower than the first resolution, each of the hue maps representing a different color channel of a second image capturing the scene at the second resolution; and

generate a target hue by processing the plurality of candidate hue maps, the target hue including information for recovering the clipped color channel of the pixel; and

a hue recovery circuit configured to generate a recovered version of the first image by adjusting the hue information of the pixel based on the generated target hue.

2. The apparatus of claim 1, wherein the second image is captured at an exposure level lower than an exposure level of the first image.

3. The apparatus of claim 1, wherein hue values of the plurality of candidate hue maps corresponding to color channels of pixels of the second image are not clipped.

4. The apparatus of claim 1, wherein the hue target circuit generates the target hue by:

upsampling the plurality of candidate hue maps into a plurality of upsampled candidate hue maps at the first resolution, and

wherein the target hue is generated based on the upsampled candidate hue maps.

5. The apparatus of claim 4, wherein the hue target circuit generates the target hue for the pixel by:

determining a weight for each upsampled candidate hue map based on how close the candidate hue value is relative to the clipped channel hue of the pixel, and

wherein the target hue is generated by aggregating weighted candidate hue values.

6. The apparatus of claim 5, wherein the hue target circuit further generates the target hue for the pixel by:

determining a conservative hue value for the pixel indicated in hue information of the received first image,

wherein the target hue is generated based on the conservative hue value and the aggregated weighted candidate hue values.

7. The apparatus of claim 6, wherein the hue target circuit generates the target hue value by mixing the conservative hue value with the aggregated weighted candidate hue values.

8. The apparatus of claim 1, wherein the hue recovery circuit generates the recovered version by:

determining possible recovered values for the pixel using ratio of color channel hue values of the target hues, and

wherein the hue information is adjusted based on the determined possible recovered values for the pixel.

9. The apparatus of claim 8, wherein the hue recovery circuit generates the recovered version by:

blending the possible recovered values with a hue value of the pixel of the received first image to generate a blended hue value, the blending based on how close the hue value of the pixel is relative to a maximum hue value of an image sensor capturing the first image, and

wherein the hue information is adjusted based on the blended hue value.

10. The apparatus of claim 9, wherein the hue recovery circuit generates the recovered version by:

performing soft clipping on the blended hue value to generate a soft clipped hue value, and

wherein the hue information is adjusted based on the soft clipped hue value.

11. The apparatus of claim 10, wherein the hue recovery circuit generates the recovered version by:

performing hue preservation on soft clipped hue value by recovering a hue value altered during the soft clipping to generate a hue preserved pixel value, and

wherein the hue information is adjusted based on the hue preserved pixel value.

19. The apparatus of claim 1, wherein the first image is a demosaiced version of an image captured by a sensor.

12. A method, comprising:

receiving a first image capturing a scene at a first resolution comprising a pixel with hue information having a clipped color channel;

receiving a plurality of candidate hue maps at a second resolution lower than the first resolution, each of the hue maps representing a different color channel of a second image capturing the scene at the second resolution;

generating a target hue by processing the plurality of candidate hue maps, the target hue including information for recovering the clipped color channel of the pixel; and

generating a recovered version of the first image by adjusting the hue information of the pixel based on the generated target hue.

13. The method of claim 12, wherein the second image is captured at an exposure level lower than an exposure level of the first image.

14. The method of claim 12, wherein hue values corresponding to color channels of pixels of the second image are not clipped.

15. The method of claim 12, wherein generating the target hue further comprises:

upsampling the plurality of candidate hue maps into a plurality of upsampled candidate hue maps at the first resolution, and

wherein the target hue is generated based on the upsampled candidate hue maps.

16. The method of claim 15, wherein generating the target value further comprises:

determining a weight for each upsampled candidate hue map based on how close the candidate hue value is relative to the clipped channel hue of the pixel, and

wherein the target hue is generated by aggregating weighted candidate hue values.

17. The method of claim 12, wherein generating the recovered version further comprises:

determining possible recovered values for the pixel using a ratio of color channel hue values of the target hues, and

wherein the hue information is adjusted based on the determined possible recovered values for the pixel.

20. The method of claim 12, wherein the first image is a demosaiced version of an image captured by a sensor.

18. A non-transitory computer-readable medium storing a digital representation of an apparatus, the apparatus comprising:

a hue target circuit configured to;

receive a first image capturing a scene at a first resolution comprising a pixel with hue information having a clipped color channel;

receive a plurality of candidate hue maps at a second resolution lower than the first resolution, each of the hue maps representing a different color channel of a second image capturing the scene at the second resolution; and

generate a target hue by processing the plurality of candidate hue maps, the target hue including information for recovering the clipped color channel of the pixel; and

a hue recovery circuit configured to generate a recovered version of the first image by adjusting the hue information of the pixel based on the generated target hue.