Content-adaptive adjustments to tone mapping operations for high dynamic range content

• US 10,176,561 B2
• Filed: 01/27/2017
• Issued: 01/08/2019
• Est. Priority Date: 01/27/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Evans, Glenn F., Kucukgoz, Mehmet, Wozniak, Matthew
• Primary Examiner(s): Woldemariam, Aklilu
• Application Number:
  US15/418,530
• Publication Number:
  US 20180218481A1
• Time in Patent Office: 711 Days
• Field of Search: 382131, 382166, 382168, 382274, 348362, 348 36, 348 47, 348 48, 345690, 345589, 345211
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G09G 2320/0271 Adjustment of the gradation levels within the range of the gradation scale, e.g. by redistribution or clipping
• CPC Class Codes: G09G 2320/0646 Modulation of illumination source brightness and image signal correlated to each other
• CPC Class Codes: G09G 2320/0673 for control of gamma adjustment, e.g. selecting another gamma curve
• CPC Class Codes: G09G 2330/021 Power management, e.g. power saving
• CPC Class Codes: G09G 2360/08 Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 2360/16 Calculation or use of calculated indices related to luminance levels in display data
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
• CPC Class Codes: G09G 3/342 using several illumination sources separately controlled corresponding to different display panel areas, e.g. along one dimension such as lines
• CPC Class Codes: G09G 5/02 characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: G09G 5/10 Intensity circuits
• CPC Class Codes: G09G 5/36 characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
• CPC Class Codes: H04N 5/20 Circuitry for controlling amplitude response
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
• CPC Class Codes: H04N 9/68 for controlling the amplitude of colour signals, e.g. automatic chroma control circuits H04N9/71, H04N9/73 take precedence; camera processing pipelines for controlling the colour saturation of colour signals H04N23/86

Abstract

Innovations in rendering of high dynamic range (“HDR”) content are described. A playback system can adjust a tone mapping function used when rendering the HDR content, thereby improving quality when rendering the HDR content. For example, the playback system includes decision logic, which is configured to receive summary information (e.g., histogram information) for sample values of a picture of HDR content, and set a tone mapping function based at least in part on the summary information. The tone mapping function can be selected from among multiple available tone mapping functions, and parameters of the selected tone mapping function can be determined based at least in part on the summary information. A tone mapper is configured to apply tone mapping to input values for the sample values of the picture of HDR content, according to the tone mapping function, and thereby produce output values.

Claims

1. A computer system comprising one or more processing units and memory, wherein the computer system implements a playback system that is to be connected to a target display device having a brightness level that is dynamically controllable by the playback system, and wherein the playback system includes:

decision logic implemented by software executable on one or more central processing units, among the one or more processing units, by software controlling special-purpose hardware, or by special-purpose hardware, wherein the decision logic is configured to;

receive histogram information for sample values of a picture of high dynamic range (“

HDR”

) content, wherein the histogram information for the sample values of the picture of HDR content is organized as multiple bins, each of the multiple bins having a count of the sample values of the picture of HDR content within that bin, and wherein each of the multiple bins represents a range of possible sample values between a minimum possible sample value for the bin and a maximum possible sample value for the bin;

set a tone mapping function based at least in part on the histogram information for the sample values of the picture of HDR content;

send, to the target display device, a control signal that indicates a brightness level of the target display device for display of the picture of HDR content; and

after tone mapping, send, to the target display device, the sample values of the picture of HDR content; and

a tone mapper configured to, as part of rendering the picture of HDR content for output to the target display device, apply tone mapping to input values for the sample values of the picture of HDR content, according to the tone mapping function, and thereby produce output values for the sample values of the picture of HDR content.

2. The computer system of claim 1, wherein the multiple bins have variable size in terms of range of possible sample values in the multiple bins, respectively.

3. The computer system of claim 2, wherein sizes of the multiple bins are perceptually weighted and define steps that are at least approximately uniform.

4. The computer system of claim 1, wherein, to set the tone mapping function, the decision logic is configured to:

select a highest bin, among the multiple bins, satisfying a threshold count; and

set a maximum point of the tone mapping function based at least in part on the selected bin.

5. The computer system of claim 1, wherein, to set the tone mapping function, the decision logic is configured to:

select the tone mapping function, from among multiple available tone mapping functions, based at least in part on the histogram information.

6. The computer system of claim 1, wherein the decision logic is further configured to buffer parameters of the tone mapping function, wherein the parameters of the tone mapping function are paired with the sample values of the picture of HDR content and/or with encoded data for the picture of HDR content.

7. In a computer system that implements a playback system connected to a target display device having a brightness level that is dynamically controllable by the playback system, a method comprising:

producing sample values of a picture of high dynamic range (“

HDR”

) content;

generating, from the sample values of the picture of HDR content, histogram information for the sample values of the picture of HDR content, wherein the histogram information for the sample values of the picture of HDR content is organized as multiple bins, each of the multiple bins having a count of the sample values of the picture of HDR content within that bin, and wherein each of the multiple bins represents a range of possible sample values between a minimum possible sample value for the bin and a maximum possible sample value for the bin;

setting a tone mapping function based at least in part on the histogram information for the sample values of the picture of HDR content;

as part of rendering the picture of HDR content for output to the target display device, with a tone mapper, applying tone mapping to input values for the sample values of the picture of HDR content, according to the tone mapping function, and thereby producing output values for the sample values of the picture of HDR content;

sending, to the target display device, a control signal that indicates a brightness level of the target display device for display of the picture of HDR content; and

after the tone mapping, sending, to the target display device, the sample values of the picture of HDR content.

8. The method of claim 7, wherein the setting the tone mapping function includes:

selecting the tone mapping function, from among multiple available tone mapping functions, based at least in part on the histogram information.

9. The method of claim 8, wherein the multiple available tone mapping functions include a generic tone mapping function having first, second, and third segments, and wherein:

the first segment begins at a minimum point, ends at a first switch point, and maps input values along the first segment to output values along the first segment according to a first non-linear mapping;

the second segment begins at the first switch point, ends at a second switch point, and maps input values along the second segment to output values along the second segment according to a linear mapping; and

the third segment begins at the second switch point, ends at a maximum point, and maps input values along the third segment to output values along the third segment according to a second non-linear mapping.

10. The method of claim 9, wherein the setting the tone mapping function further includes:

setting one or more of the first switch point and the second switch point based at least in part on the histogram information.

11. The method of claim 9, wherein the second segment is configured such that the input values along the second segment are boosted by an offset according to the linear mapping, and wherein the setting the tone mapping function further includes setting the offset based at least in part on the histogram information.

12. The method of claim 8, wherein the multiple available tone mapping functions include a dark-to-mid-range tone mapping function having first and second segments, and wherein:

the first segment begins at a minimum point, ends at a switch point, and maps input values along the first segment to output values along the first segment according to a non-linear mapping; and

the second segment begins at the switch point, ends at a maximum point, and maps input values along the second segment to output values along the second segment according to a linear mapping.

13. The method of claim 12, wherein the dark-to-mid range tone mapping function is selected if a maximum sample value among the sample values of the picture of HDR content satisfies a threshold dark sample value but not a threshold bright sample value.

14. The method of claim 8, wherein the multiple available tone mapping functions include a simplified tone mapping function that begins at a minimum point, ends at a maximum point, and maps input values along the simplified tone mapping function to output values along the simplified tone mapping function according to a linear mapping.

15. The method of claim 14, wherein the simplified tone mapping function is selected if:

at least a threshold proportion of the sample values of the picture of HDR content are within a middle range of possible sample values;

a power-saving mode is active; and

/or

a GPU-optimized mode is active.

16. The method of claim 8, wherein the multiple available tone mapping functions include a dark-only tone mapping function that begins at a minimum point, ends at a maximum point, and maps input values along the dark-only tone mapping function to output values along the dark-only tone mapping function according to a non-linear mapping.

17. The method of claim 16, wherein the dark-only tone mapping function is selected if a maximum sample value among the sample values of the picture of HDR content satisfies a threshold dark sample value.

20. The method of claim 7, wherein the multiple bins have variable size in terms of range of possible sample values in the multiple bins, respectively.

18. One or more computer-readable media storing computer-executable instructions for causing a computer system, when programmed thereby, to perform operations, the computer system including a target display device having a brightness level that is dynamically controllable by the playback system, the operations comprising, for each of multiple pictures of high dynamic range (“

HDR”

) content in a video sequence;

decoding encoded data for the picture of HDR content, thereby producing sample values of the picture of HDR content;

generating histogram information for the sample values of the picture of HDR content, wherein the histogram information for the sample values of the picture of HDR content is organized as multiple bins, each of the multiple bins having a count of the sample values of the picture of HDR content within that bin, and wherein each of the multiple bins represents a range of possible sample values between a minimum possible sample value for the bin and a maximum possible sample value for the bin;

setting a tone mapping function based at least in part on the histogram information for the sample values of the picture of HDR content;

as part of rendering the picture of HDR content for output to the target display device, with a tone mapper, applying tone mapping to input values for the sample values of the picture of HDR content, according to the tone mapping function, and thereby producing output values for the sample values of the picture of HDR content;

sending, to the target display device, a control signal that indicates a brightness level of the target display device for display of the picture of HDR content; and

after the tone mapping, sending, to the target display device, the sample values of the picture of HDR content.

19. The or more computer-readable media of claim 18, wherein the multiple bins have variable size in terms of range of possible sample values in the multiple bins, respectively.