Architecture for rendering high dynamic range video on enhanced dynamic range display devices

• US 10,218,952 B2
• Filed: 12/20/2016
• Issued: 02/26/2019
• Est. Priority Date: 11/28/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wozniak, Matthew, Kucukgoz, Mehmet, Sarkar, Abhijit
• Primary Examiner(s): Caschera, Antonio A
• Application Number:
  US15/385,711
• Publication Number:
  US 20180152684A1
• Time in Patent Office: 798 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: H04N 11/24 High-definition television systems
• CPC Class Codes: H04N 5/20 Circuitry for controlling amplitude response
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85
• CPC Class Codes: H04N 9/68 for controlling the amplitude of colour signals, e.g. automatic chroma control circuits H04N9/71, H04N9/73 take precedence; camera processing pipelines for controlling the colour saturation of colour signals H04N23/86

Abstract

Innovations in rendering of high dynamic range (“HDR”) video on a display device having enhanced dynamic range (“EDR”). The peak brightness for an EDR display device is lower than the peak brightness for a typical HDR display device but higher than the peak brightness for a typical display device having standard dynamic range. The increased range of brightness values in an EDR display device can be utilized effectively to show bright highlights of the HDR video. For example, decision logic is configured to evaluate a peak brightness of a target display device and select an HDR-to-EDR tone mapping mode. A tone mapper is configured to apply tone mapping to input values for the sample values of HDR video, according to a tone mapping function, thereby producing output values for sample values of EDR video.

Claims

1. A computer system comprising one or more processing units and memory, wherein the computer system implements a video playback system that includes:

decision logic configured to;

receive an indication of a peak brightness of a target display device;

determine if the peak brightness of the target display device is above a first peak brightness threshold and below a second peak brightness threshold, wherein the first peak brightness threshold is defined to be at or above a peak brightness of 100 nits, for a standard dynamic range (“

SDR”

) display device, and wherein the second peak brightness threshold is defined to be at or below a peak brightness of 4000 nits, for a high dynamic range (“

HDR”

) display device; and

if so, select an HDR-to-enhanced dynamic range (“

EDR”

) tone mapping mode;

a streaming controller configured to request encoded data for HDR video;

an input buffer configured to store the encoded data;

a video decoder configured to decode the encoded data, and thereby produce sample values of HDR video; and

a tone mapper configured to apply tone mapping to input values for the sample values of HDR video, according to a tone mapping function that is based at least in part on the peak brightness of the target display device, and thereby produce output values for sample values of EDR video.

2. The computer system of claim 1, wherein, to determine if the peak brightness of the target display device is above the first peak brightness threshold, the decision logic is configured to:

compare the peak brightness of the target display device to the first peak brightness threshold, wherein the first peak brightness threshold is selected from the group consisting of 100 nits, 200 nits, and 300 nits.

3. The computer system of claim 1, wherein, to determine if the peak brightness of the target display device is below the second peak brightness threshold, the decision logic is configured to:

compare the peak brightness of the target display device to the second peak brightness threshold, wherein the second peak brightness threshold is selected from the group consisting of 800 nits and 1000 nits.

4. The computer system of claim 1, wherein the target display device has a screen area, and wherein the decision logic is further configured to:

determine a proportion of the screen area that is to be used for video rendering; and

compare the proportion of the screen area to a screen area threshold, wherein the selection of the HDR-to-EDR tone mapping mode is also based at least in part on whether the proportion of the screen area is at or above the screen area threshold.

5. The computer system of claim 1, wherein the decision logic is further configured to:

determine whether a brightness level of the target display device can be programmatically controlled, wherein the selection of the HDR-to-EDR tone mapping mode is also based at least in part on whether the brightness level of the target display device can be programmatically controlled.

6. The computer system of claim 1, wherein the decision logic is further configured to:

determine whether the target display device is in a power-saving mode, wherein the selection of the HDR-to-EDR tone mapping mode is also based at least in part on whether the target display device is in the power-saving mode.

7. The computer system of claim 1, wherein the tone mapping function is also based at least in part on one or more of:

peak brightness of HDR video;

metadata provided by the video decoder;

an indication of ambient light in an environment around the target display device;

content type of the sample values of HDR video; and

analysis of content of the sample values of HDR video.

8. The computer system of claim 1, wherein the HDR-to-EDR tone mapping mode is selected from among multiple available tone mapping modes.

9. The computer system of claim 8, wherein the decision logic is further configured to:

if the peak brightness of the target display device is below the first peak brightness threshold, select an HDR-to-SDR tone mapping mode, among the multiple available tone mapping modes, in which the input values for the sample values of HDR video are mapped to output values for sample values of SDR video; and

if the peak brightness of the target display device is above the second peak brightness threshold, select an HDR-to-HDR tone mapping mode, among the multiple available tone mapping modes, in which the input values for the sample values of HDR video are not adjusted or are mapped to output values for sample values of HDR video.

10. The computer system of claim 8, wherein the decision logic is further configured to:

if the peak brightness of the target display device is below the first peak brightness threshold, cause the streaming controller to request encoded data for SDR video; and

if the peak brightness of the target display device is above the second peak brightness threshold, select an HDR-to-HDR tone mapping mode, among the multiple available tone mapping modes, in which the input values for the sample values of HDR video are not adjusted or are mapped to output values for sample values of HDR video.

11. The computer system of claim 1, wherein the video playback system further includes:

a color converter configured to convert the sample values of HDR video from an initial color space to an intermediate color space, and thereby produce intensity values and color values in the intermediate color space.

12. The computer system of claim 11, wherein the initial color space is a YUV-type color space, and the intermediate color space is a perceptually uniform, hue linear color space.

13. The computer system of claim 11, wherein the input values for the tone mapping are the intensity values in the intermediate color space but not the color values in the intermediate color space, and wherein the output values from the tone mapping are tone-mapped intensity values in the intermediate color space.

14. The computer system of claim 13, wherein the tone mapper or another module of the video playback system is configured to:

adjust at least some of the color values in the intermediate color space depending on a ratio of original intensity value to tone-mapped intensity value.

15. The computer system of claim 13, wherein the video playback system further includes:

a second color converter configured to convert the tone-mapped intensity values and the color values from the intermediate color space to a final color space for the sample values of EDR video.

16. The computer system of claim 1, wherein the tone mapper is configured to apply the tone mapping on a region-by-region basis within a picture.

17. The computer system of claim 1, wherein the tone mapper is configured to apply the tone mapping throughout a picture.

18. The computer system of claim 1, wherein:

a first segment of the tone mapping function begins at a minimum point and ends at a first switch point, wherein input values along the first segment are boosted according to a first non-linear mapping while range of the input values along the first segment is expanded;

a second segment of the tone mapping function begins at the first switch point and ends at a second switch point, wherein input values along the second segment are boosted by an offset according to a linear mapping; and

a third segment of the tone mapping function begins at the second switch point and ends at a maximum point, wherein at least some input values along the third segment are boosted according to a second non-linear mapping while range of the input values along the third segment is compressed.

19. In a computer system, a method comprising:

receiving an indication of a peak brightness of a target display device;

determining that the peak brightness of the target display device is above a first peak brightness threshold and below a second peak brightness threshold, wherein the first peak brightness threshold is defined to be at or above a peak brightness of 100 nits, for a standard dynamic range (“

SDR”

) display device, and wherein the second peak brightness threshold is defined to be at or below a peak brightness of 4000 nits, for a high dynamic range (“

HDR”

) display device;

selecting an HDR-to-enhanced dynamic range (“

EDR”

) tone mapping mode;

receiving encoded data for HDR video;

decoding the encoded data, thereby producing sample values of HDR video; and

applying tone mapping to input values for the sample values of HDR video, according to a tone mapping function that is based at least in part on the peak brightness of the target display device, thereby producing output values for sample values of EDR video.

20. One or more computer-readable media storing computer-executable instructions for causing a computer system, when programmed thereby, to perform operations comprising:

receiving an indication of a peak brightness of a target display device;

determining that the peak brightness of the target display device is above a first peak brightness threshold and below a second peak brightness threshold, wherein the first peak brightness threshold is defined to be at or above a peak brightness of 100 nits, for a standard dynamic range (“

SDR”

) display device, and wherein the second peak brightness threshold is defined to be at or below a typical peak brightness of 4000 nits, for a high dynamic range (“

HDR”

) display device;

selecting an HDR-to-enhanced dynamic range (“

EDR”

) tone mapping mode;

receiving encoded data for HDR video;

decoding the encoded data, thereby producing sample values of HDR video; and

applying tone mapping to input values for the sample values of HDR video, according to a tone mapping function that is based at least in part on the peak brightness of the target display device, thereby producing output values for sample values of EDR video.