Rendering and displaying high dynamic range content

• US 10,249,263 B2
• Filed: 06/03/2016
• Issued: 04/02/2019
• Est. Priority Date: 06/05/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hendry, Ian C., Greenebaum, Kenneth I., Dacquay, Eric L.
• Primary Examiner(s): Pringle-Parker, Jason A
• Application Number:
  US15/173,425
• Publication Number:
  US 20160358346A1
• Time in Patent Office: 1,033 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G09G 2320/0626 for control of overall brightness
• CPC Class Codes: G09G 2320/066 for control of contrast
• CPC Class Codes: G09G 2340/06 Colour space transformation
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 5/02 characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: G09G 5/10 Intensity circuits
• CPC Class Codes: H04N 1/6061 involving the consideration or construction of a gamut surface
• CPC Class Codes: H04N 1/6088 by viewing conditions, i.e. conditions at picture output
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/98 Adaptive-dynamic-range coding [ADRC]
• CPC Class Codes: H04N 5/202 Gamma control circuits for controlling camera response irrespective of the scene brightness H04N23/82
• CPC Class Codes: H04N 5/58 in dependence upon ambient light

Abstract

Methods and apparatus for rendering and displaying high dynamic range (HDR) digital image content. An HDR rendering and display system may support the rendering and display of standard dynamic range (SDR) and HDR content to both HDR-enabled and standard displays. The HDR rendering and display system renders digital image content into the HDR space and maps the rendered HDR content into the display space of HDR or standard displays using display processing techniques that may preserve at least some of the HDR content even for standard displays. The HDR rendering and display system may take into account various information including but not limited to display characteristics such as size, control inputs, current image characteristics such as image brightness, and environmental information such as viewer position and ambient lighting levels to dynamically adapt the rendering and display of the digital image content according to ambient viewing conditions at the target display.

Claims

1. A system, comprising:

a display panel with a display space defined by bit depth of the display panel, the display space comprising N codes for representing pixel values; and

one or more processors configured to implement;

a rendering pipeline configured to render received digital image content according to a maximum rendering value M to generate high dynamic range (HDR) content in a dynamic range of (0.0-M); and

a display pipeline configured to;

obtain the rendered HDR content;

map the rendered HDR content into the display space of the display panel according to a brightness level B that defines a lower portion (codes 0 to n) and an upper portion (codes n to (N−

1)) of the display space;

wherein, to map the rendered HDR content into the display space of the display panel, the display pipeline is configured to;

map a first portion of the rendered HDR content in a standard range (0.0-1.0) into codes in the lower portion of the display space; and

map a second portion of the rendered HDR content in an extended range (1.0-M) into codes in the upper portion of the display space; and

output the mapped HDR content to the display panel for display.

2. The system as recited in claim 1, wherein the one or more processors are further configured to implement a dynamic display module that iteratively performs:

determine a current brightness level B according to one or more of an ambient lighting level or a brightness control setting;

determine a current maximum rendering value M according to B;

provide the current maximum rendering value M to the rendering pipeline, wherein said rendering is adjusted according to the current maximum rendering value M; and

provide the current brightness level B to the display pipeline, wherein said mapping is adjusted according to the current brightness level B.

3. The system as recited in claim 2, further comprising one or more sensors configured to detect one or more environmental conditions of the display panel including the ambient lighting level.

4. The system as recited in claim 2, wherein the brightness control setting is determined according to input from a brightness control user interface element that allows brightness to be set below a minimum brightness level of a display brightness control of the display panel.

5. The system as recited in claim 1, wherein the display pipeline is further configured to:

receive standard dynamic range (SDR) content from an SDR content source; and

composite or blend the SDR content with the standard range (0.0-1.0) portion of the HDR content in the codes in the lower portion of the display space.

6. The system as recited in claim 5, wherein the display panel comprises a display backlight, and wherein the display pipeline is further configured to:

dynamically adjust brightness level of the backlight up or down according one or more of an ambient lighting level or a brightness control setting; and

adjust at least some pixel values of the HDR content according to the brightness level of the backlight to maintain apparent brightness of the HDR content at different backlight levels, wherein pixel values of the SDR content that is composited or blended with the HDR content are not adjusted.

7. The system as recited in claim 1,

wherein the rendering pipeline is further configured to encode the rendered HDR content according to an encoding format to generate encoded HDR content;

and

wherein the display pipeline is further configured to decode the encoded HDR content prior to said mapping.

8. A method, comprising:

obtaining, by a display pipeline implemented on a device, high dynamic range (HDR) content rendered in a dynamic range of (0.0-M) by one or more rendering pipelines, where M is a maximum rendering value;

mapping, by the display pipeline, the rendered HDR content obtained from the one or more rendering pipelines into a display space of a display panel according to a brightness level B that defines a lower portion (codes 0 to n) and an upper portion (codes n to (N−

1)) of the display space, wherein the display space is defined by bit depth of the display panel and comprises N codes for representing pixel values, wherein said mapping comprises;

mapping a first portion of the rendered HDR content in a standard range (0.0-1.0) into codes in the lower portion of the display space; and

mapping a second portion of the rendered HDR content in an extended range (1.0-M) into codes in the upper portion of the display space; and

providing the mapped HDR content to the display panel for display.

9. The method as recited in claim 8, further comprising iteratively performing:

determining a current brightness level B according to one or more of an ambient lighting level or a brightness control setting;

determining a current maximum rendering value M according to B;

providing the current maximum rendering value M to the one or more rendering pipelines, wherein the one or more rendering pipelines adjust rendering according to the current maximum rendering value M; and

providing the current brightness level B to the display pipeline, wherein said mapping is adjusted according to the current brightness level B.

10. The method as recited in claim 8, further comprising:

determining a model of viewer perceptual range under current ambient conditions of the display panel according to a perceptual model, wherein the perceptual model determines the model of viewer perceptual range according to inputs including one or more of ambient lighting conditions, display panel characteristics, or display panel settings; and

determining the brightness level B, the lower portion and the upper portion of the display space, and the maximum rendering value M according to the model of viewer perceptual range.

11. The method as recited in claim 10, wherein said determining the maximum rendering value M comprises determining a maximum value for M that does not exceed one or more thresholds according to the perceptual model.

12. The method as recited in claim 8, further comprising:

maintaining a backlight brightness setting of the display panel at a level at or near maximum brightness for the backlight; and

adjusting the brightness level B to provide a digital dimming effect on the display panel.

13. The method as recited in claim 8, further comprising determining the brightness level B, the lower portion and an upper portion of the display space, and the maximum rendering value M at least in part according to measured light leakage from the display panel and measured reflected ambient light off the display panel.

14. The method as recited in claim 8, further comprising:

receiving, by the display pipeline, standard dynamic range (SDR) content from an SDR content source; and

compositing or blending the SDR content with the standard range (0.0-1.0) portion of the HDR content in the codes in the lower portion of the display space.

15. The method as recited in claim 14, further comprising:

receiving highlight HDR content in the SDR content; and

compositing or blending the highlight HDR content in the codes in the upper portion of the display space to highlight portions of the SDR content on the display panel.

16. The method as recited in claim 14, further comprising:

dynamically adjusting brightness level of a backlight of the display up or down according one or more of an ambient lighting level or a brightness control setting; and

adjusting at least some pixel values of the HDR content according to the brightness level of the backlight to maintain apparent brightness of the HDR content at different backlight levels, wherein pixel values of the SDR content that is composited or blended with the HDR content are not adjusted.

17. An apparatus, comprising:

one or more processors configured to implement;

an encoder configured to render input digital image content according to a maximum rendering value M to generate high dynamic range (HDR) content in a dynamic range of (0.0-M) and encode the rendered HDR content to generate encoded HDR content; and

a decoder configured to;

decode encoded HDR content to generate HDR content in the dynamic range (0.0-M);

map the HDR content into a display space of a display panel according to a brightness level B that defines a lower portion (codes 0 to n) and an upper portion (codes n to (N−

1)) of the display space, wherein the display space is defined by bit depth of the display panel and comprises N codes for representing pixel values;

wherein, to map the rendered HDR content into the display space of the display panel, the display pipeline is configured to;

map a first portion of the rendered HDR content in a standard range (0.0-1.0) into codes in the lower portion of the display space; and

map a second portion of the rendered HDR content in an extended range (1.0-M) into codes in the upper portion of the display space; and

output the mapped HDR content to the display panel for display.

18. The apparatus as recited in claim 17, wherein the one or more processors are further configured to implement a dynamic display module that iteratively performs:

determine a current brightness level B according to one or more of an ambient lighting level or a brightness control setting;

determine a current maximum rendering value M according to B, wherein B is expressed as a percentage, and wherein M is determined as an inverse of B;

provide the current maximum rendering value M to the encoder, wherein said rendering is adjusted according to the current maximum rendering value M; and

provide the current brightness level B to the decoder, wherein said mapping is adjusted according to the current brightness level B.

19. The apparatus as recited in claim 17, wherein the decoder is further configured to:

receive standard dynamic range (SDR) content;

receive highlight HDR content;

composite or blend the SDR content with the standard range (0.0-1.0) portion of the HDR content in the codes in the lower portion of the display space; and

composite or blend the highlight HDR content in the codes in the upper portion of the display space to highlight portions of the SDR content on the display panel.

20. The apparatus as recited in claim 17, wherein the decoder is further configured to:

dynamically adjust brightness level of a backlight of the display up or down according one or more of an ambient lighting level or a brightness control setting; and

adjust at least some pixel values of the HDR content according to the brightness level of the backlight to maintain apparent brightness of the HDR content at different backlight levels, wherein pixel values of the SDR content that is composited or blended with the HDR content are not adjusted.