User interface and graphics composition with high dynamic range video

• US 9,973,723 B2
• Filed: 09/30/2014
• Issued: 05/15/2018
• Est. Priority Date: 02/24/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Guo, Haitao, Greenebaum, Kenneth I., Cote, Guy, Singer, David W., Tourapis, Alexandros
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): ROBINSON, TERRELL M
• Application Number:
  US14/502,373
• Publication Number:
  US 20150245004A1
• Time in Patent Office: 1,323 Days
• Field of Search: 345629
• CPC Class Codes: G09G 2340/06 Colour space transformation
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence
• CPC Class Codes: G09G 5/363 Graphics controllers
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/42653 for processing graphics
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast
• CPC Class Codes: H04N 5/46 for receiving on more than one standard at will deflecting circuits of multi-standard receivers H04N3/27

Abstract

A method and system for adaptively mixing video components with graphics/UI components, where the video components and graphics/UI components may be of different types, e.g., different dynamic ranges (such as HDR, SDR) and/or color gamut (such as WCG). The mixing may result in a frame optimized for a display device'"'"'s color space, ambient conditions, viewing distance and angle, etc., while accounting for characteristics of the received data. The methods include receiving video and graphics/UI elements, converting the video to HDR and/or WCG, performing statistical analysis of received data and any additional applicable rendering information, and assembling a video frame with the received components based on the statistical analysis. The assembled video frame may be matched to a color space and displayed. The video data and graphics/UI data may have or be adjusted to have the same white point and/or primaries.

Claims

1. A computer-implemented compositing method, comprising:

receiving, by a processor, first data from a first source having a first dynamic range and an associated first electro-optical transfer function (EOTF);

receiving, by the processor, second data, from a second source different than the first source, having a second dynamic range that is lower than the first dynamic range and an associated second electro-optical transfer function;

converting, by the processor, the first data to a converted first data having a third dynamic range and an associated third electro-optical transfer function that matches at least a segment of the second electro-optical transfer function;

assembling, by the processor, a video frame from the converted first data and the second data, and

rendering, by the processor, an output frame based on the assembled video frame.

2. The method of claim 1, further comprising converting, by the processor, the second data to a fourth dynamic range at least partially overlapping with the third dynamic range.

3. The method of claim 1, wherein the conversion of the first data retains at least one of a white point and primary chromaticities of the first data.

4. The method of claim 1, wherein the conversion of the first data includes converting the first data to one of a high dynamic range (HDR) and a wide color gamut (WCG).

5. The method of claim 1, wherein the converted first data and the second data share at least one primary chromaticity.

6. The method of claim 1, wherein the EOTFs are represented by a gamma curve.

7. The method of claim 1, wherein the assembly of the video frame is based on at least one of:

(i) characteristics of the first data and the second data, (ii) ambient conditions of a display device, and (iii) commands received at a user interface of the display device.

8. The method of claim 1, further comprising performing statistical analysis based on additional rendering information relating to a display device and includes a level of light leakage, a level of light reflection, a backlight level, a physical size, a white point, and an average pixel brightness.

9. The method of claim 1, further comprising matching the assembled frame to a color space of the display device.

10. The method of claim 1, wherein the rendering of the output frame includes at least one of:

storing the output frame in a non-transitory computer-readable medium, outputting the output frame to a network, and displaying the output frame on a display device.

11. The method of claim 1, wherein the second data has a EOTF matching at least a segment of a EOTF of the first dynamic range.

12. A video compositor, comprising:

a first analyzer operating on one or more processors to perform statistical analysis on a first video sequence with an associated first electro-optical transfer function representing natural video content;

a second analyzer operating on the one or more processors to perform statistical analysis on a second video sequence with an associated electro-optical second transfer function representing synthetic video content;

a controller operating on the one or more processors, responsive to data representing output conditions, configured to;

convert one of the video sequences from an original dynamic range domain to a converted domain having a third electro-optical transfer function (EOTF) curve that matches a shape of the curve of the electro-optical transfer function associated with the other video sequence;

blend the converted video sequence with the other, non-converted video sequence; and

adjust at least a color component of the synthetic video component to match the native video component.

13. The video compositor of claim 12, wherein the output condition is characteristics of the display device.

14. The video compositor of claim 12, wherein the output condition is an ambient viewing condition about a display.

15. The video compositor of claim 12, wherein the color component is a brightness level.

16. The video compositor of claim 12, wherein the color component is a luminance level.

17. A video composition pipeline for processing a pair of video components, each from different sources and associated with different electro-optical transfer functions, a first video component having a higher dynamic range than the second video component, comprising:

a first data converter configured to convert the first video component having the higher dynamic range to a color space different from its original color space; and

a composition unit configured to;

convert the first video component to a third dynamic range, the third dynamic range being higher than the second dynamic range, wherein electro-optical transfer functions (EOTFs) associated with the converted first component and the second component are of complementary shapes;

assemble a video frame based on the converted first component and the second component; and

render an output frame based on the assembled video frame.

18. The video composition pipeline of claim 17, further comprising a second data converter configured to convert the video component having the lower dynamic range to a color space different from its original color space.

19. The video composition pipeline of claim 17, wherein the first component is of a higher dynamic range (HDR) and the second component is of an HDR different from the first component.

20. The video composition pipeline of claim 17, wherein

the second component is of a standard dynamic range (sRGB) format and

a color space of the first component is of a range [−

a, b], where a and b are each integer values, the range being programmable and greater than the sRGB format.

21. The video composition pipeline of claim 17, wherein the assembly of the received components is based on ambient conditions including at least one of:

a level of light leakage, a level of light reflection, a backlight level, a physical size of a display, a white point, an average pixel brightness of the display, and at least one color of light reflected by the display.

22. The video composition pipeline of claim 21, wherein the level of light leakage is measured by a photo detector facing a viewer of the display.

23. The video composition pipeline of claim 21, wherein, based on the ambient conditions, the assembly of the video frame includes at least one of:

shading the frame, performing local tone mapping, and altering lookup tables for mapping one color space to another.

24. The video composition pipeline of claim 17, wherein the evaluation of the received components is based on a mode of operation specifiable by a user, wherein the mode of operation overrides other bases for the assembly of the received components.

25. The video composition pipeline of claim 17, wherein the assembly of the video frame is based on at least one position of a viewer relative to the display.