Display-side adaptive video processing

• US 10,271,054 B2
• Filed: 02/25/2015
• Issued: 04/23/2019
• Est. Priority Date: 02/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Greenebaum, Kenneth I., Guo, Haitao, Pan, Hao, Cote, Guy, Bai, Andrew
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Wu, Yanna
• Application Number:
  US14/631,394
• Publication Number:
  US 20150245043A1
• Time in Patent Office: 1,518 Days
• Field of Search: None
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2320/0271 Adjustment of the gradation levels within the range of the gradation scale, e.g. by redistribution or clipping
• CPC Class Codes: G09G 2320/0613 The adjustment depending on the type of the information to be displayed
• CPC Class Codes: G09G 2320/062 Adjustment of illumination source parameters
• CPC Class Codes: G09G 2320/0626 for control of overall brightness
• CPC Class Codes: G09G 2320/066 for control of contrast
• CPC Class Codes: G09G 2320/0666 for control of colour parameters, e.g. colour temperature
• CPC Class Codes: G09G 2320/0673 for control of gamma adjustment, e.g. selecting another gamma curve
• CPC Class Codes: G09G 2320/0686 with two or more screen areas displaying information with different brightness or colours
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 2320/08 Arrangements within a display terminal for setting, manually or automatically, display parameters of the display terminal
• CPC Class Codes: G09G 2320/103 Detection of image changes, e.g. determination of an index representative of the image change
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 2370/042 for monitor identification
• CPC Class Codes: G09G 3/2007 Display of intermediate tones
• CPC Class Codes: G09G 5/005 Adapting incoming signals to the display format of the display terminal
• CPC Class Codes: G09G 5/02 : characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: G09G 5/10 : Intensity circuits
• CPC Class Codes: H04N 1/6066 : dependent on the gamut of the image to be reproduced
• CPC Class Codes: H04N 1/64 : Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor H04N19/00 takes precedence
• CPC Class Codes: H04N 19/102 : characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/124 : Quantisation
• CPC Class Codes: H04N 19/136 : Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/137 : Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 : Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/154 : Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/17 : the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/182 : the unit being a pixel
• CPC Class Codes: H04N 19/184 : the unit being bits, e.g. of the compressed video stream
• CPC Class Codes: H04N 19/186 : the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/30 : using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/52 : by predictive encoding
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/98 : Adaptive-dynamic-range coding [ADRC]
• CPC Class Codes: H04N 21/42202 : environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4318 : by altering the content in the rendering process, e.g. blanking, blurring or masking an image region image enhancement or restoration in general G06T5/00
• CPC Class Codes: H04N 21/44008 : involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4402 : involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4854 : for modifying image parameters, e.g. image brightness, contrast
• CPC Class Codes: H04N 23/741 : by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 5/20 : Circuitry for controlling amplitude response
• CPC Class Codes: H04N 5/505 : Invisible or silent tuning
• CPC Class Codes: H04N 9/641 : Multi-purpose receivers, e.g. for auxiliary information H04N9/642 takes precedence
• CPC Class Codes: H04N 9/67 : for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

Adaptive video processing for a target display panel may be implemented in or by a decoding/display pipeline associated with the target display panel. The adaptive video processing methods may take into account video content, display characteristics, and environmental conditions including but not limited to ambient lighting and viewer location when processing and rendering video content for a target display panel in an ambient setting or environment. The display-side adaptive video processing methods may use this information to adjust one or more video processing functions as applied to the video data to render video for the target display panel that is adapted to the display panel according to the ambient viewing conditions.

Claims

1. A system, comprising:

one or more sensors configured to detect one or more current environmental conditions;

a display panel; and

a decoding pipeline configured to;

receive encoded video data;

decode the encoded video data according to a dynamic range capability of the display panel to generate video content adapted to the dynamic range capability of the display panel;

process the video content to generate processed video content adapted for viewing on the display panel under the one or more current environmental conditions, wherein the video content is processed according to one or more characteristics of the video content, one or more characteristics of the display panel, the one or more current environmental conditions as detected by the one or more sensors, and a field of view that the display panel subtends, wherein the one or more current environmental conditions detected by the one or more sensors include a location of a human viewer with respect to that the display panel, wherein the field of view is determined from the location of the human viewer, and wherein to process the video content, the decoding pipeline is configured to;

determine a measured response luminance range for the display panel according to the display panel characteristics;

determine an adapted human vision luminance range according to the one or more current environmental conditions;

map the video content to the measured response luminance range to generate intermediate video content; and

map the intermediate video content to the adapted human vision luminance range to generate the video content adapted for viewing on the display panel under the one or more current environmental conditions; and

output the processed video content to the display panel for display.

2. The system as recited in claim 1, wherein the decoding pipeline is configured to process the video content according to one or more video processing techniques, wherein at least one of the one or more video processing techniques is adjusted according to the one or more current environmental conditions to generate the video content adapted for viewing on the display panel under the one or more current environmental conditions.

3. The system as recited in claim 2, wherein the one or more video processing techniques include one or more of noise reduction, artifact reduction, scaling, sharpening, frame rate conversion, color space conversion, color gamut adjustment, dynamic backlight adjustment, panel gamma correction, black point correction, or white point correction.

4. The system as recited in claim 1, wherein, to process the video content, the decoding pipeline is configured to apply color gamut mapping to the video content to map color components of the video content into a color space and bit depth of the display panel, and wherein the decoding pipeline is configured to adjust the color gamut mapping as applied to the video content according to the one or more current environmental conditions to generate the video content adapted for viewing on the display panel under the one or more current environmental conditions.

5. The system as recited in claim 4, wherein, to adjust the color gamut mapping as applied to the video content according to the one or more current environmental conditions, the decoding pipeline is configured to modify a color lookup table used in the color gamut mapping according to the one or more current environmental conditions.

6. The system as recited in claim 1, wherein, to process the video content, the decoding pipeline is configured to apply tone mapping to the video content to map luminance of the video content into a color space and bit depth of the display panel, and wherein the decoding pipeline to adjust the tone mapping as applied to the video content according to the one or more current environmental conditions to generate the video content adapted for viewing on the display panel under the one or more current environmental conditions.

7. The system as recited in claim 6, wherein, to adjust the tone mapping as applied to the video content according to the one or more current environmental conditions, the decoding pipeline is configured to adjust one or more tone curves used in the tone mapping according to the one or more current environmental conditions.

8. The system as recited in claim 1, wherein the decoding pipeline is further configured to analyze the video content to determine the one or more characteristics of the video content, wherein the one or more characteristics of the video content include dynamic range and color characteristics of the video content.

9. The system as recited in claim 1, wherein the one or more current environmental conditions include ambient lighting metrics.

10. The system as recited in claim 1, wherein the one or more characteristics of the display panel include one or more measured characteristics for the display panel, wherein the one or more measured characteristics include one or more of measured response, native white point, light leakage, reflective light, or backlight levels for the display panel.

11. The system as recited in claim 1, wherein the one or more characteristics of the display panel include one or more of color gamut, dynamic range, bit depth, or physical dimensions of the display panel.

12. The system as recited in claim 1, wherein the video content is standard dynamic range (SDR) video content or high dynamic range (HDR), wherein the display panel is a high dynamic range (HDR)-enabled display panel, and wherein the processed video content is HDR video content.

13. A method, comprising:

performing, by a decoding module implemented by one or more hardware processors of a device;

receiving encoded video data;

determining one or more characteristics of a target display panel;

decoding the encoded video data according to a dynamic range capability of the target display panel to generate video content adapted to the dynamic range capability of the target display panel;

determining one or more characteristics of the video content;

obtaining one or more current environmental conditions;

processing the video content to generate processed video content adapted for viewing on the target display panel under one or more current environmental conditions, wherein the video content is processed according to one or more video processing techniques, wherein said processing comprises adjusting at least one of the one or more video processing techniques according to the one or more characteristics of the video content, the one or more characteristics of the target display panel characteristics, the obtained one or more current environmental conditions, and a field of view that the target display panel subtends, wherein the obtained one or more current environmental conditions include a location of a human viewer with respect to the target display panel, and wherein the field of view is determined from the location of the human viewer, and

wherein processing the video content comprises;

mapping the video content to a measured response luminance range of the target display panel to generate intermediate video content; and

mapping the intermediate video content to an adapted human vision luminance range according to the one or more current environmental conditions to generate the video content adapted for viewing on the target display panel under the one or more current environmental conditions; and

providing the processed video content to the target display panel for display.

14. The method as recited in claim 13, wherein the one or more video processing techniques include a color gamut mapping technique and a tone mapping technique that map the video content into a color space and bit depth of the target display panel, and wherein processing the video content comprises adjusting the color gamut mapping and the tone mapping as applied to the video content according to the one or more current environmental conditions to generate the video content adapted for viewing on the target display panel under the one or more current environmental conditions.

15. The method as recited in claim 13, wherein the one or more current environmental conditions include ambient lighting metrics.

16. The method as recited in claim 13, wherein the video content is one of standard dynamic range (SDR) or high dynamic range (HDR) video content, wherein the target display panel is an HDR-enabled display panel, and wherein the processed video content is HDR video content.

17. An apparatus, comprising:

an encoding module implemented by one or more hardware processors and configured to encode input video data represented at a high dynamic range to generate encoded video data; and

a decoding module implemented by one or more hardware processors and configured to;

decode the encoded video data according to a dynamic range capability of a display panel to generate video content adapted to the dynamic range capability of the display panel;

obtain one or more current environmental conditions;

process the video content to generate processed video content adapted for viewing on a display panel under the one or more current environmental conditions, wherein the video content is processed according to one or more characteristics of the video content, one or more characteristics of the display panel, the obtained one or more current environmental conditions, and a field of view that the display panel subtends, wherein the one or more current environmental conditions include a location of a viewer with respect to the display panel as obtained by one or more sensors of the apparatus, wherein the field of view is determined from the location of the viewer, and wherein to process the video content, the decoding module is configured to;

map the video content to a measured response luminance range of the target display panel to generate intermediate video content and

map the intermediate video content to an adapted human vision luminance range according to the one or more current environmental conditions to generate the video content adapted for viewing on the target display panel under the one or more current environmental conditions; and

output the processed video content to the display panel for display.

18. The apparatus as recited in claim 17, wherein, to process the video content, the decoding module is further configured to map the video content into a color space and bit depth of the target display panel according to the one or more characteristics of the video content, the one or more characteristics of the display panel, and the one or more current environmental conditions to generate the video content adapted for viewing on the display panel under the one or more current environmental conditions.