Metadata describing nominal lighting conditions of a reference viewing environment for video playback

• US 10,368,105 B2
• Filed: 05/04/2016
• Issued: 07/30/2019
• Est. Priority Date: 06/09/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sullivan, Gary J.
• Primary Examiner(s): Jones, Heather R
• Application Number:
  US15/146,800
• Publication Number:
  US 20160366444A1
• Time in Patent Office: 1,182 Days
• Field of Search: None
• CPC Class Codes: H04N 19/134 characterised by the element, parameter or criterion affecting or controlling the adaptive coding
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

Innovations in the areas of generating, parsing, and using metadata that describes nominal lighting conditions of a reference viewing environment for video playback are presented herein. In various examples described herein, metadata includes parameters that describe the nominal lighting conditions (e.g., level of ambient light, color characteristics of ambient light) of a reference viewing environment. By conveying a representation of the nominal lighting conditions of the reference viewing environment (e.g., one assumed when mastering image content), a transmitter system can enable a receiver system to adapt its local display of the image content. Upon receiving image content and the metadata, the receiver system can identify characteristics of the actual viewing environment, use the metadata to determine whether the actual viewing environment matches the reference viewing environment, and, if not, adjust sample values of the image content, adjust a display device, or adjust lighting conditions of the actual viewing environment.

Claims

1. A computing system comprising:

a buffer configured to receive video;

a pre-processor configured to master content of the video by selectively adjusting at least some sample values of the video based on one or more of (a) characteristics of a reference display device, and (b) one or more nominal lighting conditions of a reference viewing environment in place when mastering the content of the video;

a metadata generator configured to generate metadata that describes the one or more nominal lighting conditions of the reference viewing environment in place when mastering the content of the video, wherein the metadata includes;

one or more parameters that specify a nominal level of ambient light in the reference viewing environment, the one or more parameters that specify the nominal level of ambient light including an indicator of light per unit of area in units of lux; and

one or more parameters that specify a nominal color characteristic of the ambient light in the reference viewing environment, the one or more parameters that specify the nominal color characteristic of ambient light including normalized x and y chromaticity coordinates in a multi-dimensional mapping of values in a color space; and

a buffer configured to store the metadata for output with encoded data for the video, wherein the metadata is organized for output as part of a video elementary bitstream for the video, the video elementary bitstream also including the encoded data for the video.

2. The computing system of claim 1, further comprising:

a video encoder configured to encode the video to produce the encoded data for the video.

3. The computing system of claim 1, wherein the indicator specifies the nominal level of ambient light in units of 0.0001 lux, and wherein the normalized x and y chromaticity coordinates are expressed in increments of 0.00002.

4. The computing system of claim 1, wherein the one or more parameters that specify the nominal color characteristic of the ambient light further include a value in a color temperature scale.

5. The computing system of claim 1, wherein the metadata further includes one or more parameters that specify one or more of:

a nominal level of light reflected by the reference display device in the reference viewing environment; and

one or more color characteristics of the light reflected by the reference display device in the reference viewing environment.

6. The computing system of claim 1, wherein the metadata further includes one or more parameters that specify, for the one or more nominal lighting conditions of the reference viewing environment, adjustments, if any, to reach a final lighting state from an initial lighting state of a hypothetical viewing environment.

7. The computing system of claim 1, wherein the metadata further includes multiple sets of parameters, and wherein each of the multiple sets of parameters:

is associated with a different hypothetical viewing environment having an initial lighting state; and

includes one or more parameters that specify, for the one or more nominal lighting conditions of the reference viewing environment, adjustments, if any, to reach a final lighting state from the initial lighting state of the hypothetical viewing environment for that set of parameters.

8. The computing system of claim 1, wherein the reference viewing environment includes multiple regions, and wherein the metadata includes, for each of the multiple regions, one or more parameters that specify those of the one or more nominal lighting conditions that apply for the region, the multiple regions including a first region and a second region, the first region being behind an expected position of a viewer in the reference viewing environment, the second region being in front of the expected position of the viewer in the reference viewing environment.

9. The computing system of claim 1, wherein the metadata is organized as part of:

a supplemental enhancement information (“

SEI”

) message in the video elementary bitstream for the video;

or

a video usability information (“

VUI”

) message in the video elementary bitstream for the video.

10. The computing system of claim 1, wherein the metadata is organized as part of a supplemental enhancement information (“

SEI”

) message that is part of the video elementary bitstream for the video, and wherein persistence scope for the SEI message is coded layer-wise video sequence (“

CLVS”

), such that the SEI message persists, for a current layer in decoding order, from a current picture until end of the CLVS.

11. In a computing system, a method comprising:

receiving metadata that describes one or more nominal lighting conditions of a reference viewing environment in place when mastering content of video, wherein the metadata is received as part of a video elementary bitstream for the video, the video elementary bitstream also including encoded data for the video, and wherein the metadata includes;

one or more parameters that specify a nominal level of ambient light in the reference viewing environment, the one or more parameters that specify the nominal level of ambient light including an indicator of light per unit of area in units of lux; and

one or more parameters that specify a nominal color characteristic of the ambient light in the reference viewing environment, the one or more parameters that specify the nominal color characteristic of ambient light including normalized x and y chromaticity coordinates in a multi-dimensional mapping of values in a color space; and

parsing the metadata.

12. The method of claim 11, further comprising:

determining one or more actual lighting conditions of an actual viewing environment; and

to compensate for differences between the one or more actual lighting conditions and the one or more nominal lighting conditions, adjusting a characteristic of at least some sample values of the video.

13. The method of claim 12, wherein the one or more actual lighting conditions include an actual level of ambient light in the actual viewing environment, and wherein the adjusting comprises:

if the nominal level of ambient light is brighter than the actual level of ambient light, decreasing brightness of the at least some sample values of the video; and

if the nominal level of ambient light is darker than the actual level of ambient light, increasing brightness of the at least some sample values of the video.

14. The method of claim 12, wherein the one or more actual lighting conditions include an actual color characteristic of ambient light in the actual viewing environment, and wherein the adjusting comprises scaling or attenuating the at least some sample values of the video to compensate for differences between the actual color characteristic and the nominal color characteristic.

15. The method of claim 12, wherein the determining the one or more actual lighting conditions includes measuring at least one of the one or more actual lighting conditions.

16. The method of claim 12, further comprising:

receiving user input that specifies a value for at least one of the one or more actual lighting conditions; and

storing the specified value in a storage area;

wherein the determining the one or more actual lighting conditions includes looking up the specified value in the storage area.

17. The method of claim 11, further comprising:

determining one or more actual lighting conditions of an actual viewing environment; and

to compensate for differences between the one or more actual lighting conditions and the one or more nominal lighting conditions, adjusting at least one of the one or more actual lighting conditions of the actual viewing environment;

wherein the metadata includes multiple sets of parameters, and wherein each of the multiple sets of parameters;

is associated with a different hypothetical viewing environment having an initial lighting state; and

includes one or more parameters that specify, for the one or more nominal lighting conditions of the reference viewing environment, adjustments, if any, to reach a final lighting state from the initial lighting state of the hypothetical viewing environment for that set of parameters.

18. The method of claim 17, further comprising:

based at least in part on the one or more actual lighting conditions of the actual viewing environment, selecting one of the multiple sets of parameters whose different hypothetical viewing environment is closest to the actual viewing environment; and

for the adjusting, performing adjustments, if any, for the selected set of parameters to reach the final lighting state.

19. The method of claim 11, further comprising:

receiving, as part of the video elementary bitstream, the encoded data for the video; and

decoding the encoded data for the video.

20. The method of claim 11, wherein the metadata is organized as part of a supplemental enhancement information (“

SEI”

) message that is part of the video elementary bitstream for the video, and wherein persistence scope for the SEI message is coded layer-wise video sequence (“

CLVS”

), such that the SEI message persists, for a current layer in decoding order, from a current picture until end of the CLVS.

21. The method of claim 11, wherein the indicator specifies the nominal level of ambient light in units of 0.0001 lux, and wherein the normalized x and y chromaticity coordinates are expressed in increments of 0.00002.

22. A computing system comprising:

a buffer configured to store metadata that describes one or more nominal lighting conditions of a reference viewing environment in place when mastering content of video, wherein the metadata is part of a video elementary bitstream for the video, the video elementary bitstream also including encoded data for the video, and wherein the metadata includes;

one or more parameters that specify a nominal level of ambient light in the reference viewing environment, the one or more parameters that specify the nominal level of ambient light including an indicator of light per unit of area in units of lux; and

one or more parameters that specify a nominal color characteristic of the ambient light in the reference viewing environment, the one or more parameters that specify the nominal color characteristic of ambient light including normalized x and y chromaticity coordinates in a multi-dimensional mapping of values in a color space;

a metadata parser configured to parse the metadata;

a buffer configured to store the encoded data for the video;

a video decoder configured to decode the encoded data for the video;

a condition detector configured to determine one or more actual lighting conditions of an actual viewing environment; and

a post-processor configured to compensate for differences between the one or more actual lighting conditions and the one or more nominal lighting conditions by adjusting a characteristic of at least some sample values of the video.

23. The computing system of claim 22, wherein the indicator specifies the nominal level of ambient light in units of 0.0001 lux, and wherein the normalized x and y chromaticity coordinates are expressed in increments of 0.00002.

24. The computing system of claim 22, wherein the metadata is organized as part of a supplemental enhancement information (“

SEI”

) message that is part of the video elementary bitstream for the video, and wherein persistence scope for the SEI message is coded layer-wise video sequence (“

CLVS”

), such that the SEI message persists, for a current layer in decoding order, from a current picture until end of the CLVS.