Selective upsampling or refresh of chroma sample values

• US 10,368,080 B2
• Filed: 02/21/2017
• Issued: 07/30/2019
• Est. Priority Date: 10/21/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Holcomb, Thomas W., Owunwanne, Chinweizu E., Andrews, Matthew
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Nirjhar, Nasim N
• Application Number:
  US15/438,507
• Publication Number:
  US 20180115776A1
• Time in Patent Office: 889 Days
• CPC Class Codes: H04N 1/646 Transmitting or storing colour television type signals, e.g. PAL, Lab; Their conversion into additive or subtractive colour signals or vice versa therefor H04N1/642, H04N1/644 take precedence
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Approaches to delivering video in a chroma sampling format with a higher chroma sampling rate (such as a YUV 4:4:4 format) using a video encoder and decoder that operate on video in another chroma sampling format with a lower chroma sampling rate (such as YUV 4:2:0) are described. A video decoder can recover stationary content in video at the higher chroma sampling rate, even when the video is encoded and decoded at the lower chroma sampling rate, without significantly increasing bit rate. In some example implementations, the approaches preserve chroma information from pictures in a higher-resolution chroma sampling format, while leveraging commercially available codecs adapted for a lower-resolution chroma sampling format such as YUV 4:2:0, which is widely supported in products.

Claims

1. A computer system comprising one or more processing units and memory, wherein the computer system implements a video processing tool configured to perform operations comprising:

receiving a current picture in a first chroma sampling format that has a first chroma sampling rate, wherein the current picture includes one or more regions;

for each of the one or more regions, determining whether the region is stationary or non-stationary relative to a previous picture in display order;

outputting region change metadata that indicates whether the one or more regions, respectively, are stationary or non-stationary relative to the previous picture; and

converting the current picture to a second chroma sampling format that has a second chroma sampling rate lower than the first chroma sampling rate, including;

retaining chroma sample values of the current picture in the first chroma sampling format that are at selected positions among positions of the first chroma sampling format, wherein the selected positions vary according to a refresh pattern that facilitates recovery of stationary content at the first chroma sampling rate; and

discarding chroma sample values of the current picture in the first chroma sampling format that are at other, non-selected positions among the positions of the first chroma sampling format.

2. The system of claim 1, wherein the operations further comprise:

with a video encoder, encoding the current picture converted to the second chroma sampling format, thereby producing encoded data; and

outputting the encoded data as part of a bitstream.

3. The system of claim 1, wherein the operations further comprise, for each of one or more subsequent pictures, repeating the receiving, the determining, the outputting, and the converting for the subsequent picture as the current picture.

4. The system of claim 1, wherein the operations further comprise:

outputting pattern metadata that indicates, for the refresh pattern as applied for the current picture, which of the chroma sample values of the current picture in the first chroma sampling format have been retained for the current picture in the second chroma sampling format.

5. The system of claim 1, wherein the determining whether the region is stationary or non-stationary relative to the previous picture is based at least in part on:

information from an operating system component indicating which of the one or more regions has changed relative to the previous picture;

or

results of comparing chroma sample values in the region with corresponding chroma sample values of a co-located region in the previous picture.

6. The system of claim 1, wherein, according to the refresh pattern:

the selected positions vary on a picture-by-picture basis; and

/or

different positions, among the positions of the first chroma sampling format, are the selected positions for different pictures of a set of pictures that includes the current picture, such that all of the positions of the first chroma sampling format are represented, collectively, with the selected positions for the different pictures of the set of pictures.

7. The system of claim 1, wherein, according to the refresh pattern:

the selected positions are one quarter of the positions of the first chroma sampling format, wherein the selected positions are alternating positions in alternating lines of the positions of the first chroma sampling format, such that the selected positions are;

(a) even positions of even lines, (b) odd positions of even lines, (c) even positions of odd lines, or (d) odd positions of odd lines, of the positions of the first chroma sampling format;

or

the selected positions are one half of the positions of the first chroma sampling format, wherein the selected positions are alternating positions in each line of the positions of the first chroma sampling format, such that the selected positions are even positions or odd positions of each line of the positions of the first chroma sampling format.

8. The system of claim 1, wherein:

the first chroma sampling format is YUV 4;

4;

4 format, and the second chroma sampling format is YUV 4;

2;

2 format, YUV 4;

2;

0 format, or YUV 4;

1;

1 format;

or

the first chroma sampling format is YUV 4;

2;

2 format, and the second chroma sampling format is YUV 4;

2;

0 format or YUV 4;

1;

1 format.

9. In a computer system, a method of processing video in a first chroma sampling format that has a first chroma sampling rate, the method comprising:

receiving a current picture in a second chroma sampling format that has a second chroma sampling rate lower than the first chroma sampling rate; and

converting the current picture to the first chroma sampling format, including, for a given region among multiple regions of the current picture;

determining that the given region of the current picture is stationary;

within the given region of the current picture, at selected positions among positions of the first chroma sampling format, refreshing chroma sample values in the first chroma sampling format using chroma sample values of the current picture in the second chroma sampling format, wherein the selected positions vary according to a refresh pattern that facilitates recovery of stationary content at the first chroma sampling rate, and wherein the refreshing is contingent on the given region of the current picture being stationary; and

within the given region of the current picture, at other, non-selected positions among the positions of the first chroma sampling format, retaining chroma sample values reconstructed from one or more previous pictures, in display order, in the second chroma sampling format, wherein the retaining is contingent on the given region of the current picture being stationary.

10. The method of claim 9, wherein the converting the current picture to the first chroma sampling format further includes, for each of one or more other regions among the multiple regions of the current picture:

determining whether the other region of the current picture is stationary;

if the other region of the current picture is stationary, within the other region of the current picture;

at selected positions among the positions of the first chroma sampling format, refreshing chroma sample values in the first chroma sampling format using chroma sample values of the current picture in the second chroma sampling format, wherein the selected positions vary according to the refresh pattern; and

at other, non-selected positions among the positions of the first chroma sampling format, retaining chroma sample values reconstructed from the one or more previous pictures, in display order, in the second chroma sampling format;

if the other region of the current picture is not stationary, within the other region of the current picture, for all of the positions of the first chroma sampling format, upsampling the chroma sample values of the current picture in the second chroma sampling format.

11. The method of claim 9, further comprising:

receiving, as part of a bitstream, encoded data; and

with a video decoder, decoding the encoded data, thereby reconstructing the current picture in the second chroma sampling format.

12. The method of claim 9, further comprising, for each of one or more subsequent pictures, repeating the receiving the current picture and the converting for the subsequent picture as the current picture.

13. The method of claim 9, further comprising:

receiving pattern metadata that indicates, for the refresh pattern as applied for the current picture, which of the chroma sample values of the current picture in the first chroma sampling format have been retained for the current picture in the second chroma sampling format; and

using the pattern metadata to identify, for the current picture, the selected positions among the positions of the first chroma sampling format.

14. The method of claim 9, further comprising:

storing, in a buffer, a previous picture, in display order, in the first chroma sampling format, the previous picture in the first chroma sampling format including the chroma sample values reconstructed from the one or more previous pictures in the second chroma sampling format.

15. The method of claim 9, wherein, according to the refresh pattern:

the selected positions vary on a picture-by-picture basis; and

/or

different positions, among the positions of the first chroma sampling format, are the selected positions for different pictures of a set of pictures that includes the current picture, such that all of the positions of the first chroma sampling format are represented, collectively, with the selected positions for the different pictures of the set of pictures.

20. The method of claim 9, further comprising:

receiving region change metadata that indicates whether the multiple regions, respectively, are stationary or non-stationary relative to a previous picture, in display order, in the first chroma sampling format; and

using the region change metadata to determine, for each of the multiple regions, whether the region is stationary or non-stationary relative to the previous picture in the first chroma sampling format.

21. The method of claim 9, wherein the determining that the given region is stationary includes:

determining a count of how many reconstructed luma sample values in the given region have changed compared to corresponding reconstructed luma sample values of a co-located region in an immediately previous picture, in display order, among the one or more previous pictures, in display order, in the second chroma sampling format; and

comparing the count to a threshold.

16. One or more computer-readable media storing computer-executable instructions for causing a computer system, when programmed thereby, to perform operations to process video in a first chroma sampling format that has a first chroma sampling rate, the operations comprising:

receiving a current picture in a second chroma sampling format that has a second chroma sampling rate lower than the first chroma sampling rate; and

converting the current picture to the first chroma sampling format, including, for each of one or more regions of the current picture;

determining whether the region of the current picture is stationary;

if the region of the current picture is stationary, within the region of the current picture;

at selected positions among positions of the first chroma sampling format, refreshing chroma sample values in the first chroma sampling format using chroma sample values of the current picture in the second chroma sampling format, wherein the selected positions vary according to a refresh pattern that facilitates recovery of stationary content at the first chroma sampling rate; and

at other, non-selected positions among the positions of the first chroma sampling format, retaining chroma sample values reconstructed from one or more previous pictures, in display order, in the second chroma sampling format; and

if the region of the current picture is not stationary, within the region of the current picture, for all of the positions of the first chroma sampling format, upsampling the chroma sample values of the current picture in the second chroma sampling format.

17. The one or more computer-readable media of claim 16, wherein the operations further comprise:

receiving region change metadata that indicates whether the one or more regions, respectively, are stationary or non-stationary relative to a previous picture in display order in the first chroma sampling format; and

using the region change metadata to determine, for each of the one or more regions, whether the region is stationary or non-stationary relative to the previous picture in the first chroma sampling format.

18. The one or more computer-readable media of claim 16, wherein the determining whether the region is stationary includes:

determining a count of how many reconstructed luma sample values in the region have changed compared to corresponding reconstructed luma sample values of a co-located region in an immediately previous picture, in display order, among the one or more previous pictures, in display order, in the second chroma sampling format; and

comparing the count to a threshold.

19. The one or more computer-readable media of claim 16, wherein the operations further comprise:

storing, in a buffer, a previous picture, in display order, in the first chroma sampling format, the previous picture in the first chroma sampling format including the chroma sample values reconstructed from the one or more previous pictures in the second chroma sampling format.