Frame packing and unpacking higher-resolution chroma sampling formats

• US 10,523,953 B2
• Filed: 04/17/2018
• Issued: 12/31/2019
• Est. Priority Date: 10/01/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhu, Lihua, Sankuratri, Sridhar, Kumar, B. Anil, Wu, Yongjun, Kanumuri, Sandeep, Sadhwani, Shyam, Sullivan, Gary J.
• Primary Examiner(s): Matt, Marnie A
• Application Number:
  US15/955,558
• Publication Number:
  US 20180234686A1
• Time in Patent Office: 623 Days
• Field of Search: 37524029
• CPC Class Codes: H04N 1/646 Transmitting or storing colour television type signals, e.g. PAL, Lab; Their conversion into additive or subtractive colour signals or vice versa therefor H04N1/642, H04N1/644 take precedence
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/88 involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

Video frames of a higher-resolution chroma sampling format such as YUV 4:4:4 are packed into video frames of a lower-resolution chroma sampling format such as YUV 4:2:0 for purposes of video encoding. For example, sample values for a frame in YUV 4:4:4 format are packed into two frames in YUV 4:2:0 format. After decoding, the video frames of the lower-resolution chroma sampling format can be unpacked to reconstruct the video frames of the higher-resolution chroma sampling format. In this way, available encoders and decoders operating at the lower-resolution chroma sampling format can be used, while still retaining higher resolution chroma information. In example implementations, frames in YUV 4:4:4 format are packed into frames in YUV 4:2:0 format such that geometric correspondence is maintained between Y, U and V components for the frames in YUV 4:2:0 format.

Claims

1. A computing device comprising:

one or more processing units;

volatile memory; and

non-volatile memory and/or storage, the non-volatile memory and/or storage having stored therein computer-executable instructions for causing the computing device, when programmed thereby, to perform operations comprising;

receiving a video frame of a higher-resolution format, the video frame of the higher-resolution format including sample values of first, second, and third component planes;

assigning the sample values of the first component plane of the video frame of the higher-resolution format to a first component plane of a first video frame of a lower-resolution format, the lower-resolution format having a lower resolution than the higher-resolution format;

assigning at least some of the sample values of the second and third component planes of the video frame of the higher-resolution format to second and third component planes of the first video frame of the lower-resolution format; and

assigning at least some of the sample values of the second and third component planes of the video frame of the higher-resolution format to first, second, and third component planes of a second video frame of the lower-resolution format.

2. The computing device of claim 1, wherein the higher-resolution format is a 4:

4;

4 format and the lower-resolution format is a 4;

2;

0 format.

3. The computing device of claim 1, wherein the sample values are in a color space selected from the group consisting of Y′

UV, YIQ, Y′

IQ, YdbDr, YCbCr, and YCoCg, the first component planes being luma, and the second and third component planes being chroma.

4. The computing device of claim 1, wherein the sample values are in a color space selected from the group consisting of RGB and GBR.

5. The computing device of claim 1, wherein the operations further comprise encoding the first and second video frames of the lower-resolution format.

6. The computing device of claim 5, wherein the second assigning and the third assigning maintain geometric correspondence between adjacent sample values of the second and third component planes of the video frame of the higher-resolution format as adjacent sample values and/or collocated portions of the first and second video frames of the lower-resolution format, and wherein the encoding exploits the geometric correspondence.

7. The computing device of claim 6, wherein the encoding exploits the geometric correspondence (a) in encoding operations that include one or more of derivation of motion vectors and derivation of prediction modes, and/or (b) to guide encoding decisions that include one or more of motion estimation, selection of quantization parameters and selection of prediction modes.

8. The computing device of claim 1, wherein the first, second, and third assigning are consistent with plural design constraints, the plural design constraints including:

maintaining geometric correspondence among the sample values of the second and third component planes of the video frame of the higher-resolution format after the second assigning and the third assigning; and

embedding a lower-resolution version of the video frame of the higher-resolution format as the first video frame of the lower-resolution format.

9. The computing device of claim 1, wherein the operations further comprise repeating the receiving, the first assigning, the second assigning, and the third assigning on a frame-by-frame basis for a sequence of video frames of the higher-resolution format.

10. A method of reconstructing a video frame of a higher-resolution format, the video frame of the higher-resolution format including sample values of first, second, and third component planes, the method comprising:

receiving first and second video frames of a lower-resolution format, the lower-resolution format having a lower resolution than the higher-resolution format;

assigning sample values of a first component plane of the first video frame of the lower-resolution format to the first component plane of the video frame of the higher-resolution format;

assigning at least some sample values of second and third component planes of the first video frame of the lower-resolution format to the second and third component planes of the video frame of the higher-resolution format; and

assigning at least some of the sample values of first, second, and third component planes of a second video frame of the lower-resolution format to the second and third component planes of the video frame of the higher-resolution format.

11. The method of claim 10, wherein the higher-resolution format is a 4:

4;

4 format and the lower-resolution format is a 4;

2;

0 format.

12. The method of claim 10, wherein the sample values are in a color space selected from the group consisting of Y′

UV, YIQ, Y′

IQ, YdbDr, YCbCr, and YCoCg, the first component planes being luma, and the second and third component planes being chroma.

13. The method of claim 10, wherein the sample values are in a color space selected from the group consisting of RGB and GBR.

14. The method of claim 10, further comprising decoding the first and second video frames of the lower-resolution format.

15. The method of claim 10, further comprising repeating the receiving, the first assigning, the second assigning, and the third assigning on a frame-by-frame basis for a sequence of video frames of the higher-resolution format.