Prediction mode information upsampling for scalable video coding

• US 10,721,478 B2
• Filed: 05/03/2017
• Issued: 07/21/2020
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Seregin, Vadim, Chen, Jianle, Li, Xiang, Rapaka, Krishnakanth, Karczewicz, Marta, Chen, Ying
• Primary Examiner(s): Huang, Frank F
• Application Number:
  US15/585,918
• Publication Number:
  US 20170237990A1
• Time in Patent Office: 1,175 Days
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/52 by predictive encoding

Abstract

In one embodiment, an apparatus configured to code video data includes a processor and a memory unit. The memory unit stores video data associated with a first layer having a first spatial resolution and a second layer having a second spatial resolution. The video data associated with the first layer includes at least a first layer block and first layer prediction mode information associated with the first layer block, and the first layer block includes a plurality of sub-blocks where each sub-block is associated with respective prediction mode data of the first layer prediction mode information. The processor derives the predication mode data associated with one of the plurality of sub-blocks based at least on a selection rule, upsamples the derived prediction mode data and the first layer block, and associates the upsampled prediction mode data with each upsampled sub-block of the upsampled first layer block.

Claims

1. An apparatus configured to code video data, the apparatus comprising:

a memory unit configured to store video data associated with a base layer having a first spatial resolution and an enhancement layer having a second spatial resolution greater than the first spatial resolution, the video data associated with the base layer including at least a base layer block and base layer prediction mode information associated with the base layer block, the base layer block including a plurality of sub-blocks where each sub-block is associated with respective prediction mode data of the base layer prediction mode information, the respective prediction mode data including non-pixel information; and

a processor implemented in circuitry and in communication with the memory unit, the processor being configured to;

upsample pixel data of a picture of the base layer to form an inter-layer picture comprising a plurality of blocks, the inter-layer picture having the second spatial resolution associated with the enhancement layer and being usable for inter-prediction of one or more pictures of the enhancement layer;

for every respective block of the inter-layer picture;

identify a single sub-block of the base layer block corresponding to the respective block of the inter-layer picture, the identified sub-block neighbored on each side by a respective sub-block of the plurality of sub-blocks of the base layer block;

select respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block;

upsample the selected respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block; and

associate the upsampled prediction mode data with the respective block of the inter-layer picture;

add the inter-layer picture to a reference picture list, pictures within the reference picture list being usable for inter-prediction of the enhancement layer having the second spatial second resolution;

inter-predict a first block of a first picture of the one or more pictures of the enhancement layer from the inter-layer picture to form a predicted block for the first block, wherein to predict the first block, the processor is configured to code a reference index of the first block, the reference index identifying the inter-layer picture in the reference picture list; and

predict prediction mode data of a second block of a second picture of the one or more pictures of the enhancement layer using the upsampled prediction mode data of the inter-layer picture of the reference picture list.

2. The apparatus of claim 1, wherein every respective block of the inter-layer picture has a fixed size.

3. The apparatus of claim 1, wherein the processor is further configured to determine a predicted value of an enhancement layer block in the enhancement layer based at least on the inter-layer picture being added to the reference picture list.

4. The apparatus of claim 3, wherein the base layer block is located at a position in the base layer corresponding to a position of the base layer block in the base layer.

5. The apparatus of claim 3, wherein the base layer block is located at a position in the base layer corresponding to the position of a neighboring enhancement layer block positioned adjacent to the enhancement layer block in the second layer.

6. The apparatus of claim 1, wherein a spatial resolution ratio between the upsampled prediction mode data and the selected respective prediction mode data matches a spatial resolution ratio between the second spatial resolution and the first spatial resolution.

7. The apparatus of claim 1, wherein the non-pixel information comprises at least one of an intra prediction mode, an inter prediction mode, motion information, a reference index, or an inter direction that is usable for compressing the video data.

8. The apparatus of claim 1, wherein the processor comprises a decoder or an encoder.

9. The apparatus of claim 1, wherein the apparatus comprises at least one of a computer, a tablet computer, a set-top box, a telephone handset, a smart phone, a smart pad, a television, a camera, a display device, a digital media player, a video gaming console, an in-car computer or a wireless communication device.

10. A method of coding video data, the method comprising:

storing video data associated with a base layer having a first spatial resolution and an enhancement layer having a second spatial resolution greater than the first spatial resolution in a memory unit, the video data associated with the base layer including at least a base layer block and base layer prediction mode information associated with the base layer block, the base layer block including a plurality of sub-blocks where each sub-block is associated with respective prediction mode data of the base layer prediction mode information, the respective prediction mode data including non-pixel information;

upsampling, by a processor in communication with the memory unit, pixel data of a picture of the base layer to form an inter-layer picture comprising a plurality of blocks, the inter-layer picture having the second spatial resolution associated with the enhancement layer and being usable for inter-prediction of one or more pictures of the enhancement layer;

for every respective block of the inter-layer picture;

identifying, by the processor, a single sub-block of the base layer block corresponding to the respective block of the inter-layer picture, the identified sub-block neighbored on each side by a respective sub-block of the plurality of sub-blocks of the base layer block;

selecting, by the processor, respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block;

upsampling, by the processor, the selected respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block; and

associating, by the processor, the upsampled prediction mode data with the respective block of the inter-layer picture;

adding, by the processor, the inter-layer picture to a reference picture list, pictures within the reference picture list being usable for inter-prediction of the enhancement layer having the second spatial second resolution;

inter-predicting a first block of a first picture of the one or more pictures of the enhancement layer from the inter-layer picture to form a predicted block for the first block, wherein inter-predicting the first block comprises coding a reference index of the first block, the reference index identifying the inter-layer picture in the reference picture list; and

predicting prediction mode data of a second block of a second picture of the one or more pictures of the enhancement layer using the upsampled prediction mode data of the inter-layer picture of the reference picture list.

11. The method of claim 10, further comprising determining a predicted value of an enhancement layer block in the enhancement layer based at least on the inter-layer picture being added to the reference picture list, the base layer block being located at a position in the base layer corresponding to the position of the enhancement layer block in the enhancement layer or located at the position in the base layer corresponding to the position of a neighboring enhancement layer block positioned adjacent to the enhancement layer block in the enhancement layer.

12. The method of claim 10, wherein a spatial resolution ratio between the upsampled prediction mode data and the selected respective prediction mode data matches a spatial resolution ratio between the second spatial resolution and the first spatial resolution.

13. The method of claim 10, wherein every respective block of the inter-layer picture has a fixed size.

14. The method of claim 10, wherein the non-pixel information comprises at least one of an intra prediction mode, an inter prediction mode, a motion information, a reference index, or an inter direction that is usable for compressing video data.

22. The method of claim 10, wherein storing the base layer prediction mode information comprises storing the base layer prediction mode information at a granularity of at least 16×

16 samples.

15. An apparatus configured to code video data, the apparatus comprising:

means for storing video data associated with a base layer having a first spatial resolution and an enhancement layer having a second spatial resolution greater than the first spatial resolution, the video data associated with the base layer including at least a base layer block and base layer prediction mode information associated with the base layer block, the base layer block including a plurality of sub-blocks where each sub-block is associated with respective prediction mode data of the base layer prediction mode information, the respective prediction mode data including non-pixel information; and

means for upsampling pixel data of a picture of the base layer to form an inter-layer picture comprising a plurality of blocks, the inter-layer picture having the second spatial resolution associated with the enhancement layer and being usable for inter-prediction of one or more pictures of the enhancement layer;

means for identifying, for every respective block of the inter-layer picture, a single sub-block of the base layer block corresponding to the respective block of the inter-layer picture, the identified sub-block neighbored on each side by a respective sub-block of the plurality of sub-blocks of the base layer block;

means for selecting, for every respective block of the inter-layer picture, respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block;

means for upsampling, for every respective block of the inter-layer picture, the selected respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block;

means for associating, for every respective block of the inter-layer picture, the upsampled prediction mode data with the respective block of the inter-layer picture;

means for adding the inter-layer picture to a reference picture list, pictures within the reference picture list being usable for inter-prediction of the enhancement layer having the second spatial second resolution;

means for inter-predicting a first block of a first picture of the one or more pictures of the enhancement layer from the inter-layer picture to form a predicted block for the first block, wherein the means for inter-predicting the first block comprises means for coding a reference index of the first block, the reference index identifying the inter-layer picture in the reference picture list; and

means for predicting prediction mode data of a second block of a second picture of the one or more pictures of the enhancement layer using the upsampled prediction mode data of the inter-layer picture of the reference picture list.

16. The apparatus of claim 15, wherein every respective block of the inter-layer picture has a fixed size.

17. The apparatus of claim 15, further comprising means for determining a predicted value of an enhancement layer block in the enhancement layer based at least on the inter-layer picture added to the reference picture list, the base layer block being located at a position in the base layer corresponding to the position of the enhancement layer block in the enhancement layer or located at the position in the base layer corresponding to the position of a neighboring enhancement layer block positioned adjacent to the enhancement layer block in the second layer.

18. The apparatus of claim 15, wherein the non-pixel information comprises at least one of an intra prediction mode, an inter prediction mode, a motion information, a reference index, or an inter direction that is usable for compressing the video data.

19. A non-transitory computer storage that stores executable program instructions that, when executed, direct an apparatus for coding video data to perform a process that comprises:

storing video data associated with a base layer having a first spatial resolution and an enhancement layer having a second spatial resolution greater than the first spatial resolution in a memory unit, the video data associated with the base layer including at least a base layer block and base layer prediction mode information associated with the base layer block, the base layer block including a plurality of sub-blocks where each sub-block is associated with respective prediction mode data of the base layer prediction mode information, the respective prediction mode data including non-pixel information;

upsampling, by a processor in communication with the memory unit, pixel data of a picture of the base layer to form an inter-layer picture comprising a plurality of blocks, the inter-layer picture being usable for inter-prediction of one or more pictures of the enhancement layer;

for every respective block of the inter-layer picture;

identifying a single sub-block of the base layer block corresponding to the respective block of the inter-layer picture, the identified sub-block neighbored on each side by a respective sub-block of the plurality of sub-blocks of the base layer block;

selecting respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block;

upsampling the selected respective prediction mode data associated with the identified sub-block of the plurality of sub-blocks of the base layer block; and

associating the upsampled prediction mode data with the respective block of the inter-layer picture;

adding the inter-layer picture to a reference picture list, pictures within the reference picture list being usable for inter-prediction of the enhancement layer having the second spatial second resolution;

inter-predicting a first block of a first picture of the one or more pictures of the enhancement layer from the inter-layer picture to form a predicted block for the first block, wherein inter-predicting the first block comprises coding a reference index of the first block, the reference index identifying the inter-layer picture in the reference picture list; and

predicting prediction mode data of a second block of a second picture of the one or more pictures of the enhancement layer using the upsampled prediction mode data of the inter-layer picture of reference picture list.

20. The non-transitory computer storage of claim 19, wherein the process further comprises determining a predicted value of an enhancement layer block in the enhancement layer based at least on the inter-layer picture added to the reference picture list, the base layer block being located at a position in the base layer corresponding to the position of the enhancement layer block in the enhancement layer or located at the position in the base layer corresponding to the position of a neighboring enhancement layer block positioned adjacent to the enhancement layer block in the enhancement layer.

21. The non-transitory computer storage of claim 19, wherein the non-pixel information includes an intra prediction mode information, an inter prediction mode information, a motion information, a reference index, or an inter direction that is usable for compressing the video data.