360 neighbor-based quality selector, range adjuster, viewport manager, and motion estimator for graphics

• US 10,506,196 B2
• Filed: 04/01/2017
• Issued: 12/10/2019
• Est. Priority Date: 04/01/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Ray, Joydeep, Appu, Abhishek R., Baran, Stanley J., Lee, Sang-Hee, Mohammed, Atthar H., Oh, Jong Dae, Chan, Hiu-Fai R., Zhang, Ximin
• Primary Examiner(s): Harold, Jefferey F
• Assistant Examiner(s): Desir, Jean W
• Application Number:
  US15/477,014
• Publication Number:
  US 20180288356A1
• Time in Patent Office: 983 Days
• Field of Search: 3484201, 348 36, 348 51, 382232, 382236, 37524024, 37524001
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/08 Projecting images onto non-planar surfaces, e.g. geodetic screens
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/52 Automatic gain control

Abstract

An embodiment may include a display processor, memory to store a 2D frame corresponding to a projection from a 360 video, and a quality selector to select a quality factor for a block of the 2D frame based on quality information from neighboring blocks of the 2D frame, including blocks which are neighboring only in the 360 video space. The system may also include a range adjuster to adjust a search range for the 2D frame based on a search area of the 2D frame, a viewport manager to determine if a request for a viewport of the 2D frame extends beyond a first edge of the 2D frame and to fill the requested viewport with wrap-around image information, and/or a motion estimator to estimate motion information based on both color information and depth information. Other embodiments are disclosed and claimed.

Claims

1. An electronic processing system, comprising:

a display processor to generate image data for a display;

a memory communicatively coupled to the display processor to store a two-dimensional (2D) frame which corresponds to a projection from a 360 video space;

a quality selector communicatively coupled to the display processor to;

determine a plurality of quantization parameters for a plurality of blocks of the 2D frame based on a budget to support a target bit-rate, wherein at least two of the quantization parameters are different from each other,

determine a first quantization parameter for a first block of the 2D frame based on the budget,

determine that the first block is a neighbor of each of the plurality of blocks in one or more of the 360 video space or the 2D frame, and

in response to the first block being determined to be the neighbor of each of the plurality of blocks in the one or more of the 360 video space or the 2D frame, determine an average of the plurality of quantization parameters and adjust the first quantization parameter based on the average, wherein at least one of the plurality of blocks is a neighbor of the first block only in the 360 video space;

a range adjuster communicatively coupled to the display processor to adjust a search range for the 2D frame based on a search area of the 2D frame; and

a viewport manager communicatively coupled to the display processor to determine if a request for a viewport of the 2D frame extends beyond a first edge of the 2D frame and to fill the requested viewport with wrap-around image information.

2. The system of claim 1, further comprising:

a motion estimator communicatively coupled to the display processor to;

determine a motion vector, based on both color information and depth information, to be used to extrapolate a frame for frame rate up conversion.

3. The system of claim 2, wherein the motion estimator is further to determine a first motion estimate based on the depth information, and to determine a second motion estimate based on the first motion estimate and the color information.

4. The system of claim 1, wherein the 2D frame corresponds to an equirectangular projection of the 360 video space, and wherein the range adjuster is further to adjust a first search range for a pole area of the 2D frame to be relatively larger than a second search range for a central area of the 2D frame.

5. The system of claim 1, wherein the viewport manager is further to fill the requested viewport with image information starting from a second edge of the frame which is contiguous with the first edge in 360 video space.

6. A graphics apparatus, comprising:

an encoder to encode a plurality of blocks of a two-dimensional (2D) frame based on a plurality of quantization parameters, where the 2D frame corresponds to a projection of a 360 video space, wherein at least two of the quantization parameters are different from each other; and

a quality selector communicatively coupled to the encoder to;

determine the plurality of quantization parameters for the plurality of blocks based on a budget to support a target bit-rate,

determine a first quantization parameter for a first block of the 2D frame based on the budget,

determine that the first block is a neighbor of each of the plurality of blocks in one or more of the 360 video space or the 2D frame, wherein at least one of the plurality of blocks is a neighbor of the first block only in the 360 video space, and

in response to the first block being determined to be the neighbor of each of the plurality of blocks in the one or more of the 360 video space or the 2D frame, determine an average of the plurality of quantization parameters and adjust the first quantization parameter based on the average.

7. The apparatus of claim 6, wherein in response to the first block being determined to be the neighbor of each of the plurality of blocks in the one or more of the 360 video space or the 2D frame, the quality selector is further to:

determine a difference between the first quantization parameter and the average,

determine that the difference is not within a difference threshold, and

in response to the difference not being within the difference threshold, adjust the first quantization parameter to keep the difference within the difference threshold while leaving the plurality of quantization parameters unmodified.

8. The apparatus of claim 6, wherein the encoder is further to encode the first block of the 2D frame based on the first quantization parameter.

9. The apparatus of claim 6, further comprising:

a range adjuster communicatively coupled to the encoder to adjust a search range for the 2D frame based on an amount of geometric distortion of a search area of the 2D frame to increase the search range as the amount of geometric distortion increases, and

wherein the encoder is to perform motion estimation for the 2D frame based on the search range.

10. The apparatus of claim 6, further comprising:

a viewport manager communicatively coupled to the encoder to determine if a request for a viewport of the 2D frame extends beyond a first edge of the 2D frame and to fill the requested viewport with wrap-around image information.

11. The apparatus of claim 6, further comprising:

a motion estimator communicatively coupled to the quality selector to;

determine a first motion estimate based on depth information;

determine a second motion estimate based on the first motion estimate and color information; and

determine a motion vector, based on both the first and second motion estimates, to be used to extrapolate a frame for frame rate up conversion.

12. The apparatus of claim 6, further comprising a range adjuster communicatively coupled to the encoder to:

adjust a search range for the 2D frame based on a search area of the 2D frame; and

adjust a first search range for a first search area of the 2D frame to be relatively larger than a second search range for a second search area of the 2D frame based on the first search area having relatively more geometric distortion as compared to the second search area.

13. The apparatus of claim 6, wherein:

the first quantization parameter is greater than the average; and

in response to the first block being determined to be the neighbor of each of the blocks in the one or more of the 360 video space or the 2D frame, reduce the first quantization parameter to approach the average.

14. A method of processing a 360 video, comprising:

encoding a plurality of blocks of a two-dimensional (2D) frame based on a plurality of quantization parameters, where the 2D frame corresponds to a projection of a 360 video space, wherein at least two of the quantization parameters are different from each other;

determining the plurality of quantization parameters for the plurality of blocks based on a budget to support a target bit-rate;

determining a first quantization parameter for a first block of the 2D frame based on the budget;

determining that the first block is a neighbor of each of the plurality of blocks in one or more of the 360 video space or the 2D frame, wherein at least one of the plurality of blocks is a neighbor of the first block only in the 360 video space; and

in response to the first block being determined to be the neighbor of each of the plurality of blocks in the one or more of the 360 video space or the 2D frame, determining an average of the plurality of quantization parameters and adjusting the first quantization parameter based on the average.

15. The method of claim 14, further comprising in response to the first block being determined to be the neighbor of each of the plurality of blocks in the one or more of the 360 video space or the 2D frame:

determining a difference between the first quantization parameter and the average;

determining that the difference is not within a difference threshold; and

in response to the difference not being within the difference threshold, adjusting the first quantization parameter to keep the difference within the difference threshold while leaving the plurality of quantization parameters unmodified.

16. The method of claim 14, further comprising:

encoding the first block of the 2D frame based on the first quantization parameter.

17. At least one non-transitory computer readable medium, comprising a set of instructions, which when executed by a computing device cause the computing device to:

encode a plurality of blocks of a two-dimensional (2D) frame based on a plurality of quantization parameters, where the 2D frame corresponds to a projection of a 360 video space, wherein at least two of the quantization parameters are different from each other;

determine the plurality of quantization parameters for the plurality of blocks based on a budget to support a target bit-rate;

determine a first quantization parameter for a first block of the 2D frame based on the budget;

determine that the first block is a neighbor of each of the plurality of blocks in one or more of the 360 video space or the 2D frame, wherein at least one of the plurality of blocks is a neighbor of the first block only in the 360 video space; and

in response to the first block being determined to be the neighbor of each of the plurality of blocks in the one or more of the 360 video space or the 2D frame, determine an average of the plurality of quantization parameters and adjust the first quantization parameter based on the average.

18. The at least one non-transitory computer readable medium of claim 17, comprising a further set of instructions, which when executed by a computing device cause the computing device to, in response to the first block being determined to be the neighbor of each of the plurality of blocks in the one or more of the 360 video space or the 2D frame:

determine a difference between the first quantization parameter and the average;

determine that the difference is not within a difference threshold; and

in response to the difference not being within the difference threshold, adjust the first quantization parameter to keep the difference within the difference threshold while leaving the plurality of quantization parameters unmodified.

19. The at least one non-transitory computer readable medium of claim 17, comprising a further set of instructions, which when executed by a computing device cause the computing device to:

before the first quantization parameter is adjusted, encode the plurality of blocks based on the plurality of quantization parameters to generate part of an output bitstream; and

after the first quantization parameter is adjusted, encode the first block of the 2D frame based on the first quantization parameter to generate another part of the output bitstream.