MV/mode prediction, ROI-based transmit, metadata capture, and format detection for 360 video

• US 10,506,255 B2
• Filed: 04/01/2017
• Issued: 12/10/2019
• Est. Priority Date: 04/01/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Boyce, Jill M., Lee, Sang-Hee, Appu, Abhishek R., Kao, Wen-Fu, Ray, Joydeep, Peng, Ya-Ti, Rowe, Keith W., Fu, Fangwen, Yedidi, Satya N., Mohan, Sumit, Holland, James M.
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US15/476,989
• Publication Number:
  US 20180288435A1
• Time in Patent Office: 983 Days
• Field of Search: 348 46
• CPC Class Codes: G06T 11/003 Reconstruction from projections, e.g. tomography
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

An embodiment of an electronic processing system may include a 2D frame which corresponds to a projection of a 360 video space, and a component predictor to predict an encode component for a first block of a 2D frame based on encode information from a neighboring block which is neighboring to the first block of the 2D frame only in the 360 video space, a prioritizer to prioritize transmission for a second block of the 2D frame based on an identified region of interest, and/or a format detector to detect a 360 video format of the 2D frame based on image content. A 360 video capture device may include a contextual tagger to tag 360 video content with contextual information which is contemporaneous with the captured 360 video content. Other embodiments are disclosed and claimed.

Claims

1. An electronic processing system, comprising:

a display processor to generate image data for a display;

a memory communicatively coupled to the display processor to store a two-dimensional (2D) frame which corresponds to a projection from a 360 video space;

a component predictor communicatively coupled to the display processor to predict an encode component for a first block of the 2D frame based on encode information from one or more neighboring blocks of the 2D frame, wherein the one or more neighboring blocks of the 2D frame are contiguous to the first block of the 2D frame only when displayed in the 360 video space;

a prioritizer communicatively coupled to the display processor to prioritize transmission for a packet of the 2D frame based on an identified region of interest; and

a format detector communicatively coupled to the display processor to detect a 360 video format of the 2D frame based on an image content of the 2D frame,

wherein the first block of the 2D frame and the one or more neighboring blocks of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

2. The system of claim 1, further comprising:

a 360 video capture device communicatively coupled to the display processor to capture 360 video content; and

a contextual tagger communicatively coupled to the 360 video capture device to tag the 360 video content with contextual information which is contemporaneous with the captured 360 video content.

3. The system of claim 2, wherein the contextual information includes one or more of motion information, location information, velocity information, acceleration information, orientation information, and direction information.

4. The system of claim 1, wherein the encode information includes one or more of motion vector information and mode information.

5. The system of claim 1, wherein the prioritizer is further to prioritize encode for a second block of the 2D frame based on the identified region of interest.

6. The system of claim 1, wherein the detected 360 video format includes one or more of a fish-eye format, an equirectangular projection format, a cube-map format, and a packed cube-map format.

7. A graphics apparatus, comprising:

an encoder to encode a first block of a two-dimensional (2D) frame, where the 2D frame corresponds to a projection of a 360 video space; and

a component predictor communicatively coupled to the encoder to determine if the first block is a neighbor of a second block of the 2D frame in the 360 video space, and to predict an encode component for the second block based on encode information for the first block if the first block is determined to be contiguous to the second block, the first block being contiguous to the second block when displayed in the 360 video space,

wherein the first block of the 2D frame and the second block of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

8. The apparatus of claim 7, wherein the component predictor is further to predict a motion vector for the second block based on encoded motion vector information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

9. The apparatus of claim 7, wherein the component predictor is further to predict a mode for the second block based on encoded mode information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

10. The apparatus of claim 7, wherein the encoder is further to encode the second block of the 2D frame based on the predicted encode component.

11. The apparatus of claim 7, further comprising:

a prioritizer communicatively coupled to the encoder to prioritize transmission for a packet of the 2D frame based on an identified region of interest.

12. The apparatus of claim 7, further comprising:

a format detector communicatively coupled to the encoder to detect a 360 video format of the 2D frame based on an image content of the 2D frame.

13. The apparatus of claim 7, further comprising:

a 360 video capture device communicatively coupled to the encoder to capture 360 video content; and

a contextual tagger communicatively coupled to the 360 video capture device to tag the 360 video content with contextual information which is contemporaneous with the captured 360 video content.

14. A method of processing a 360 video, comprising:

encoding a first block of a two-dimensional (2D) frame, where the 2D frame corresponds to a projection of a 360 video space;

determining if the first block is a neighbor of a second block of the 2D frame in the 360 video space; and

predicting an encode component for the second block based on encode information for the first block if the first block is determined to be contiguous to the second block, the first block being contiguous to the second block when displayed in the 360 video space,

wherein the first block of the 2D frame and the second block of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

15. The method of claim 14, further comprising:

predicting a motion vector for the second block based on encoded motion vector information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

16. The method of claim 14, further comprising:

predicting a mode for the second block based on encoded mode information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

17. The method of claim 14, further comprising:

encoding the second block of the 2D frame based on the predicted encode component.

18. At least one computer readable medium, comprising a set of instructions, which when executed by a computing device cause the computing device to:

encode a first block of a two-dimensional (2D) frame, where the 2D frame corresponds to a projection of a 360 video space;

determine if the first block is a neighbor of a second block of the 2D frame in the 360 video space; and

predict an encode component for the second block based on encode information for the first block if the first block is determined to be contiguous to the second block, the first block being contiguous to the second block when displayed in the 360 video space,

wherein the first block of the 2D frame and the second block of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

19. The at least one computer readable medium of claim 18, comprising a further set of instructions, which when executed by a computing device cause the computing device to:

predict a motion vector for the second block based on encoded motion vector information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

20. The at least one computer readable medium of claim 18, comprising a further set of instructions, which when executed by a computing device cause the computing device to:

predict a mode for the second block based on encoded mode information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

21. The at least one computer readable medium of claim 18, comprising a further set of instructions, which when executed by a computing device cause the computing device to:

encode the second block of the 2D frame based on the predicted encode component.