Excluding masked regions of virtual reality (VR) frames from encoder processing

• US 10,580,167 B1
• Filed: 06/12/2017
• Issued: 03/03/2020
• Est. Priority Date: 01/24/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Waggoner, Charles Benjamin Franklin, Wu, Yongjun
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US15/620,690
• Time in Patent Office: 995 Days
• Field of Search: 345427, 345421
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06T 9/20 Contour coding, e.g. using detection of edges
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Techniques are described that enable a two-dimensional (2D) representation of three-dimensional (3D) virtual reality (VR) content to be encoded. These techniques include encoding VR content while excluding non-display pixels of the VR content from motion estimation during encoder processing.

Claims

1. A method, comprising:

obtaining a two dimensional (2D) rectangular representation of virtual reality (VR) content, the VR content including one or more images corresponding to a plurality of faces of a virtual three dimensional (3D) polygonal projection space, the virtual 3D polygonal projection space corresponding to one of a hexahedron, a cube, an octahedron, a dodecahedron, or an icosahedron, the 2D rectangular representation including a first set of regions containing a plurality of non-display pixels and a second set of regions containing a plurality of display pixels, the second set of regions including a plurality of regions, each region of the second set of regions corresponding to a different one of the plurality of faces of the 3D projection space, the display pixels containing image content of the VR content and the non-display pixels not containing image content of the VR content, each of the display pixels and non-display pixels being represented by a corresponding set of pixel values including an intensity value;

modifying the 2D rectangular representation, wherein modifying the 2D rectangular representation includes modifying the intensity value of each pixel of at least a portion of the non-display pixels of a first region of the first set of regions within the 2D rectangular representation based, at least in part, on the intensity value of at least one pixel of at least a portion of the display pixels of a second region of the second set of regions within the 2D rectangular representation, the first region being adjacent to the second region within the 2D representation;

after modifying the 2D rectangular representation, obtaining a mask indicating positions of the non-display pixels and the display pixels within the 2D representation, the mask including a first value for each of the non-display pixels and a second value for each of the display pixels; and

encoding the 2D representation using the mask such that the non-display pixels of the first set of regions are excluded from motion estimation.

2. The method as recited in claim 1, wherein encoding the 2D representation comprises:

encoding DC coefficients for the non-display pixels of the first set of regions without encoding AC coefficients for the non-display pixels of the first set of regions.

3. The method as recited in claim 1, wherein encoding the 2D representation comprises:

encoding the non-display pixels of the first set of regions using a skip encoding mode.

4. The method as recited in claim 1, wherein encoding the 2D representation is performed according to a variable block size to minimize a total number of non-display pixels in blocks that contain display pixels.

5. The method as recited in claim 1, wherein encoding the 2D representation comprises:

encoding only the display pixels using a warp transform.

21. The method as recited in claim 1, further comprising:

generating the 2D rectangular representation of the VR content by mapping the images of VR content corresponding to the plurality of faces of the virtual three dimensional (3D) polygonal projection space to the 2D projection space.

22. The method as recited in claim 1, the first value being a first one of two different binary values and the second value being a second one of the two different binary values.

23. The method as recited in claim 1, the first value being zero and the second value being one.

6. A computer program product, comprising one or more non-transitory computer readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more processors, the computer program instructions cause the one or more processors to:

obtain a two dimensional (2D) representation of virtual reality (VR) content, the VR content including one or more images corresponding to a plurality of faces of a virtual three dimensional (3D) projection space, the 2D representation including a first set of regions containing a plurality of non-display pixels and a second set of regions containing a plurality of display pixels, the second set of regions including a plurality of regions, each region of the second set of regions corresponding to a different one of the plurality of faces of the 3D projection space, the display pixels containing image content of the VR content and the non-display pixels not containing image content of the VR content;

obtain a mask indicating positions of the non-display pixels and the display pixels within the 2D representation, the mask including a first value for each of the non-display pixels and a second value for each of the display pixels; and

encode the 2D representation using the mask such that the non-display pixels of the first set of regions are excluded from motion estimation.

7. The computer program product as recited in claim 6, wherein the computer program instructions are further configured to cause the one or more processors to:

encode the 2D representation using the mask by encoding DC coefficients for the non-display pixels of the first set of regions without encoding AC coefficients for the non-display pixels of the first set of regions.

8. The computer program product as recited in claim 6, wherein the computer program instructions are further configured to cause the one or more processors to:

operate an encoder to encode the 2D representation using the mask by encoding the display pixels using Asymmetric Motion Partitioning (AMP).

9. The computer program product as recited in claim 6, wherein the computer program instructions are further configured to cause the one or more processors to:

encode the 2D representation using the mask by encoding at least a subset of the display pixels according to a variable block size to minimize a total number of non-display pixels in blocks that contain display pixels.

10. The computer program product as recited in claim 6, wherein the computer program instructions are further configured to cause the one or more processors to:

instruct an encoder to encode a subset of the display pixels according to a specified block size.

11. The computer program product as recited in claim 6, wherein encoding the 2D representation comprises:

encoding the non-display pixels of the first set of regions using a skip encoding mode.

12. The computer program product as recited in claim 6, wherein the computer program instructions are further configured to cause the one or more processors to:

identify an edge within the 2D representation, the edge separating a first region of the first set of regions from a second region of the second set of regions;

identify at least a portion of the second region;

determine an intensity value for each of the display pixels of the portion of the second region;

for each of the non-display pixels of a portion of the first region, determine a corresponding replacement intensity value based, at least in part, on the intensity value of at least one of the display pixels of the portion of the second region;

modify each of the non-display pixels of the portion of the first region to include the corresponding replacement intensity value; and

generate or update the mask according to a result of modifying each of the non-display pixels of the portion of the first region to include the corresponding replacement intensity value.

13. The computer program product as recited in claim 12, wherein the replacement intensity value for each of the non-display pixels of the portion of the first region comprises:

the intensity value of a corresponding display pixel of the portion of the second region;

or

an average intensity value of the display pixels of the portion of the second region.

14. A system, comprising:

an encoder configured to encode video frames according to a particular encoding standard; and

one or more processors and memory configured to;

obtain a two dimensional (2D) representation of virtual reality (VR) content, the VR content including one or more images corresponding to a plurality of faces of a virtual three dimensional (3D) projection space, the 2D representation including a first set of regions containing a plurality of non-display pixels and a second set of regions containing a plurality of display pixels, the second set of regions including a plurality of regions, each region of the second set of regions corresponding to a different one of the plurality of faces of the 3D projection space, the display pixels containing image content of the VR content and the non-display pixels not containing image content of the VR content;

obtain a mask indicating positions of the non-display pixels and the display pixels within the 2D representation, the mask including a first value for each of the non-display pixels and a second value for each of the display pixels; and

encode the 2D representation using the mask such that the non-display pixels of the first set of regions are excluded from motion estimation.

15. The system as recited in claim 14, the one or more processors and memory being further configured to:

encode the 2D representation by encoding DC coefficients for the non-display pixels of the first set of regions without encoding AC coefficients for the non-display pixels of the first set of regions.

16. The system as recited in claim 14, the one or more processors and memory being further configured to:

encode the 2D representation such that a skip encoding mode is used to encode the non-display pixels of the first set of regions.

17. The system as recited in claim 14, the one or more processors and memory being further configured to:

operate an encoder to encode the 2D representation by encoding the display pixels using Asymmetric Motion Partitioning (AMP).

18. The system as recited in claim 14, the one or more processors and memory being further configured to:

encode the 2D representation by encoding at least a subset of the display pixels according to a variable block size to minimize a total number of non-display pixels in blocks that contain display pixels.

19. The system as recited in claim 14, the one or more processors and memory being further configured to:

identify an edge within the 2D representation, the edge separating a first region of the first set of regions from a second region of the second set of regions;

identify at least a portion of the second region;

determine an intensity value for each of the display pixels of the portion of the second region;

for each of the non-display pixels of a portion of the first region, determine a corresponding replacement intensity value based, at least in part, on the intensity value of at least one of the display pixels of the portion of the second region;

modify each of the non-display pixels of the portion of the first region to include the corresponding replacement intensity value; and

generate or update the mask according to a result of modifying each of the non-display pixels of the portion of the first region to include the corresponding replacement intensity value.

20. The system as recited in claim 19, wherein the replacement intensity value for each of the non-display pixels of the portion of the first region comprises:

the intensity value of a corresponding display pixel of the portion of the second region;

or

an average intensity value of the display pixels of the portion of the second region.