Conversion and Pre-Processing of Spherical Video for Streaming and Rendering

• US 20170270634A1
• Filed: 03/21/2017
• Published: 09/21/2017
• Est. Priority Date: 03/21/2016
• Status: Active Grant

• Current Assignee: Hulu LLC (The Walt Disney Company)
• Original Assignee: Hulu LLC (The Walt Disney Company)
• Inventors: Eggebrecht, Julian, Engel, Thomas, Petersam, Jens
• Granted Patent: US 10,672,102 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/36 Level of detail
• CPC Class Codes: G06T 2219/2008 Assembling, disassembling
• CPC Class Codes: G06T 3/073 Transforming surfaces of revolution to planar images, e.g. cylindrical surfaces to planar images
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 7/0117 involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407

Abstract

In one embodiment, a method receives spherical content for video and generates face images from the spherical content to represent an image in the video. A two dimensional sheet for the face images is generated. A size of the face images is reduced and a pixel frame around each of the plurality of face images is added on the sheet. Also, a plurality of gaps are added on the sheet in between edges of the face images that are neighboring. The method then adds gap content in the plurality of gaps where the gap content is based on content in an area proximate to the plurality of gaps. The method encodes the face images, the pixel frame, and gap content on the sheet and sends the encoded sheet to a decoder. The face images are decoded for placement on an object structure to display the spherical content.

Claims

1. A method comprising:

receiving, by a computing device, spherical content for video;

generating, by the computing device, a plurality of face images from the spherical content to represent an image in the video;

generating, by the computing device, a two dimensional sheet for the plurality of face images, wherein one or more face images are positioned next to each other based on the one or more face images associated position on an object structure;

reducing, by the computing device, a size of the plurality of face images;

adding, by the computing device, a pixel frame around each of the plurality of face images on the sheet, the pixel frame including pixels from a neighboring face image on the object structure;

adding, by the computing device, a plurality of gaps on the sheet next to edges of the plurality of face images;

adding, by the computing device, gap content in the plurality of gaps, wherein the gap content is based on content in an area proximate to the plurality of gaps;

encoding, by the computing device, the plurality of face images, the pixel frame, and the gap content on the sheet; and

sending, by the computing device, the encoded sheet to a decoder, wherein the plurality of face images are decoded using the pixel frame and the gap content for placement on the object structure to display the spherical content.

2. The method of claim 1, wherein:

the plurality of face images comprise six face images, and

the object structure includes a cube structure with six sides.

3. The method of claim 1, wherein the object structure displays the spherical content around a point associated with a user viewing the video.

4. The method of claim 1, wherein the plurality of face images are positioned on the sheet by performing:

selecting a set of face images designated with having priority; and

positioning the set of face images with natural boundaries next to each other on the sheet.

5. The method of claim 4, wherein the set of face images comprises a front face image, a left face image, a right face image, and a top face image.

6. The method of claim 1, wherein:

spherical content is in an equirectangular format,

the sheet include the plurality of face images in a two dimensional format, wherein the sheet in the two-dimensional format is encoded, and

the object structure displays the plurality of the face images in a three dimensional format.

7. The method of claim 1, wherein gaps in the plurality of gaps are only added in between neighboring edges of face images on the sheet.

8. The method of claim 1, wherein adding the gap content in the plurality of gaps comprises:

adding a first color to pixels in the gap based on a second color of pixels proximate to the pixels in the gap next to a face image.

9. The method of claim 8, wherein the pixels proximate to the pixels in the gap have been added to the face image from a neighboring face image in the pixel frame.

10. The method of claim 1, wherein the gap content is used to perform filtering on the first face image to render the first face image on the object structure.

11. The method of claim 1, wherein the gap content is not rendered on the object structure.

12. The method of claim 1, wherein:

the pixel frame surrounds each face image in the plurality of face images, and

each face image includes a gap only in between edges of the plurality of face images that are neighboring on the sheet.

13. The method of claim 1, wherein:

decoding of the sheet generates a set of miniaturized face images for use in filtering of the face images, and

a size of the pixel frame is based on a number of the set of miniaturized face images needed such that at least a single pixel frame is included in a smallest of the set of miniaturized face images.

14. The method of claim 1, wherein:

the two dimension sheet includes a first set of face images for a first eye and a second set of face images for a second eye,

the object structure comprises a first object structure and a second object structure, and

the first set of face images are placed on a first object structure for the first eye and the second set of images are placed on a second object structure for the second eye.

15. The method of claim 14, wherein encoding uses content across the first set of face images and the second set of face images to optimize encoding.

16. The method of claim 1, wherein at least one of the face images uses a different number of pixels from another face image.

17. A non-transitory computer-readable storage medium containing instructions, that when executed, control a computer system to be configured for:

receiving spherical content for video;

generating a plurality of face images from the spherical content to represent an image in the video;

generating a two dimensional sheet for the plurality of face images, wherein one or more face images are positioned next to each other based on the one or more face images associated position on an object structure;

reducing a size of the plurality of face images;

adding a pixel frame around each of the plurality of face images on the sheet, the pixel frame including pixels from a neighboring face image on the object structure;

adding a plurality of gaps on the sheet in between edges of the plurality of face images that are neighboring;

adding gap content in the plurality of gaps, wherein the gap content is based on content in an area proximate to the plurality of gaps;

encoding the plurality of face images, the pixel frame, and the gap content on the sheet; and

sending the encoded sheet to a decoder, wherein the plurality of face images are decoded using the pixel frame and the gap content for placement on the object structure to display the spherical content.

18. A method comprising:

receiving, by a computing device, an encoded sheet for spherical content for a video, wherein the encoded sheet includes a plurality of face images from the spherical content to represent an image in the video, gap content in a plurality of gaps on the sheet in between edges of the plurality of face images that are neighboring, and a pixel frame around each of the plurality of face images on the sheet, the pixel frame including pixels from a neighboring face image on an object structure;

performing, by the computing device, a decoding process for the encoded sheet, the decoding process performing;

generating, by the computing device, a set of miniaturized face images, wherein each miniaturized face image in the set of miniaturized face images includes at least a single pixel frame from the pixel frame around each of the plurality of face images on the sheet;

performing, by the computing device, filtering of the plurality of face images on the sheet, wherein filtering uses the pixel frame in the set of miniaturized faces and the sheet;

rendering, by the computing device, the plurality of face images on the object structure to display the spherical content, wherein artifacts that exist in the gap due to the decoding process are not rendered on the object structure.

19. The method of claim 18, wherein a size of the pixel frame is based on a number of the set of miniaturized face images needed such that at least a single pixel frame is included in a smallest of the set of miniaturized face images.

20. The method of claim 18, wherein the gap content minimizes artifacts in the decoding process.