Method and apparatus for generating and encoding projection-based frame with 360-degree content represented by triangular projection faces packed in octahedron projection layout

• US 10,380,715 B2
• Filed: 11/28/2017
• Issued: 08/13/2019
• Est. Priority Date: 12/07/2016
• Status: Active Grant

• Current Assignee: MediaTek, Inc.
• Original Assignee: MediaTek, Inc.
• Inventors: Lin, Hung-Chih, Huang, Chao-Chih, Li, Chia-Ying, Ou Yang, Hui, Lin, Jian-Liang, Chang, Shen-Kai
• Primary Examiner(s): Natnael, Paulos M
• Application Number:
  US15/825,081
• Publication Number:
  US 20180158170A1
• Time in Patent Office: 623 Days
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof

Abstract

A video processing method includes: receiving an omnidirectional image/video content corresponding to a viewing sphere, generating a sequence of projection-based frames according to the omnidirectional image/video content and an octahedron projection layout, and encoding, by a video encoder, the sequence of projection-based frames to generate a bitstream. Each projection-based frame has a 360-degree image/video content represented by triangular projection faces packed in the octahedron projection layout. The omnidirectional image/video content of the viewing sphere is mapped onto the triangular projection faces via an octahedron projection of the viewing sphere. An equator of the viewing sphere is not mapped along any side of each of the triangular projection faces.

Claims

1. A video processing method comprising:

receiving an omnidirectional image/video content corresponding to a viewing sphere;

generating a projection-based frame according to the omnidirectional image/video content and an octahedron projection layout, wherein the projection-based frame has a 360-degree image/video content represented by triangular projection faces packed in the octahedron projection layout, the omnidirectional image/video content of the viewing sphere is mapped onto the triangular projection faces via an octahedron projection of the viewing sphere, and an equator of the viewing sphere is not mapped along any side of each of the triangular projection faces; and

encoding, by a video encoder, the projection-based frame to generate a part of a bitstream;

wherein the triangular projection faces packed in the octahedron projection layout comprise a first triangular projection face, a second triangular projection face and a third triangular projection face, one side of the first triangular projection face connects with one side of the second triangular projection face, one side of the third triangular projection face connects with another side of the second triangular projection face, there is an image content continuity boundary between said one side of the first triangular projection face and said one side of the second triangular projection face, and there is an image content continuity boundary between said one side of the third triangular projection face and said another side of the second triangular projection face; and

wherein the triangular projection faces packed in the octahedron projection layout further comprise a fourth triangular projection face, one side of the fourth triangular projection face connects with yet another side of the second triangular projection face, and there is an image content continuity boundary between said one side of the fourth triangular projection face and said yet another side of the second triangular projection face.

2. The video processing method of claim 1, wherein the viewing sphere is composed of a first hemisphere and a second hemisphere;

the first triangular projection face, the second triangular projection face and the third triangular projection face are all derived from the first hemisphere via the octahedron projection of the viewing sphere; and

the fourth triangular projection face is derived from the second hemisphere via the octahedron projection of the viewing sphere.

3. The video processing method of claim 1, wherein a shape of the octahedron projection layout is a rectangle, the triangular projection faces packed in the octahedron projection layout further comprise a fifth triangular projection face, the fifth triangular projection face is split into a first right-triangle-shaped part and a second right-triangle-shaped part, the first right-triangle-shaped part has a first side being one side of the fifth triangular projection face and a second side being a first part of another side of the fifth triangular projection face, the second right-triangle-shaped part has a first side being yet another side of the fifth triangular projection face and a second side being a second part of said another side of the fifth triangular projection face, the first side of the second right-triangle-shaped part connects another side of the first triangular projection face, and the first side of the first right-triangle-shaped part connects another side of the third triangular projection face.

4. The video processing method of claim 3, wherein the triangular projection faces packed in the octahedron projection layout further comprise a sixth triangular projection face, a seventh triangular projection face, and an eighth triangular projection face, one side of the seventh triangular projection face connects with another side of the fourth triangular projection face, one side of the eighth triangular projection face connects with yet another side of the fourth triangular projection face, there is an image content continuity boundary between said one side of the seventh triangular projection face and said another side of the fourth triangular projection face, there is an image content continuity boundary between said one side of the eighth triangular projection face and said yet another side of the fourth triangular projection face, the sixth triangular projection face is split into a third right-triangle-shaped part and a fourth right-triangle-shaped part, the third right-triangle-shaped part has a first side being one side of the sixth triangular projection face and a second side being a first part of another side of the sixth triangular projection face, the fourth right-triangle-shaped part has a first side being yet another side of the sixth triangular projection face and a second side being a second part of said another side of the sixth triangular projection face, the first side of the third right-triangle-shaped part connects with another side of the seventh triangular projection face, and the first side of the fourth right-triangle-shaped part connects with another side of the eighth triangular projection face.

5. The video processing method of claim 4, wherein the equator of the viewing sphere is mapped to the second triangular projection face, the fourth triangular projection face, the fifth triangular projection face, and the sixth triangular projection face.

6. A video processing method comprising:

receiving an omnidirectional image/video content corresponding to a viewing sphere;

generating a projection-based frame according to the omnidirectional image/video content and an octahedron projection layout, wherein the projection-based frame has a 360-degree image/video content represented by triangular projection faces packed in the octahedron projection layout, the omnidirectional image/video content of the viewing sphere is mapped onto the triangular projection faces via an octahedron projection of the viewing sphere, and an equator of the viewing sphere is not mapped along any side of each of the triangular projection faces; and

encoding, by a video encoder, the projection-based frame to generate a part of a bitstream;

wherein the triangular projection faces packed in the octahedron projection layout comprise a first triangular projection face, a second triangular projection face and a third triangular projection face, one side of the first triangular projection face connects with one side of the second triangular projection face, one side of the third triangular projection face connects with another side of the second triangular projection face, there is an image content continuity boundary between said one side of the first triangular projection face and said one side of the second triangular projection face, and there is an image content continuity boundary between said one side of the third triangular projection face and said another side of the second triangular projection face; and

wherein the triangular projection faces packed in the octahedron projection layout further comprise a fourth triangular projection face, a fifth triangular projection face, and a sixth triangular projection face, another side of the third triangular projection face connects with one side of the fourth triangular projection face, another side of the fourth triangular projection face connects with one side of the fifth triangular projection face, another side of the fifth triangular projection face connects with one side of the sixth triangular projection face, there is an image content continuity boundary between said another side of the third triangular projection face and said one side of the fourth triangular projection face, there is an image content continuity boundary between said another side of the fourth triangular projection face and said one side of the fifth triangular projection face, and there is an image content continuity boundary between said another side of the fifth triangular projection face and said one side of the sixth triangular projection face.

7. The video processing method of claim 6, wherein a shape of the octahedron projection layout is a rectangle, the triangular projection faces packed in the octahedron projection layout further comprise a seventh triangular projection face and an eighth triangular projection face, one side of the eighth triangular projection face connects with another side of the sixth triangular projection face, the seventh triangular projection face is split into a first right-triangle-shaped part and a second right-triangle-shaped part, the first right-triangle-shaped part has a first side being one side of the seventh triangular projection face and a second side being a first part of another side of the seventh triangular projection face, the second right-triangle-shaped part has a first side being yet another side of the seventh triangular projection face and a second side being a second part of said another side of the seventh triangular projection face, the first side of the second right-triangle-shaped part connects with another side of the first triangular projection face, and the first side of the first right-triangle-shaped part connects another side of the eighth triangular projection face.

8. The video processing method of claim 6, wherein the equator of the viewing sphere is mapped to the first triangular projection face, the second triangular projection face, the fourth triangular projection face, and the fifth triangular projection face.

9. The video processing method of claim 6, wherein the viewing sphere is composed of a first hemisphere and a second hemisphere;

the first triangular projection face, the fifth triangular projection face and the sixth triangular projection face are all derived from the first hemisphere via the octahedron projection of the viewing sphere; and

the second triangular projection face, the third triangular projection face and the fourth triangular projection face are all derived from the second hemisphere via the octahedron projection of the viewing sphere.

10. The video processing method of claim 6, wherein the equator of the viewing sphere is mapped to the first triangular projection face, the third triangular projection face, the fourth triangular projection face, and the sixth triangular projection face.

11. The video processing method of claim 6, wherein the viewing sphere is composed of a first hemisphere and a second hemisphere;

the first triangular projection face, the second triangular projection face and the third triangular projection face are all derived from the first hemisphere via the octahedron projection of the viewing sphere; and

the fourth triangular projection face, the fifth triangular projection face and the sixth triangular projection face are all derived from the second hemisphere via the octahedron projection of the viewing sphere.

12. The video processing method of claim 6, wherein the triangular projection faces packed in the octahedron projection layout further comprise a seventh triangular projection face and an eighth triangular projection face, another side of the sixth triangular projection face connects with one side of the seventh triangular projection face, another side of the seventh triangular projection face connects with one side of the eighth triangular projection face, there is an image content continuity boundary between said another side of the sixth triangular projection face and said one side of the seventh triangular projection face, and there is an image content continuity boundary between said another side of the seventh triangular projection face and said one side of the eighth triangular projection face.

13. The video processing method of claim 12, wherein a shape of the octahedron projection layout is a rectangle, the second triangular projection face is split into a first right-triangle-shaped part and a second right-triangle-shaped part, the first right-triangle-shaped part has a first side being said one side of the second triangular projection face and a second side being a first part of said another side of the second triangular projection face, the second right-triangle-shaped part has a first side being yet another side of the second triangular projection face and a second side being a second part of said another side of the second triangular projection face, the third triangular projection face is split into a third right-triangle-shaped part and a fourth right-triangle-shaped part, the third right-triangle-shaped part has a first side being said another side of the third triangular projection face and a second side being a first part of said one side of the third triangular projection face, the fourth right-triangle-shaped part has a first side being yet another side of the third triangular projection face and a second side being a second part of said one side of the third triangular projection face, the sixth triangular projection face is split into a fifth right-triangle-shaped part and a sixth right-triangle-shaped part, the fifth right-triangle-shaped part has a first side being said one side of the sixth triangular projection face and a second side being a first part of said another side of the sixth triangular projection face, the sixth right-triangle-shaped part has a first side being yet another side of the sixth triangular projection face and a second side being a second part of said another side of the sixth triangular projection face, the seventh triangular projection face is split into a seventh right-triangle-shaped part and an eighth right-triangle-shaped part, the seventh right-triangle-shaped part has a first side being said another side of the seventh triangular projection face and a second side being a first part of said one side of the seventh triangular projection face, the eighth right-triangle-shaped part has a first side being yet another side of the seventh triangular projection face and a second side being a second part of said one side of the seventh triangular projection face, the first side of the second right-triangle-shaped part connects with another side of the first triangular projection face, the first side of the fourth right-triangle-shaped part connects with yet another side of the fourth triangular projection face, the first side of the sixth right-triangle-shaped part connects with yet another side of the fifth triangular projection face, and the first side of the eighth right-triangle-shaped part connects with another side of the eighth triangular projection face.

14. The video processing method of claim 12, wherein the equator of the viewing sphere is mapped to the second triangular projection face, the fourth triangular projection face, the fifth triangular projection face, and the seventh triangular projection face.

15. The video processing method of claim 12, wherein the viewing sphere is composed of a first hemisphere and a second hemisphere;

the first triangular projection face, the second triangular projection face, the third triangular projection face and the fourth triangular projection face are all derived from the first hemisphere via the octahedron projection of the viewing sphere; and

the fifth triangular projection face, the sixth triangular projection face, the seventh triangular projection face and the eighth triangular projection face are all derived from the second hemisphere via the octahedron projection of the viewing sphere.

16. A video processing method comprising:

receiving an omnidirectional image/video content corresponding to a viewing sphere;

generating a projection-based frame according to the omnidirectional image/video content and an octahedron projection layout, wherein the projection-based frame has a 360-degree image/video content represented by triangular projection faces packed in the octahedron projection layout, the omnidirectional image/video content of the viewing sphere is mapped onto the triangular projection faces via an octahedron projection of the viewing sphere, and an equator of the viewing sphere is not mapped along any side of each of the triangular projection faces; and

encoding, by a video encoder, the projection-based frame to generate a part of a bitstream;

wherein there is an image content discontinuity boundary between sides of any two adjacent triangular projection faces in the octahedron projection layout; and

wherein the triangular projection faces packed in the octahedron projection layout comprise a first triangular projection face, a second triangular projection face, a third triangular projection face, a fourth triangular projection face, a fifth triangular projection face, a sixth triangular projection face, a seventh triangular projection face, and an eighth triangular projection face, one side of the first triangular projection face connects with one side of the second triangular projection face, one side of the third triangular projection face connects with another side of the second triangular projection face, one side of the fourth triangular projection face connects with another side of the third triangular projection face, one side of the fifth triangular projection face connects with another side of the fourth triangular projection face, one side of the sixth triangular projection face connects with another side of the fifth triangular projection face, one side of the seventh triangular projection face connects with another side of the sixth triangular projection face, the eighth triangular projection face is split into a first right-triangle-shaped part and a second right-triangle-shaped part, the first right-triangle-shaped part has a first side being one side of the eighth triangular projection face and a second side being a first part of another side of the eighth triangular projection face, the second right-triangle-shaped part has a first side being yet another side of the eighth triangular projection face and a second side being a second part of said another side of the eighth triangular projection face, the first side of the first right-triangle-shaped part connects with another side of the seventh triangular projection face, and the first side of the second right-triangle-shaped part connects with another side of the first triangular projection face.

17. A video processing method comprising:

receiving an omnidirectional image/video content corresponding to a viewing sphere;

generating a projection-based frame according to the omnidirectional image/video content and an octahedron projection layout, wherein the projection-based frame has a 360-degree image/video content represented by triangular projection faces packed in the octahedron projection layout, the omnidirectional image/video content of the viewing sphere is mapped onto the triangular projection faces via an octahedron projection of the viewing sphere, and an equator of the viewing sphere is not mapped along any side of each of the triangular projection faces; and

encoding, by a video encoder, the projection-based frame to generate a part of a bitstream;

wherein the triangular projection faces comprise a first triangular projection face and a second triangular projection face, and generating the projection-based frame according to the omnidirectional image/video content and the octahedron projection layout comprises;

before packing the first triangular projection face and the second triangular projection face, removing a plurality of boundary pixel samples at one side of the first triangular projection face; and

after the plurality of boundary pixel samples at said one side of the first triangular projection face are removed, packing the first triangular projection face and the second triangular projection face in the octahedron projection layout, wherein said one side of the first triangular projection face connects with one side of the second triangular projection face, and there is an image content continuity boundary between said one side of the first triangular projection face and said one side of the second triangular projection face.

18. A video processing apparatus comprising:

a conversion circuit, arranged to receive an omnidirectional image/video content corresponding to a viewing sphere, and generate a projection-based frame according to the omnidirectional image/video content and an octahedron projection layout, wherein the projection-based frame has a 360-degree image/video content represented by triangular projection faces packed in the octahedron projection layout, the omnidirectional image/video content of the viewing sphere is mapped onto the triangular projection faces via an octahedron projection of the viewing sphere, and an equator of the viewing sphere is not mapped along any side of each of the triangular projection faces; and

a video encoder, arranged to encode the projection-based frame to generate a part of a bitstream;

wherein the triangular projection faces packed in the octahedron projection layout comprise a first triangular projection face, a second triangular projection face and a third triangular projection face, one side of the first triangular projection face connects with one side of the second triangular projection face, one side of the third triangular projection face connects with another side of the second triangular projection face, there is an image content continuity boundary between said one side of the first triangular projection face and said one side of the second triangular projection face, and there is an image content continuity boundary between said one side of the third triangular projection face and said another side of the second triangular projection face; and

wherein the triangular projection faces packed in the octahedron projection layout further comprise a fourth triangular projection face, one side of the fourth triangular projection face connects with yet another side of the second triangular projection face, and there is an image content continuity boundary between said one side of the fourth triangular projection face and said yet another side of the second triangular projection face.