Method and apparatus for generating three-dimensional panoramic video

• US 10,643,302 B2
• Filed: 06/13/2018
• Issued: 05/05/2020
• Est. Priority Date: 08/16/2017
• Status: Active Grant

• Current Assignee: VIA Technologies Incorporated (VIA Technologies)
• Original Assignee: VIA Technologies Incorporated (VIA Technologies)
• Inventors: Cheng, Robin J.
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Joisil, Berteau
• Application Number:
  US16/006,874
• Publication Number:
  US 20190057487A1
• Time in Patent Office: 692 Days
• CPC Class Codes: G06T 3/16 Spatio-temporal transformations, e.g. video cubism
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A method and an apparatus for generating 3D panoramic video are provided. In the method, plural frames are captured from a panoramic video. Each frame is transformed into a polyhedral mapping projection comprising side planes, a top plane and a bottom plane. Displacements of pixels in the side planes are calculated by using the side planes of each frame, and displacements of pixels in the top plane and the bottom plane are calculated by using the displacements of the side planes. Then, the pixels in the side planes, the top plane and the bottom plane of each frame are shifted according the displacements of the polyhedral mapping projection to generate a shifted polyhedral mapping projection. The shifted polyhedral mapping projection is transformed into a shifted frame with 2D space format. The shifted frames and corresponding frames construct 3D images and the 3D images are encoded into a 3D panoramic video.

Claims

1. A method for generating a three-dimensional panoramic video, for use in an electronic apparatus having a processor, and comprising:

capturing a plurality of frames from a panoramic video;

transforming each of the frames into a polyhedral mapping projection, wherein the polyhedral mapping projection comprises a plurality of side planes, a top plane and a bottom plane;

calculating displacements of a plurality of pixels in the side planes by using the side planes of the transformed polyhedral mapping projection of each of the frames, and calculating displacements of a plurality of pixels in the top plane and the bottom plane of the polyhedral mapping projection by using the displacements of the pixels of the side planes;

shifting the pixels in the side planes, the top plane and the bottom plane of the transformed polyhedral mapping projection of each of the frames according the calculated displacements of the polyhedral mapping projection to generate a shifted polyhedral mapping projection; and

transforming, for each of the frames, the shifted polyhedral mapping projection into a shifted frame with a two-dimensional space format, and combining the shifted frame and its corresponding frame into a three-dimensional image, and encoding the three-dimensional images of the frames into a three-dimensional panoramic video.

2. The method as claimed in claim 1, wherein the step of calculating the displacements of the pixels in the side planes by using the side planes of the transformed polyhedral mapping projection of each of the frames comprises:

calculating an initial displacement of each of the pixels in the side planes of the transformed polyhedral mapping projection of each of the frames, and multiplying the initial displacement by a normalization factor to obtain the displacement of each of the pixels, wherein the normalization factor is proportional to a horizontal resolution of the frames in the panoramic video.

3. The method as claimed in claim 1, wherein the step of calculating the displacements of the pixels in the top plane and the bottom plane of the polyhedral mapping projection by using the displacements of the pixels of the side planes comprises:

calculating the displacements of a plurality of edge pixels in the top plane and the bottom plane located at boundaries with each of the side planes according to the displacements of a plurality of edge pixels in each of the side planes located at boundaries with the top plane and the bottom plane; and

calculating the displacements of other pixels in the top plane and the bottom plane according to the displacements of the edge pixels in the top plane and the bottom plane.

4. The method as claimed in claim 3, wherein the step of calculating the displacements of the other pixels in the top plane and the bottom plane according to the displacements of the edge pixels in the top plane and the bottom plane comprises:

respectively segmenting the top plane and the bottom plane into a plurality of blocks, and calculating the displacement of each of the other pixels in the top plane and the bottom plane according to displacements of a plurality of adjacent pixels around each of the other pixels in the corresponding block.

5. The method as claimed in claim 4, wherein the step of calculating the displacement of each of the other pixels in the top plane and the bottom plane according to the displacements of the adjacent pixels around each of the other pixels in the corresponding block comprises:

calculating the displacement of each of the other pixels according to the displacements of the adjacent pixels around each of the other pixels and a decrement parameter, wherein the decrement parameter is used for decreasing the displacements of internal pixels such that the more the internal pixel is close to a center of the top plane or the bottom plane, the smaller the displacement thereof is.

6. The method as claimed in claim 1, wherein the step of shifting the pixels in the side planes, the top plane and the bottom plane of the transformed polyhedral mapping projection of each of the frames according the calculated displacements of the polyhedral mapping projection to generate the shifted polyhedral mapping projection comprises:

shifting the pixels of the side planes rightward or leftward in a horizontal wraparound manner according to the displacements of the side planes to generate shifted side planes;

rotating the pixels in the top plane in one of a clockwise direction and a counterclockwise direction according to the displacements of the top plane to generate a shifted top plane; and

rotating the pixels in the bottom plane in the other one of the clockwise direction and the counterclockwise direction according to the displacements of the bottom plane to generate a shifted bottom plane, wherein the shifted side planes, the shifted top plane and the shifted bottom plane constitute the shifted polyhedral mapping projection.

7. The method as claimed in claim 6, wherein the step of rotating the pixels in the top plane in one of the clockwise direction and the counterclockwise direction according to the displacements of the top plane to generate the shifted top plane and rotating the pixels in the bottom plane in the other one of the clockwise direction and the counterclockwise direction according to the displacements of the bottom plane to generate the shifted bottom plane comprises:

segmenting the top plane into a plurality of blocks from a center thereof, and shifting each of the pixels of each of the blocks in a direction approximate to one of the clockwise direction and the counterclockwise direction according to the displacements of the top plane, wherein when the pixels are shifted out of the block and in to an adjacent block, the pixels are turned according to the direction in the adjacent block and are continuously shifted; and

segmenting the bottom plane into a plurality of blocks from a center thereof, and shifting each of the pixels of each of the blocks in a direction approximate to the other one of the clockwise direction and the counterclockwise direction according to the displacements of the bottom plane, wherein when the pixels are shifted out of the block and in to an adjacent block, the pixels are turned according to the direction in the adjacent block and are continuously shifted.

8. The method as claimed in claim 6, wherein when the pixels of the side planes are shifted rightward in the horizontal wraparound manner, the pixels of the top plane are rotated in the counterclockwise direction and the pixels of the bottom plane are rotated in the clockwise direction;

when the pixels of the side planes are shifted leftward in the horizontal wraparound manner, the pixels of the top plane are rotated in the clockwise direction and the pixels of the bottom plane are rotated in the counterclockwise direction.

9. The method as claimed in claim 1, wherein the polyhedral mapping projection is a cube mapping projection.

10. The method as claimed in claim 1, wherein the step of calculating the displacements of the pixels in the side planes by using the side planes of the transformed polyhedral mapping projection of each of the frames comprises:

using a principal component analysis optical flow method to calculate the displacements of the pixels in the side planes.

11. A three-dimensional panoramic video generating apparatus, comprising:

a connection device, connected to an image source device, and receiving a panoramic video from the image source device;

a storage device, storing a plurality of modules; and

a processor, coupled to the connection device and the storage device, and loading and executing the modules in the storage device, wherein the modules comprise;

a frame capturing module, capturing a plurality of frames from the panoramic video;

a mapping module, transforming each of the frames into a polyhedral mapping projection, wherein the polyhedral mapping projection comprises a plurality of side planes, a top plane and a bottom plane;

a parallax calculating module, calculating displacements of a plurality of pixels in the side planes by using the side planes of the transformed polyhedral mapping projection of each of the frames, and calculating displacements of a plurality of pixels in the top plane and the bottom plane of the polyhedral mapping projection by using the displacements of the pixels of the side planes;

a pixel shifting module, shifting the pixels in the side planes, the top plane and the bottom plane of the transformed polyhedral mapping projection of each of the frames according the calculated displacements of the polyhedral mapping projection to generate a shifted polyhedral mapping projection;

a transforming module, transforming the shifted polyhedral mapping projection of each of the frames into a shifted frame with a two-dimensional space format; and

a video encoding module, combining the shifted frame and its corresponding frame into a three-dimensional image and encoding the three-dimensional images of the frames into a three-dimensional panoramic video.

12. The three-dimensional panoramic video generating apparatus as claimed in claim 11, wherein the parallax calculating module calculates an initial displacement of each of the pixels in the side planes of the transformed polyhedral mapping projection of each of the frames, and multiplies the initial displacement by a normalization factor to obtain the displacement of each of the pixels, wherein the normalization factor is proportional to a horizontal resolution of the frames in the panoramic video.

13. The three-dimensional panoramic video generating apparatus as claimed in claim 11, wherein the parallax calculating module calculates the displacements of a plurality of edge pixels in the top plane and the bottom plane located at boundaries with each of the side planes according to the displacements of a plurality of edge pixels in each of the side planes located at boundaries with the top plane and the bottom plane, and calculates the displacements of other pixels in the top plane and the bottom plane according to the displacements of the edge pixels in the top plane and the bottom plane.

14. The three-dimensional panoramic video generating apparatus as claimed in claim 13, wherein the parallax calculating module respectively segments the top plane and the bottom plane into a plurality of blocks, and calculates the displacement of each of the other pixels in the top plane and the bottom plane according to displacements of a plurality of adjacent pixels around each of the other pixels in the corresponding block.

15. The three-dimensional panoramic video generating apparatus as claimed in claim 14, wherein the parallax calculating module calculates the displacement of each of the other pixels according to the displacements of the adjacent pixels around each of the other pixels and a decrement parameter, wherein the decrement parameter is used for decreasing the displacements of internal pixels such that the more the internal pixel is close to a center of the top plane or the bottom plane, the smaller the displacement thereof is.

16. The three-dimensional panoramic video generating apparatus as claimed in claim 11, wherein the pixel shifting module shifts the pixels of the side planes rightward or leftward in a horizontal wraparound manner according to the displacements of the side planes to generate shifted side planes, rotates the pixels in the top plane in one of a clockwise direction and a counterclockwise direction according to the displacements of the top plane to generate a shifted top plane, and rotates the pixels in the bottom plane in the other one of the clockwise direction and the counterclockwise direction according to the displacements of the bottom plane to generate a shifted bottom plane, wherein the shifted side planes, the shifted top plane and the shifted bottom plane constitute the shifted polyhedral mapping projection.

17. The three-dimensional panoramic video generating apparatus as claimed in claim 16, wherein the pixel shifting module segments the top plane into a plurality of blocks from a center thereof, and shifts each of the pixels of each of the blocks in a direction approximate to one of the clockwise direction and the counterclockwise direction according to the displacements of the top plane, wherein when the pixels are shifted out of the block and in to an adjacent block, the pixels are turned according to the direction in the adjacent block and are continuously shifted;

and segments the bottom plane into a plurality of blocks from a center thereof, and shifts each of the pixels of each of the blocks in a direction approximate to the other one of the clockwise direction and the counterclockwise direction according to the displacements of the bottom plane, wherein when the pixels are shifted out of the block and in to an adjacent block, the pixels are turned according to the direction in the adjacent block and are continuously shifted.

18. The three-dimensional panoramic video generating apparatus as claimed in claim 16, wherein when the pixels of the side planes are shifted rightward in the horizontal wraparound manner, the pixels of the top plane are rotated in the counterclockwise direction and the pixels of the bottom plane are rotated in the clockwise direction;

when the pixels of the side planes are shifted leftward in the horizontal wraparound manner, the pixels of the top plane are rotated in the clockwise direction and the pixels of the bottom plane are rotated in the counterclockwise direction.

19. The three-dimensional panoramic video generating apparatus as claimed in claim 11, wherein the polyhedral mapping projection is a cube mapping projection.

20. The three-dimensional panoramic video generating apparatus as claimed in claim 11, wherein the parallax calculating module uses a principal component analysis optical flow method to calculate the displacements of the pixels in the side planes.