Fisheye rendering with lens distortion correction for 360-degree video

• US 10,699,389 B2
• Filed: 04/24/2017
• Issued: 06/30/2020
• Est. Priority Date: 05/24/2016
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Van der Auwera, Geert, Coban, Muhammed, Karczewicz, Marta
• Primary Examiner(s): Hasan, Mainul
• Application Number:
  US15/495,709
• Publication Number:
  US 20170345136A1
• Time in Patent Office: 1,163 Days
• CPC Class Codes: G06T 3/08 Projecting images onto non-planar surfaces, e.g. geodetic screens
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

In various implementations, computing systems and computer-implemented methods can be used for correcting the distortion present in a fisheye image, and rendering the image for display as 360-degree video. In various implementations, a computing device can receive 2-dimensional video data captured by an omnidirectional camera. The computing device can map an image from each video frame to a 3-dimensional hemispherical representation. In various implementations, this mapping can be executed using a polynomial model. The 3-dimensional hemispherical representation can then be used in a 360-degree video presentation, to provide a virtual reality experience.

Claims

1. A method of processing 360-degree video data, comprising:

obtaining two-dimensional video data captured by an omnidirectional camera, wherein the two-dimensional video data includes an image of a scene, and wherein, in the image, the scene has been warped into a circular region of the image;

determining, for pixel locations in the circular region of the image, corresponding locations on a three-dimensional hemispherical representation of the image, wherein determining the corresponding locations includes applying a function that projects a point in the image, represented by a two-dimensional coordinate defined by a first angle in an image plane of the image and a distance from a center of the circular region of the image, to a corresponding point on the three-dimensional hemispherical representation, represented by a three-dimensional coordinate defined by a second angle in the image plane and a third angle in an orthogonal plane from the image plane;

mapping, using the two-dimensional coordinate and the three-dimensional coordinate, a pixel in the circular region of the image to the three-dimensional hemispherical representation of the image, wherein the point in the image provides the pixel to map to the point on the three-dimensional hemispherical representation; and

outputting the three-dimensional hemispherical representation for display in a 360-degree video presentation,

wherein the function is a polynomial function, of degree at least 2, in the distance from the center of the circular region of the image, and wherein the function applies a parameter of the omnidirectional camera to determine the three-dimensional coordinate.

2. The method of claim 1, wherein the warping is non-linear, and further comprising:

determining the two-dimensional coordinate using an adjusted radial value, wherein the adjusted radial value corresponds to the non-linearity of the warping.

3. The method of claim 1, wherein the image includes a field of view that is greater than 180 degrees, and further comprising:

adjusting the two-dimensional coordinate using a radial value that has been scaled according to the field of view; and

using the adjusted two-dimensional coordinate in the mapping.

4. The method of claim 1, wherein the image includes a field of view of at least 180 degrees.

5. The method of claim 1, wherein the omnidirectional camera includes a fisheye lens.

6. The method of claim 1, wherein the 360-degree video data is obtained from an encoded bitstream, wherein the encoded bitstream includes the parameter of the omnidirectional camera.

7. The method of claim 6, wherein the parameter of the omnidirectional camera is encoded in a supplemental enhancement information (SEI) message in the encoded bitstream.

8. The method of claim 6, wherein the encoded bitstream is included in a file formatted according to an ISO base media file format, and wherein the parameter of the omnidirectional camera is encoded in a structure in the file.

9. The method of claim 6, wherein the parameter of the omnidirectional camera is a polynomial order, a polynomial coefficient, a polynomial scaling parameter, or a mapping scaling parameter.

10. An apparatus for processing encoded video, comprising:

a memory configured to store two-dimensional video data captured by an omnidirectional camera; and

a processor implemented in circuitry and configured to;

obtain the two-dimensional video data, wherein the two-dimensional video data includes an image of a scene, and wherein, in the image, the scene has been warped into a circular region of the image;

determine, for pixel locations in the circular region of the image, corresponding locations on a three-dimensional hemispherical representation of the image, wherein determining the corresponding locations includes applying a function that projects a point in the image, represented by a two-dimensional coordinate defined by a first angle in an image plane of the image and a distance from a center of the circular region of the image, to a corresponding point on the three-dimensional hemispherical representation, represented by a three-dimensional coordinate defined by a second angle in the image plane and a third angle in an orthogonal plane from the image plane;

map, using the two-dimensional coordinate and the three-dimensional coordinate, a pixel in the circular region of the image to the three-dimensional hemispherical representation of the image, wherein the point in the image provides the pixel to map to the point on the three-dimensional hemispherical representation; and

output the three-dimensional hemispherical representation for display in a 360-degree video presentation,

wherein the function is a polynomial function, of degree at least 2, in the distance from the center of the circular region of the image, and wherein the function applies a parameter of the omnidirectional camera to determine the three-dimensional coordinate.

11. The apparatus of claim 10, wherein the warping is non-linear, and wherein the processor is further configured to:

determine the two-dimensional coordinate using an adjusted radial value, wherein the adjusted radial value corresponds to the non-linearity of the warping.

12. The apparatus of claim 10, wherein the image includes a field of view that is greater than 180 degrees, and wherein the processor is further configured to:

adjust the two-dimensional coordinate using a radial value that has been scaled according to the field of view; and

use the adjusted two-dimensional coordinate in the mapping.

13. The apparatus of claim 10, wherein the image includes a field of view of at least 180 degrees.

14. The apparatus of claim 10, wherein the omnidirectional camera includes a fisheye lens.

15. The apparatus of claim 10, wherein the 360-degree video data is obtained from an encoded bitstream, wherein the encoded bitstream includes the parameter of the omnidirectional camera.

16. The apparatus of claim 15, wherein the parameter of the omnidirectional camera is encoded in a supplemental enhancement information (SEI) message in the encoded bitstream.

17. The apparatus of claim 15, wherein the encoded bitstream is included in a file formatted according to an ISO base media file format, and wherein the parameter of the omnidirectional camera is encoded in a structure in the file.

18. The apparatus of claim 15, wherein the parameter of the omnidirectional camera is a polynomial order, a polynomial coefficient, a polynomial scaling parameter, or a mapping scaling parameter.

19. The apparatus of claim 10, further comprising:

a mobile device, wherein the mobile device includes the omnidirectional camera.

20. An apparatus for decoding video, comprising:

a memory configured to store two-dimensional video data captured by an omnidirectional camera; and

a processor implemented in circuitry, configured to;

obtain the two-dimensional video data, wherein the two-dimensional video data includes an image of a scene, and wherein, in the image, the scene has been warped into a circular region of the image;

determine, for pixel locations in the circular region of the image, corresponding locations on a three-dimensional hemispherical representation of the image, wherein determining the corresponding locations includes applying a function that projects a point in the image, represented by a two-dimensional coordinate defined by a first angle in an image plane of the image and a distance from a center of the circular region of the image, to a corresponding point on the three-dimensional hemispherical representation, represented by a three-dimensional coordinate defined by a second angle in the image plane and a third angle in an orthogonal plane from the image plane;

map, using the two-dimensional coordinate and the three-dimensional coordinate, a pixel in the circular region of the image to the three-dimensional hemispherical representation of the image, wherein the point in the image provides the pixel to map to the point on the three-dimensional hemispherical representation; and

output the three-dimensional hemispherical representation for display in a 360-degree video presentation,

wherein the function is a polynomial function, of degree at least 2, in the distance from the center of the circular region of the image, and wherein the function applies a parameter of the omnidirectional camera to determine the three-dimensional coordinate.

21. The apparatus of claim 20, wherein the warping is non-linear, and wherein the processor is further configured to:

determine the two-dimensional coordinate using an adjusted radial value, wherein the adjusted radial value corresponds to the non-linearity of the warping.

22. The apparatus of claim 20, wherein the image includes a field of view that is greater than 180 degrees, and wherein the processor is further configured to:

adjust the two-dimensional coordinate using a radial that has been scaled according to the field of view; and

use the adjusted two-dimensional coordinate in the mapping.

23. The apparatus of claim 20, wherein the image includes a field of view of at least 180 degrees.

24. The apparatus of claim 20, wherein the omnidirectional camera includes a fisheye lens.

25. The apparatus of claim 20, wherein the 360-degree video data is obtained from an encoded bitstream, wherein the encoded bitstream includes the parameter of the omnidirectional camera.

26. The apparatus of claim 25, wherein the parameter of the omnidirectional camera is encoded in a supplemental enhancement information (SEI) message in the encoded bitstream.

27. The apparatus of claim 25, wherein the encoded bitstream is included in a file formatted according to an ISO base media file format, and wherein the parameter of the omnidirectional camera is encoded in a structure in the file.

28. The apparatus of claim 25, wherein the parameter of the omnidirectional camera is a polynomial order, a polynomial coefficient, a polynomial scaling parameter, or a mapping scaling parameter.

29. The apparatus of claim 20, further comprising:

a mobile device, wherein the mobile device includes the omnidirectional camera.

30. The apparatus of claim 20, further comprising:

a mobile device, wherein the mobile device includes a display for displaying the 360-degree video presentation.

31. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:

obtain two-dimensional video data captured by an omnidirectional camera, wherein the two-dimensional video data includes an image of a scene, and wherein, in the image, the scene has been warped into a circular region of the image;

determine, for pixel locations in the circular region of the image, corresponding locations on a three-dimensional hemispherical representation of the image, wherein determining the corresponding locations includes applying a function that projects a point in the image, represented by a two-dimensional coordinate defined by a first angle in an image plane of the image and a distance from a center of the circular region of the image, to a corresponding point on the three-dimensional hemispherical representation, represented by a three-dimensional coordinate defined by a second angle in the image plane and a third angle in an orthogonal plane from the image plane;

map, using the two-dimensional coordinate and the three-dimensional coordinate, a pixel in the circular region of the image to the three-dimensional hemispherical representation of the image, wherein the point in the image provides the pixel to map to the point on the three-dimensional hemispherical representation; and

output the three-dimensional hemispherical representation for display in a 360-degree video presentation,

wherein the function is a polynomial function, of degree at least 2, in the distance from the center of the circular region of the image, and wherein the function applies a parameter of the omnidirectional camera to determine the three-dimensional coordinate.

32. An apparatus, comprising:

means for obtaining two-dimensional video data, wherein the two-dimensional video data includes an image of a scene, and wherein, in the image, the scene has been warped into a circular region of the image;

means for determining, for pixel locations in the circular region of the image, corresponding locations on a three-dimensional hemispherical representation of the image, wherein determining the corresponding locations includes applying a function that projects a point in the image, represented by a two-dimensional coordinate defined by a first angle in an image plane of the image and a distance from a center of the circular region of the image, to a corresponding point on the three-dimensional hemispherical representation, represented by a three-dimensional coordinate defined by a second angle in the image plane and a third angle in an orthogonal plane from the image plane;

means for mapping, using the two-dimensional coordinate and the three-dimensional coordinate, a pixel in the circular region of the image to the three-dimensional hemispherical representation of the image, wherein the point in the image provides the pixel to map to the point on the three-dimensional hemispherical representation; and

means for outputting the three-dimensional hemispherical representation for display in a 360-degree video presentation,

wherein the function is a polynomial function, of degree at least 2, in the distance from the center of the circular region of the image, and wherein the function applies a parameter of the omnidirectional camera to determine the three-dimensional coordinate.