VIEW-OPTIMIZED LIGHT FIELD IMAGE AND VIDEO STREAMING

• US 20180096494A1
• Filed: 09/29/2017
• Published: 04/05/2018
• Est. Priority Date: 09/30/2016
• Status: Active Application

• Current Assignee: Visbit Inc.
• Original Assignee: Visbit Inc.
• Inventors: ZHOU, Changyin
• Application Number:
  US15/720,395
• Publication Number:
  US 20180096494A1
• Time in Patent Office: Days
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/10076 4D tomography; Time-sequential 3D tomography
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

The present disclosure relates to methods and systems for providing virtual reality video content. An example method includes capturing light field image data with a plurality of cameras. The light field image data includes a plurality of sample data points. The method includes determining a viewpoint position and a viewpoint pose. The method further includes determining a nearest neighbor set based on the sample data points, the viewpoint position, and the viewpoint pose. The method also includes interpolating within the nearest neighbor set so as to form a set of resampled data points. The method yet further includes rendering a 360° image from the resampled data points. The 360° image includes a representation of the light field based on the viewpoint position and the viewpoint pose.

Claims

1. A system comprising:

a plurality of cameras disposed so as to capture images of a light field; and

a controller comprising at least one processor and a memory, wherein the at least one processor executes instructions stored in the memory so as to carry out operations, the operations comprising;

causing the plurality of cameras to capture light field image data, wherein the light field image data comprises a plurality of sample data points;

determining a viewpoint position and a viewpoint pose;

determining a nearest neighbor set based on the sample data points, the viewpoint position, and the viewpoint pose;

interpolating within the nearest neighbor set so as to form a set of resampled data points; and

rendering a 360°

image from the set of resampled data points, wherein the 360°

image comprises a representation of the light field based on the viewpoint position and the viewpoint pose.

2. The system of claim 1, wherein the plurality of cameras is disposed on a sphere, wherein each camera of the plurality of cameras is arranged facing outwards from the sphere.

3. The system of claim 1, wherein interpolating within the nearest neighbor set comprises carrying out a linear interpolation or a polynomial interpolation.

4. The system of claim 1, wherein the operations further comprise partitioning the light field image data into a plurality of light field tiles, wherein at least one light field tile is ignored or deleted based on the viewpoint position and the viewpoint pose, wherein the light field tiles comprise light field tiles within a field of view and light field tiles outside the field of view.

5. The system of claim 4, wherein partitioning the light field image data comprises partitioning the light field image data into a plurality of four-dimensional or three-dimensional tiles.

6. The system of claim 4, wherein partitioning the light field image data comprises partitioning the light field image data based on a Fourier transform or a Wavelet transform, wherein the light field tiles comprise similar ranges of spatial frequency information.

7. The system of claim 4, further comprising a client device, wherein the operations further comprise streaming the light field tiles within the field of view to the client device.

8. The system of claim 1, further comprising a sensor, wherein determining the viewpoint position and the viewpoint pose comprises receiving, via the sensor, information indicative of at least one of:

the viewpoint position or the viewpoint pose.

9. The system of claim 8, further comprising a client device, wherein the operations further comprise:

determining, based on the received sensor information, a predicted motion;

determining, based on the predicted motion, at least one predicted viewable light field tile; and

streaming the at least one predicted viewable light field tile to the client device.

10. The system of claim 9, wherein streaming the at least one predicted viewable light field tile comprises streaming a low resolution representation of the respective predicted viewable light field tile and a high resolution representation of the respective predicted viewable light field tile, wherein the operations further comprise initially presenting at least a portion of the low resolution representation to a user and when at least a threshold amount of the high resolution representation has been streamed to the client device, subsequently presenting at least a portion of the high resolution representation to the user.

11. The system of claim 9, wherein determining the at least one predicted viewable light field tile comprises determining the predicted motion is greater than a threshold amount toward a respective direction of the predicted viewable light field tile with respect to a current viewpoint position and a current viewpoint pose.

12. The system of claim 1, wherein the operations further comprise:

pre-rendering a plurality of viewable light field tiles in high resolution; and

pre-rendering at least a portion of the light field image data in low resolution.

13. The system of claim 1, wherein the plurality of sample data points has a variable sample density based at least on the viewpoint position and the viewpoint pose.

14. A method comprising:

capturing, with a plurality of cameras, light field image data from a light field, wherein the light field image data comprises a plurality of sample data points;

determining a viewpoint position and a viewpoint pose;

determining a nearest neighbor set based on the sample data points, the viewpoint position, and the viewpoint pose;

interpolating within the nearest neighbor set so as to form a set of resampled data points; and

rendering a 360°

image from the set of resampled data points, wherein the 360°

image comprises a representation of the light field based on the viewpoint position and the viewpoint pose.

15. The method of claim 14, further comprising interpolating within the nearest neighbor set comprises carrying out a linear interpolation or a polynomial interpolation.

16. The method of claim 14, further comprising partitioning the light field image data into a plurality of light field tiles, wherein at least one light field tile is ignored or deleted based on the viewpoint position and the viewpoint pose, wherein the light field tiles comprise viewable light field tiles and unviewable light field tiles.

17. The method of claim 16, wherein partitioning the light field image data comprises partitioning the light field image data into a plurality of four-dimensional or three-dimensional tiles.

18. The method of claim 16, wherein partitioning the light field image data comprises partitioning the light field image data based on a Fourier transform or a Wavelet transform, wherein the light field tiles comprise similar ranges of spatial frequency information.

19. The method of claim 16, further comprising streaming the viewable light field tiles to a client device.

20. The method of claim 14, wherein determining the viewpoint position and the viewpoint pose comprises receiving, via a sensor, information indicative of at least one of:

the viewpoint position or the viewpoint pose, wherein the method further comprises;

determining, based on the received sensor information, a predicted motion;

determining, based on the predicted motion, at least one predicted viewable light field tile; and

streaming the at least one predicted viewable light field tile to a client device.