IMMERSIVE STEREOSCOPIC VIDEO ACQUISITION, ENCODING AND VIRTUAL REALITY PLAYBACK METHODS AND APPARATUS

• US 20170150122A1
• Filed: 01/02/2017
• Published: 05/25/2017
• Est. Priority Date: 04/11/2013
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David Michael
• Granted Patent: US 10,750,154 B2
• Time in Patent Office: Days
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Methods and apparatus for capturing and displaying stereoscopic images are described in a manner that allows a user to obtain a 3d virtual reality experience simulating that of being in a seat at a football game or other event. Rear images are modified, e.g., in luminance intensity, to make them consistent with the luminance intensity of the forward images to avoid or reduce edges or differences in luminance intensity as a users turns his head from viewing a main image area to a side or rear image area. A seamless 3D presentation is made possible through the use of fisheye lenses at capture time and combining of images corresponding to forward and rear image areas as a user turns his or her head requiring a change in the captured image area which is displayed to the user.

Claims

1. A method of providing content, comprising:

capturing one or more images corresponding to a first viewing direction;

capturing one or more images corresponding to a second viewing direction which is different from said first direction; and

generating an image based on at least a portion of one of the one or more captured images corresponding to the first direction and at least a portion of one of the one or more captured images corresponding to the second direction, said generating an image including performing an image adjustment operation.

2. The method of claim 1, wherein performing an image adjustment operation includes:

determining a luminance of a portion of a captured image in the first viewing direction; and

adjusting a luminance of a portion of a captured image in the second viewing direction based on the luminance of the portion of the captured image in the first viewing direction.

3. The method of claim 1 wherein capturing one or more images corresponding to a first viewing direction includes capturing one or more images using a first camera corresponding to the first viewing direction, said first camera including a fish eye lens.

4. The method of claim 3, wherein said fish eye lens capture an image corresponding to a 180 degree viewing area or approximately a 180 degree viewing area.

5. The method of claim 3, wherein said fish eye lens captures an image corresponding to at least a 120 degree area.

6. The method of claim 3, wherein the first viewing direction is a forward direction relative to a first camera used to capture one or more images corresponding to the first viewing direction and the second viewing direction is a rear direction relative to the first camera.

7. The method of claim 1, wherein the one or more images corresponding to the second direction are captured at a different time than said one or more images corresponding to the first viewing direction.

8. The method of claim 7, further comprising:

capturing additional images corresponding to the first viewing direction at a video frame rate.

9. The method of claim 8, further comprising:

capturing additional images corresponding to the second viewing direction at a second rate which is lower than said video frame rate.

10. The method of claim 9, wherein said second rate is less than ⅕

said video frame rate.

11. The method of claim 10, wherein capturing images corresponding to the second viewing direction is performed using a lower resolution camera than is used to capture images in the first viewing direction.

12. The method of claim 1, further comprising:

making depth measurements of an area surrounding a mount on which cameras used to capture said images corresponding to the first viewing direction are mounted.

13. The method of claim 12, making depth measurements of an area surrounding a mount include making LIDAR measurements.

14. The method of claim 12, further comprising:

generating a depth map for the area surrounding said mount; and

transmitting depth map information with at least some image information corresponding to an image captured by a camera on said mount.

15. The method of claim 14, further comprising:

using said depth map information when generating an image from one or more images captured by a camera on said mount.

16. A system, comprising:

a first camera for capturing one or more images corresponding to a first viewing direction;

a second camera for capturing one or more images corresponding to a second viewing direction which is different from said first direction; and

a player rendering an image based on at least a portion of a captured image corresponding to the first direction and at least a portion of a captured image corresponding to the second direction, said player performing an adjustment operation to at least portion of an image as part of said rendering; and

a display for displaying the images generated by said rendering.

17. The system of claim 16, wherein said player includes a luminance controller that controls the luminance of a rear portion of an image as part of a luminance adjustment operation, based on the luminance of a front portion of an image when combining front and rear image portions as part of the image rendering process.

18. The system of claim 17, wherein said luminance controller limits maximum luminance value of the rear image portion to be equal to or less than the maximum luminance value of the front image portion being included in a rendered image.

19. The system of claim 18, wherein the luminance of a front image portion is used by said luminance controller to control the luninance of a rear image portion.

20. The system of claim 19, where front and rear image portions which are combined by said player when rendering an image in response to a user turning his hear head to a left or right side are images which were captured at different points in time.