VIRTUAL REALITY CINEMA-IMMERSIVE MOVIE WATCHING FOR HEADMOUNTED DISPLAYS

• US 20200134780A1
• Filed: 10/22/2019
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Chapiro, Alexandre, Atluru, Chaitanya, Wan, Chun Chi, Lakshman, Haricharan, Rozzi, William, Ruggieri, Shane, Ninan, Ajit
• Granted Patent: US 10,769,754 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

Peripheral-vision expanded images are streamed to a video streaming client. The peripheral-vision expanded images are generated from source images in reference to view directions of the viewer at respective time points. View direction data is collected and received in real time while the viewer is viewing display images derived from the peripheral-vision expanded images. A second peripheral-vision expanded image is generated from a second source image in reference to a second view direction of the viewer at a second time point. The second peripheral-vision expanded image has a focal-vision image portion covering the second view direction of the viewer and a peripheral-vision image portion outside the focal-vision image portion. The second peripheral-vision expanded image is transmitted to the video streaming client.

Claims

1. A method for providing image data to support immersive media content consumption experiences, comprising:

streaming to a video streaming client, one or more peripheral-vision expanded images, the one or more peripheral-vision expanded images to be used by the video streaming client to generate one or more display images for rendering to a viewer at one or more time points, the one or more peripheral-vision expanded images being generated from one or more source images in reference to one or more view directions of the viewer at the one or more time points;

receiving at least a part of view direction data of the viewer collected in real time while the viewer is viewing the one or more display images, the view direction data being used to determine a second view direction of the viewer at a second time point subsequent to the one or more time points;

generating a second peripheral-vision expanded image from a second source image in reference to the second view direction of the viewer at the second time point, the second peripheral-vision expanded image to be used by the video streaming client to generate a second display image for rendering to the viewer at the second time point, the second peripheral-vision expanded image having a focal-vision image portion covering the second view direction of the viewer and a peripheral-vision image portion outside the focal-vision image portion, the focal-vision image portion of the second peripheral-vision expanded image being generated from a first corresponding source image portion in the second source image to cover the viewer'"'"'s focal vision as covered by the first corresponding source image portion, the peripheral-vision image portion of the second peripheral-vision expanded image being generated by spatially stretching a second corresponding source image portion in the second source image to cover the viewer'"'"'s peripheral vision including one or more peripheral vision portions not covered by the second corresponding source image portion of the second source image;

transmitting the second peripheral-vision expanded image to the video streaming client.

2. The method of claim 1, further comprising:

generating image metadata to be transmitted to the video streaming client, wherein the image metadata is used by the video streaming client to generate peripheral-vision expanded video content.

3. The method of claim 1, wherein the second peripheral-vision expanded image represents one of:

an immersive image, a panorama image, an augmented reality image, a virtual reality image, or a remote presence image.

4. The method of claim 1, wherein the one or more time points and the second time point collectively represent consecutive time points covering a time interval.

5. The method of claim 1, wherein the second focal-vision image portion in the second display image covers the viewer'"'"'s foveal-vision up to a maximum linear angular value:

between 2 and 4 degrees, between 4 and 6 degrees, between 6 and 8 degrees, or another angular value range.

6. The method of claim 1, wherein the one or more source images and the second source image are a part of two-dimensional (2D) video content designated to be presented on a rectangular image rendering surface with a relatively limited field of view.

7. The method of claim 1, wherein the one or more display images and the second display image are a part of peripheral-vision expanded video content designated to be presented on a target display of headmounted device of the viewer with a relatively large field of view.

8. The method of claim 1, wherein the one or more display images and the second display image are a part of peripheral-vision expanded video content designated to be presented on a target display of headmounted device of the viewer with a relatively large field of view.

9. The method of claim 1, wherein a sound source as visually depicted in the one or more display images and the second display image is moved from an original spatial position in the one or more source images and the second source image to a new spatial position in the one or more display images and the second display image;

the method further comprising;

causing an audio rendering system to render sounds designated to be emitted from the sound source from the new spatial position.

10. The method of claim 1, wherein the one or more display images and the second display image is rendered with a curved screen;

and wherein the curved screen is a virtual display screen.

11. The method of claim 1, wherein the one or more display images and the second display image comprise three-dimensional (3D) display image portions.

12. The method of claim 11, wherein the one or more source images and the second source image represent two-dimensional (2D) images;

and wherein the 3D display image portions are generated using depth data portions generated from applying 2D to 3D conversion operations on corresponding 2D source image portions in the one or more source images and the second source image.

13. The method of claim 1, wherein one or more depicted objects in the peripheral-vision image portion of the second peripheral-vision expanded image are seamlessly stretched as compared with one or more corresponding depicted objects in the second corresponding image portion of the second source image.

14. The method of claim 1, wherein the peripheral-vision image portion of the second peripheral-vision expanded image is of a relatively low resolution as compared with the focal-vision image portion of the second peripheral-vision expanded image.

17. An apparatus performing the method of claim 1.

18. A system performing the method of claim 1.

19. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of the method recited in claim 1.

20. A computing device comprising one or more processors and one or more non-transitory computer-readable storage media, storing a set of instructions, which when executed by one or more processors cause performance of the method recited in method claim 1.

15. A method for providing immersive media content consumption experiences, comprising:

rendering, to a viewer, one or more display images at one or more time points, the one or more display images being generated from one or more peripheral-vision expanded images received from a video streaming server, the one or more peripheral-vision expanded images being generated by the video streaming server from one or more source images in reference to one or more view directions of the viewer at the one or more time points;

collecting, in real time, view direction data of the viewer collected while the viewer is viewing the one or more display images, the view direction data being used to determine a second view direction of the viewer at a second time point subsequent to the one or more time points;

sending at least a part of the view direction data to the video streaming server to cause the video streaming server to generate a second peripheral-vision expanded image from a second source image in reference to the second view direction of the viewer at the second time point, the second peripheral-vision expanded image to be used to generate a second display image for rendering to the viewer at the second time point, the second peripheral-vision expanded image having a focal-vision image portion covering the second view direction of the viewer and a peripheral-vision image portion outside the focal-vision image portion, the focal-vision image portion of the second peripheral-vision expanded image being generated from a first corresponding source image portion in the second source image to cover the viewer'"'"'s focal vision as covered by the first corresponding source image portion, the peripheral-vision image portion of the second peripheral-vision expanded image being generated by spatially stretching a second corresponding source image portion in the second source image to cover the viewer'"'"'s peripheral vision including one or more peripheral vision portions not covered by the second corresponding source image portion of the second source image;

receiving the second peripheral-vision expanded image from the video streaming server;

generating the second display image from the second peripheral-vision expanded image;

rendering at the second time point, to the viewer, the second display image.

16. The method of claim 15, wherein the one or more display images and the second display image are rendered on a target display on a headmounted device of the viewer.