Systems and methods for creating an immersive video content environment

• US 10,523,929 B2
• Filed: 04/27/2016
• Issued: 12/31/2019
• Est. Priority Date: 04/27/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Watson, Scott F., Smoot, Lanny S.
• Primary Examiner(s): Beck, Leron
• Application Number:
  US15/140,375
• Publication Number:
  US 20170318283A1
• Time in Patent Office: 1,343 Days
• Field of Search: 348 51
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18

Abstract

There is provided a system including a non-transitory memory storing an executable code and a hardware processor executing the code to receive a video content including a plurality of frames showing a scene from a perspective of a real camera, create a three-dimensional (3D) model of the scene using the plurality of frames, store the 3D model of the scene in the non-transitory memory, construct a synthetic view of the scene showing additional perspectives from one or more virtual cameras at one or more locations in the scene, transmit the synthetic view of the scene for being displayed on a display, display a scene of the video content on the display, track a position of a viewer moving in a room, and adjust the display of the scene being displayed on the display based on the position of the viewer in the room relative to the display.

Claims

1. A system comprising:

a non-transitory memory storing an executable code;

a hardware processor executing the executable code to;

receive a video content including a plurality of frames showing a scene from a perspective of a real camera;

create a three-dimensional (3D) model of the scene using the plurality of frames;

store the 3D model of the scene in the non-transitory memory;

construct a synthetic view showing the scene from a first perspective of a virtual camera at a first location in the scene;

transmit the synthetic view of the scene for being displayed on a display located at a fixed position in a room;

track a position of a viewer in the room relative to the fixed position of the display in the room, using a camera or a wireless identification carried by the viewer, to determine a current position of the viewer in the room as the viewer is moving along a path from a first position in the room to a second position in the room at a distance from the first position;

dynamically adjust the synthetic view to show the scene from a second perspective of the virtual camera at a second location in the scene, wherein the dynamically adjusted synthetic view of the scene is an angularly accurate display of the scene based on the current position of the viewer in the room relative to the fixed position of the display in the room; and

transmit the dynamically adjusted synthetic view of the scene for being displayed on the display such that the scene appears to the viewer as if the real camera recording the scene was located at the current position of the viewer.

2. The system of claim 1, wherein the hardware processor further executes the executable code to receive input from the viewer to select the first or second location of the virtual camera in the scene.

3. The system of claim 1, wherein the synthetic view is a dynamic view showing the scene as it appears from a perspective of the virtual camera dynamically moving in the scene based on the 3D model of the scene.

4. The system of claim 1, wherein the video content is one of a live event or a pre-recorded event.

5. The system of claim 1, wherein the hardware processor executes the executable code to create the 3D model of the scene of the video content in real-time.

6. The system of claim 5, wherein the hardware processor executes the executable code to construct the synthetic view in real-time.

7. The system of claim 1, wherein the hardware processor further executes the executable code to continuously adjust the synthetic view of the scene of the video content being displayed on the display while the viewer moves relative to the display based on the current position of the viewer relative to the display.

8. The system of claim 1, wherein the display shows an angularly accurate view of the video content for up to 360 degrees around the viewer based on the current position of the viewer relative to the display.

9. The system of claim 1, wherein the display includes a spherical display displaying the video content above and below the viewer up to 4π

steradians.

10. A method for use with a system having a non-transitory memory and a hardware processor, the method comprising:

receiving, using the hardware processor, a video content including a plurality of frames showing a scene from a perspective of a real camera;

creating, using the hardware processor, a three-dimensional (3D) model of the scene using the plurality of frames;

storing the 3D model of the scene in the non-transitory memory;

constructing, using the hardware processor, a synthetic view showing the scene from a first perspective of a virtual camera at a first location in the scene;

transmitting, using the hardware processor, the synthetic view of the scene for being displayed on a display located at a fixed position in a room;

tracking a position of a viewer in the room relative to the fixed position of the display in the room, using a camera or a wireless identification carried by the viewer, to determine a current position of the viewer in the room as the viewer is moving along a path from a first position in the room to a second position in the room at a distance from the first position;

dynamically adjusting the synthetic view to show the scene from a second perspective of the virtual camera at a second location in the scene, wherein the dynamically adjusted synthetic view of the scene is an angularly accurate display of the scene based on the current position of the viewer in the room relative to the fixed position of the display in the room; and

transmitting the dynamically adjusted synthetic view of the scene for being displayed on the display such that the scene appears to the viewer as if the real camera recording the scene was located at the current position of the viewer.

11. The method of claim 10, further comprising:

a receiving input from the viewer selecting the first or second location of the virtual camera in the scene.

12. The method of claim 10, wherein the synthetic view is a dynamic view showing the scene as it appears from a perspective of the virtual camera moving in the scene based on the 3D model of the scene.

13. The method of claim 10, wherein the media content is one of a live event or a pre-recorded event.

14. The method of claim 10, wherein creating the 3D model of the scene of the video content and constructing the synthetic view are performed in real-time.

15. The method of claim 10, further comprising continuously adjusting the synthetic view of the scene being displayed on the display based on the current position of the viewer relative to the display while the viewer moves relative to the display.

16. The method of claim 10, wherein the display includes a spherical display displaying an angularly accurate view of the video content around, above, and below the viewer up to 4π

steradians.