SPATIALIZED RENDERING OF REAL-TIME VIDEO DATA TO 3D SPACE

• US 20180342267A1
• Filed: 05/26/2017
• Published: 11/29/2018
• Est. Priority Date: 05/26/2017
• Status: Active Grant

• Current Assignee: Immersive Licensing, Inc.
• Original Assignee: Digital Domain, Inc.
• Inventors: Popa, Alexander, Nostrand, S. Lance Van
• Granted Patent: US 10,796,723 B2
• Time in Patent Office: Days
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/9305 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
• CPC Class Codes: H04N 9/8715 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

A 360 video is presented in a three dimensional (3D) environment. Rather than simply stacking graphics in two dimensions, graphics are placed using both 3D models and textures. The 3D models may be altered so that the texture is aligned in three dimensions into the 360 video space. An instance of a 3D model combined with a key and fill texture form a group. The group has a 3D orientation and placement so that the group as aligned into the 360 degree video space may not be visible from all user look directions. The inserted groups, including live video as well as static graphics, may be projected into either mono or stereo views to give the viewer a sense of space, depth, and orientation.

Claims

1. A video processing system comprising:

a video capture device that receives a video fill and a video key, the video capture device combining the video fill and video key into a video texture;

a model mapper that selects a first three dimensional (3D) model from a plurality of 3D models for mapping into a 360 degree video space;

a compositor that places the video texture over an instance of the first 3D model to generate a first group;

a virtual reality workspace editor that orients the first group into the 360 degree video space responsive to an instruction; and

an output that transmits an output signal including the first group mapped to the 360 degree video space.

2. The video processing system of claim 1, further comprising:

a memory that stores a plurality of groups including the first group; and

an group selector that receives a selection of one or more of the plurality of groups for inclusion into the output signal.

3. The video processing system of claim 1, further comprising a model designer that generates the first 3D model.

4. The video processing system of claim 1, wherein the output signal includes a 3D key component and a 3D fill component, each of the 3D key component and the 3D fill component mapped into the 360 degree video space.

5. The video processing system of claim 1, further comprising a real-time video input that receives a live video signal.

6. The video processing system of claim 5, wherein the live video signal is mapped into another instance of one of the plurality of 3D models to generate a second group.

7. The video processing system of claim 1, wherein the output signal includes the first group having an azimuth and elevation for orientation in the 360 degree video space.

8. The video processing system of claim 7, further comprising a user interface that interacts with the virtual reality workspace editor for orienting the first group within the 360 degree video space.

9. A method of processing video in a 360 degree video space, the method comprising:

receiving a video fill and a video key;

combining the video fill and the video key into a video texture;

selecting a 3D model from a plurality of 3D models;

placing the video texture into an instance of the 3D model to create a first group;

orienting the first group in the 360 degree video space; and

generating an output signal with the first group mapped into the 360 degree video space.

10. The method of claim 9, further comprising:

storing a plurality of groups including the first group; and

receiving a selection of one or more of the plurality of groups for inclusion into the output signal.

11. The method of claim 9, further comprising generating the 3D model via a model designer.

12. The method of claim 9, wherein generating the output signal with the first group comprises mapping a key component and a fill component of the first group into the 360 degree video space.

13. The method of claim 9, further comprising receiving a live video signal via a real time video input.

14. The method of claim 13, further comprising, generating a second group by mapping the live video signal into another instance of one of the plurality of 3D models.

15. The method of claim 9, further comprising orienting the first group by azimuth and elevation into the 360 degree video space.

16. The method of claim 15, wherein orienting the first group in the 360 degree video space comprises:

presenting a representation of the 360 degree video space via a user interface; and

receiving, via the user interface, an instruction for orienting the first group within the 360 degree video space.

17. A video processing system comprising:

a compositor that places a video texture over an instance of a 3D model to generate a first group, the first group including a VR mapped key component and a VR mapped fill component;

a virtual reality workspace editor that orients the 3D model into a 360 degree video space responsive to an instruction; and

an output that transmits an output signal that includes the first group mapped to the 360 degree video space.

18. The video processing system of claim 17, further comprising a model mapper that maps the video texture into a 3D model, the model mapper including a selector for picking the 3D model from a plurality of 3D models.

19. The video processing system of claim 17, further comprising a video capture unit that receives a video fill and a video key and combines the video fill and the video key to generate the video texture.

20. The video processing system of claim 17, further comprising a user interface that presents a representation of the 360 degree video space and receives an instruction for placing multiple copies of first group into the 360 degree video space to ensure viewing at every look angle in the 360 degree video space.