Collaborative multi-modal mixed-reality system and methods leveraging reconfigurable tangible user interfaces for the production of immersive, cinematic, and interactive content

• US 10,565,802 B2
• Filed: 02/20/2018
• Issued: 02/18/2020
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Grossmann, Benjamin, Legato, Robert
• Primary Examiner(s): Mushambo, Martin
• Application Number:
  US15/900,694
• Publication Number:
  US 20190066387A1
• Time in Patent Office: 728 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

The disclosure is directed to collaborative multi-modal mixed-reality systems and methods leveraging reconfigurable tangible user interfaces for the production of immersive, cinematic, and interactive content. As described herein, content may be created as either a mixture of live action media and computer generated media, combined together in context; or content made up entirely from computer-generated media alone. The technology described herein may leverage physical and software-based user interfaces that merge and unite the physical world and the virtual world, which allow users to produce new forms of content with a user experience and user interactions that emulate traditional processes.

Claims

1. A method, comprising:

initializing a virtual production application instance on a virtual reality (VR) client device;

loading scene files and dependencies at the virtual production application;

based on the loaded scene files and dependencies, rendering a virtual world for display by the VR client device; and

the VR client device making changes to the scene files using a virtual media production tool rendered in the virtual world, wherein the virtual media production tool corresponds to a physical media production tool communicatively coupled to the VR client device.

2. The method of claim 1, wherein the VR client device comprises a VR head-mounted display (HMD), wherein the virtual world is displayed on the VR HMD, and wherein the VR client device makes changes to the scene files using a controller communicatively coupled to the VR HMD.

3. The method of claim 2, further comprising:

receiving data corresponding to user input actuating a control coupled to the physical media production tool; and

registering the data with the virtual media production tool rendered in the virtual world.

4. The method of claim 3, wherein the controller is a handheld controller, and wherein receiving data corresponding to user input actuating the control coupled to the physical media production tool and registering the data with the virtual world comprises:

mounting the handheld controller on a mount coupled to the physical media production tool;

tracking movement of the physical media production tool using a motion sensor of the mounted handheld controller; and

registering the tracked movement with an object in the virtual world.

5. The method of claim 2, further comprising:

displaying a user interface in the rendered virtual world for selecting a plurality of virtual media production tools;

receiving data corresponding to user input at the controller of the VR HMD selecting the virtual media production tool; and

instantiating and rendering the virtual media production tool.

6. The method of claim 5, further comprising:

receiving data corresponding to user input actuating a virtual control for changing a state of the virtual media production tool; and

making a corresponding change to the physical media production tool.

7. The method of claim 1, further comprising:

receiving data corresponding to user input actuating a control coupled to the physical media production tool; and

registering the data with the virtual media production tool rendered in the virtual world.

8. The method of claim 7, wherein receiving data corresponding to user input actuating the control coupled to the physical media production tool and registering the data with the virtual world comprises:

tracking movement of the physical media production tool using a motion sensor mounted on the physical media production tool;

encoding the tracked movement; and

transmitting the encoded tracked movement to the VR device for registration with the virtual media production tool of the virtual world.

9. The method of claim 7, wherein the physical media production tool comprises:

a video camera, a camera dolly, a camera crane, or a tripod, and wherein the virtual media production tool comprises;

a corresponding virtual video camera, virtual camera dolly, virtual camera crane, or virtual tripod.

10. The method of claim 9, wherein the physical media production tool comprises the camera dolly, wherein receiving data corresponding to user input actuating the control coupled to the physical media production tool comprises moving the camera dolly along a dolly track, and wherein registering the data with the virtual world comprises registering the movement of the camera dolly to a virtual world spline attachment point.

11. The method of claim 7, wherein registering the data with the virtual world comprises:

passing the data corresponding to user input actuating the control coupled to the physical media production tool as an input to a node graph; and

evaluating the node graph using the input.

12. The method of claim 1, further comprising:

transmitting, via the client device, the changes to the scene files to a server; and

broadcasting, via the server, the changes to the scene files to a plurality of other VR client devices to synchronize a virtual world rendered by the plurality of other VR client devices with the virtual world rendered by the client device.

13. A system, comprising:

a virtual reality (VR) client device;

one or more non-transitory computer-readable mediums having instructions stored thereon that, when executed by one or more processors, causes the system to;

initialize a virtual production application instance;

load scene files and dependencies at the virtual production application;

based on the loaded scene files and dependencies, render a virtual world for display by the VR client device; and

the VR client device making changes to the scene files using a virtual media production tool rendered in the virtual world, wherein the virtual media production tool corresponds to a physical media production tool communicatively coupled to the VR client device.

14. The system of claim 13, wherein the VR client device comprises a VR head-mounted display (HMD), wherein the virtual world is displayed on the VR HMD, the system further comprising:

the physical media production tool communicatively coupled to the VR client device, wherein the instructions when executed by the one or more processors, further causes the system to;

receive data corresponding to user input actuating a control coupled to the physical media production tool; and

register the data with the virtual media production tool rendered in the virtual world.

15. The system of claim 14, wherein the VR HMD is communicatively coupled to a handheld controller, and wherein receiving data corresponding to user input actuating the control coupled to the physical media production tool and registering the data with the virtual media production tool rendered in the virtual world comprises:

mounting the handheld controller on a mount of the physical media production tool;

tracking movement of the physical media production tool using a motion sensor of the mounted handheld controller; and

registering the tracked movement with an object in the virtual world.

16. The system of claim 14, wherein the physical media production tool comprises:

a video camera, a camera dolly, a camera crane, or a tripod, and wherein the virtual media production tool comprises;

a corresponding virtual video camera, virtual camera dolly, virtual camera crane, or virtual tripod.

17. The system of claim 16, wherein the physical media production tool comprises the camera dolly, wherein receiving data corresponding to user input actuating the control coupled to the physical media production tool comprises moving the camera dolly along a dolly track, and wherein registering the data with the virtual world comprises registering the movement of the camera dolly to a virtual world spline attachment point.

18. The system of claim 14, wherein registering the data with the virtual media production tool rendered in the virtual world comprises:

passing the data corresponding to user input actuating the control coupled to the physical media production tool as an input to a node graph; and

evaluating the node graph using the input.

19. The system of claim 13, wherein the instructions when executed by the one or more processors, further causes the system to:

display a user interface in the rendered virtual world for selecting a plurality of virtual media production tools;

receive data corresponding to user input at a controller of the VR client device selecting the virtual media production tool; and

instantiate and render the virtual media production tool.

20. The system of claim 19, wherein the instructions when executed by the one or more processors, further causes the system to:

receive data corresponding to user input actuating a virtual control for changing a state of the virtual media production tool; and

make a corresponding change to the physical media production tool communicatively coupled to the VR client device.

21. A method, comprising:

consolidating scene files and dependencies at a master media production device;

transmitting the consolidated scene files and dependencies from the master media production device to each of a plurality of virtual reality (VR) client devices;

transmitting the consolidated scene files and dependencies from the master media production device to a server;

loading, at each of the plurality of VR client devices, the consolidated scene files and dependencies to render a virtual world, wherein a view of the rendered virtual world is displayed by each of the plurality of VR client devices; and

synchronizing, via the server, across the plurality of VR client devices, any changes made to the consolidated scene files and dependencies by one of the plurality of VR client devices.

22. The method of claim 21, wherein the changes made to the consolidated scene files and dependencies are synchronized in real-time by the server, wherein the view of the rendered virtual world is displayed by a head-mounted display (HMD) of each of the plurality of VR client devices.

23. The method of claim 22, further comprising:

transmitting an instruction from the master media production device to each of the plurality of VR client devices to begin recording changes to the consolidated scene files and dependencies.