Network-controlled 3D video capture

• US 10,594,917 B2
• Filed: 10/30/2017
• Issued: 03/17/2020
• Est. Priority Date: 10/30/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Fowers, Spencer G
• Primary Examiner(s): Hansell, Jr., Richard A
• Application Number:
  US15/797,909
• Publication Number:
  US 20190132504A1
• Time in Patent Office: 869 Days
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: H04L 43/0811 by checking connectivity
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Techniques are described for controlling the process of capturing three-dimensional (3D) video content. For example a controller can provide centralized control over the various components that participate in the capture, and processing, of the 3D video content. For example, the controller can establish connections with a number of components (e.g., running on other computing devices). The controller can receive state update messages from the components (e.g., comprising state change information, network address information, etc.). The controller can also broadcast messages to the components. For example, the controller can broadcast system state messages to the components where the system state messages comprise current state information of the components. The controller can also broadcast other types of messages, such as start messages that instruct the components to enter a start state.

Claims

1. A computing device comprising:

a processing unit;

memory; and

a network connection;

the computing device programmed, via computer-executable instructions, to implement a controller that performs operations for controlling three-dimensional (3D) video capture, the operations comprising;

receiving, via the network connection, connections from a plurality of components, wherein the plurality of components process information for generating a three-dimensional holographic video output from a captured real-world video input, wherein the plurality of components run on a plurality of other computing devices, and wherein the plurality of other computing devices are separate from the computing device and communicate with the computing device via the network connection, and wherein the plurality of components comprise;

a plurality of depth generators;

an encoder, wherein the encoder transmits encoded video images from a plurality of video cameras to a distributor; and

the distributor, wherein the distributor distributes image data from the encoded video images to the plurality of depth generators;

receiving, via the network connection, state update messages from the plurality of components, wherein the state update messages comprise indications that the plurality of components are in a ready state; and

after receiving the state update messages from the plurality of components, broadcasting, via the network connection, a start command to the plurality of components;

wherein the plurality of components start processing information for generating the three-dimensional holographic video output from the captured real-world video input based, at least in part, on the start command received from the computing device.

2. The computing device of claim 1 wherein the distributor component operates in a remote network from the encoder component.

3. The computing device of claim 1 wherein the distributor component establishes direct network connections with the plurality of depth generator components before sending the state update message reporting the ready state to the computing device.

4. The computing device of claim 1 wherein the encoder component establishes a direct network connection with the distributor component before sending the state update message reporting the ready state to the computing device.

5. The computing device of claim 1 the operations further comprising:

when a state update message is received from a component of the plurality of components, broadcasting current system state to the plurality of components, wherein the current system state comprises state information for the plurality of components.

6. The computing device of claim 1 wherein the plurality of components are connected via one or more local area networks or wide area networks, and wherein the plurality of components run on a corresponding plurality of different computing devices.

7. The computing device of claim 1 wherein the plurality of components comprise:

a fusion component; and

a renderer component.

8. A method, implemented by a computing device comprising a processing unit and memory, the method comprising:

by a controller running on the computing device;

receiving, via a network connection, connections from a plurality of components, wherein the plurality of components process information for generating a three-dimensional holographic video output from a captured real-world video input, wherein the plurality of components run on a plurality of other computing devices, wherein the plurality of other computing devices are separate from the computing device and communicate with the computing device via the network connection, and wherein the plurality of components comprise;

a plurality of depth generators;

an encoder, wherein the encoder transmits encoded video images from a plurality of video cameras to a distributor; and

the distributor, wherein the distributor distributes image data from the encoded video images to the plurality of depth generators;

broadcasting, via the network connection, system state messages to the plurality of components, wherein the system state messages comprise current state information for the plurality of components, and wherein the system state messages are broadcast in response to state changes among the plurality of components;

receiving, via the network connection, state update messages from the plurality of components, wherein the state update messages comprise indications that the plurality of components are in a ready state;

after receiving the state update messages from the plurality of components, broadcasting, via the network connection, a start command to the plurality of components;

wherein the plurality of components start processing information for generating the three-dimensional holographic video output from the captured real-world video input according to a startup order that is based, at least in part, on the broadcast system state messages.

9. The method of claim 8 wherein the broadcast system state messages comprise network connection information for the plurality of components, and wherein the plurality of components use the network connection information to establish direct network connections among the plurality of components.

10. The method of claim 8 wherein the real-world video input is captured by the plurality of video cameras that capture video images and transmit the captured video images to the encoder.

11. The method of claim 10 wherein the distributor operates in a remote network from the encoder.

12. The method of claim 10 wherein the encoder performs operations comprising:

receiving the video images from the plurality of video cameras;

encoding the video images using a video codec to generate the encoded video images; and

transmitting the encoded video images to the distributor via a wireless network connection;

wherein the distributor is in a remote geographical environment from the encoder.

13. The method of claim 8 wherein the distributor establishes a direct network connection with the plurality of depth generators before sending the state update message reporting the ready state, and wherein the encoder establishes a direct network connection with the distributor before sending the state update message reporting the ready state.

14. The method of claim 8 wherein the plurality of depth generators start processing in response to receiving the start command, wherein the distributor starts processing in response to receiving a broadcast system state message indicating a state change of the depth generators to a running state, and wherein the encoder starts processing in response to receiving a broadcast system state message indicating a state change of the distributor to a running state.

15. A method, implemented by a plurality of computing devices comprising processing units and memory, the method comprising:

by a controller running on a first computing device;

receiving, via a network connection, connections from a plurality of components, wherein the plurality of components process information for generating a three-dimensional holographic video output from a captured real-world video input, and wherein the plurality of components comprise;

an encoder, wherein the encoder transmits encoded video images from a plurality of video cameras to a distributor;

the distributor, wherein the distributor distributes image data from the encoded video images to a plurality of depth generators; and

the plurality of depth generators;

when a state update message is received from a component, broadcasting a system state message to the plurality of components, wherein the system state message comprises current state information of the plurality of components;

by each of the plurality of depth generators, the distributor, and the encoder, running on a plurality of other computing devices;

starting up in an order based at least in part on broadcast messages received from the controller;

wherein the plurality of other computing devices are separate from the first computing device and communicate with the first computing device via the network connection.

16. The method of claim 15 wherein starting up in the order comprises:

by the plurality of depth generators;

connecting to the controller at a known network address and reporting a ready state;

receiving a broadcast start command from the controller; and

in response to the start command, transitioning from the ready state to a running state.

17. The method of claim 15 wherein starting up in the order comprises:

by the distributor;

connecting to the controller at a known network address;

in response to a first broadcast system state message from the controller indicating network addresses for the plurality of depth generators, establishing direct network connections to the plurality of depth generators; and

in response to a second broadcast system state message from the controller indicating that the depth generators are in a running state, transitioning from a ready state to a running state.

18. The method of claim 15 wherein starting up in the order comprises:

by the encoder component;

connecting to the controller at a known network address;

in response to a first broadcast system state message from the controller indicating a network address for the distributor, establishing a direct network connection to the distributor; and

in response to a second broadcast system state message from the controller indicating that the distributor is in a running state, transitioning from a ready state to a running state.