Methods and systems for rendering frames of a virtual scene from different vantage points based on a virtual entity description frame of the virtual scene

• US 10,311,630 B2
• Filed: 05/31/2017
• Issued: 06/04/2019
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Gervasio, William Patrick, Castaneda, Oliver S., Breitenfeld, Denny
• Primary Examiner(s): Hajnik, Daniel F
• Application Number:
  US15/610,572
• Publication Number:
  US 20180350133A1
• Time in Patent Office: 734 Days
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

An exemplary virtual scene capture system (“system”) maintains data representative of a plurality of virtual entities included within a virtual 3D space of a virtual scene. The plurality of virtual entities includes a virtual object and a plurality of virtual vantage points into the virtual 3D space. Based on the maintained data, the system generates a virtual entity description frame representative of a state of at least one of the virtual entities at a particular point in a temporal sequence. The system then provides the virtual entity description frame to a plurality of server-side 3D rendering engines corresponding to different vantage points in the plurality of virtual vantage points and configured to render, based on the virtual entity description frame, surface data frames representative of color and depth data of surfaces of the virtual object visible from the different virtual vantage points at the particular point in the temporal sequence.

Claims

1. A method comprising:

maintaining, by a virtual scene capture system, data representative of a plurality of virtual entities included within a virtual three-dimensional (3D) space of a virtual scene, wherein

the plurality of virtual entities includes a virtual object and a plurality of virtual vantage points into the virtual 3D space,

the plurality of virtual vantage points includes a first virtual vantage point and a second virtual vantage point different from the first virtual vantage point, and

the maintained data representative of the plurality of virtual entities includes a link to color and depth data representative of the virtual object stored in an asset storage system associated with the virtual scene capture system;

generating, by the virtual scene capture system based on the maintained data representative of the plurality of virtual entities, a virtual entity description frame representative of a state of at least one virtual entity in the plurality of virtual entities at a particular point in a temporal sequence, the virtual entity description frame including the link to the color and depth data representative of the virtual object stored in the asset storage system; and

providing, by the virtual scene capture system, the virtual entity description frame to a plurality of server-side 3D rendering engines including a first 3D rendering engine associated with the first virtual vantage point and a second 3D rendering engine associated with the second virtual vantage point, the first and second 3D rendering engines configured to render first and second surface data frames, respectively, by performing operations including;

receiving the virtual entity description frame from the virtual scene capture system, and

accessing, from the asset storage system using the link included within the virtual entity description frame, the color and depth data representative of the virtual object stored in the asset storage system.

2. The method of claim 1, further comprising:

receiving, by the virtual scene capture system from a virtual scene control system, a command to modify the maintained data representative of the plurality of virtual entities; and

modifying, by the virtual scene capture system in response to the receiving of the command, the maintained data representative of the plurality of virtual entities in accordance with the command.

3. The method of claim 2, wherein the modifying of the maintained data representative of the plurality of virtual entities in accordance with the command includes at least one of:

adding an additional virtual object to the plurality of virtual entities included within the virtual 3D space of the virtual scene;

replacing the virtual object included within the plurality of virtual entities included within the virtual 3D space of the virtual scene with the additional virtual object;

removing the virtual object from the plurality of virtual entities included within the virtual 3D space of the virtual scene; and

modifying at least one property of the virtual object included in the plurality of virtual entities included within the virtual 3D space of the virtual scene.

4. The method of claim 2, wherein the modifying of the maintained data representative of the plurality of virtual entities in accordance with the command includes at least one of:

adding an additional virtual vantage point to the plurality of virtual entities included within the virtual 3D space of the virtual scene;

modifying at least one of the plurality of virtual vantage points included within the plurality of virtual entities included within the virtual 3D space of the virtual scene; and

removing the at least one of the plurality of virtual vantage points from the plurality of virtual entities included within the virtual 3D space of the virtual scene.

5. The method of claim 1, wherein the maintaining of the data representative of the plurality of virtual entities includes applying, to the virtual object included within the virtual 3D space of the virtual scene, at least one of a physics-based object behavior and an artificial intelligence-based (AI-based) object behavior.

6. The method of claim 1, wherein:

the virtual entity description frame representative of the state of the at least one virtual entity in the plurality of virtual entities at the particular point in the temporal sequence is a key description frame representing, at the particular point in the temporal sequence, a state of all the virtual entities in the plurality of virtual entities; and

the key description frame is associated with a sequence number indicative of a position of the key description frame in the temporal sequence with respect to other key description frames in the temporal sequence.

7. The method of claim 1, wherein:

the virtual entity description frame representative of the state of the at least one virtual entity in the plurality of virtual entities at the particular point in the temporal sequence is an update description frame representing, at the particular point in the temporal sequence, a state of only those virtual entities in the plurality of virtual entities that have changed since a previous key description frame was generated representing, at a previous point in the temporal sequence, a state of all the virtual entities in the plurality of virtual entities; and

the update description frame is associated with a sequence number indicative of a position of the update description frame in the temporal sequence with respect to the previous key description frame and with respect to other update description frames in the temporal sequence.

8. The method of claim 1, wherein:

the first 3D rendering engine to which the virtual entity description frame is provided is configured to render the first surface data frame based on the virtual entity description frame, the first surface data frame representative of color and depth data of surfaces of the virtual object visible from the first virtual vantage point at the particular point in the temporal sequence; and

the second 3D rendering engine to which the virtual entity description frame is provided is configured to render the second surface data frame based on the virtual entity description frame, the second surface data frame representative of color and depth data of surfaces of the virtual object visible from the second virtual vantage point at the particular point in the temporal sequence.

9. The method of claim 8, wherein:

the first surface data frame is included within a first sequence of surface data frames representative of color and depth data of the surfaces of the virtual object visible from the first virtual vantage point during the temporal sequence;

the second surface data frame is included within a second sequence of surface data frames representative of color and depth data of the surfaces of the virtual object visible from the second virtual vantage point during the temporal sequence; and

the method further comprises

generating, by a video data packaging system communicatively coupled to the plurality of server-side 3D rendering engines and based on the first and second sequences of surface data frames, a transport stream that includes a color video data stream and a depth video data stream for each of the virtual vantage points in the plurality of virtual vantage points, and

providing, by the video data packaging system, the transport stream for streaming to a client-side media player device associated with a user, the client-side media player device configured to generate, based on the color video data stream and the depth video data stream for each of the virtual vantage points included within the transport stream, a 3D representation of the virtual 3D space of the virtual scene to be experienced by the user from a dynamically selectable virtual vantage point selected by the user and corresponding to an arbitrary virtual location within the virtual 3D space.

10. The method of claim 1, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

11. A virtual scene capture system comprising:

an asset storage system storing color and depth data representative of a virtual object;

a plurality of server-side three-dimensional (3D) rendering engines including a first 3D rendering engine and a second 3D rendering engine each communicatively coupled to the asset storage system; and

a virtual entity state tracking system communicatively coupled to the asset storage system and the first and second 3D rendering engines, the virtual entity state tracking system configured to

maintain data representative of the virtual object included within a virtual 3D space of a virtual scene and of a plurality of virtual vantage points into the virtual 3D space, the plurality of virtual vantage points including a first virtual vantage point and a second virtual vantage point different from the first virtual vantage point, and the maintained data representative of the virtual object including a link to the color and depth data representative of the virtual object stored in the asset storage system,

generate, based on the maintained data representative of the plurality of virtual entities, a virtual entity description frame representative of a state of at least one virtual entity in the plurality of virtual entities at a particular point in a temporal sequence, and

provide the virtual entity description frame to the plurality of server-side 3D rendering engines including the first and second 3D rendering engines;

wherein the first 3D rendering engine is associated with the first virtual vantage point and is configured to render, based on the virtual entity description frame and the color and depth data representative of the virtual object accessed from the asset storage system by way of the link, a first surface data frame representative of color and depth data of surfaces of the virtual object visible from the first virtual vantage point at the particular point in the temporal sequence; and

wherein the second 3D rendering engine is associated with the second virtual vantage point and is configured to render, based on the virtual entity description frame and the color and depth data representative of the virtual object accessed from the asset storage system by way of the link, a second surface data frame representative of color and depth data of surfaces of the virtual object visible from the second virtual vantage point at the particular point in the temporal sequence.

12. A virtual scene capture system comprising:

at least one physical computing device that

maintains data representative of a plurality of virtual entities included within a virtual three-dimensional (3D) space of a virtual scene, wherein

the plurality of virtual entities includes a virtual object and a plurality of virtual vantage points into the virtual 3D space,

the plurality of virtual vantage points includes a first virtual vantage point and a second virtual vantage point different from the first virtual vantage point, and

the maintained data representative of the plurality of virtual entities includes a link to color and depth data representative of the virtual object stored in an asset storage system associated with the virtual scene capture system;

generates, based on the maintained data representative of the plurality of virtual entities, a virtual entity description frame representative of a state of at least one virtual entity in the plurality of virtual entities at a particular point in a temporal sequence, the virtual entity description frame including the link to the color and depth data representative of the virtual object stored in the asset storage system; and

provides the virtual entity description frame to a plurality of server-side 3D rendering engines including a first 3D rendering engine associated with the first virtual vantage point and a second 3D rendering engine associated with the second virtual vantage point, the first and second 3D rendering engines configured to render first and second surface data frames, respectively, by performing operations including;

receiving the virtual entity description frame from the virtual scene capture system, and

accessing, from the asset storage system using the link included within the virtual entity description frame, the color and depth data representative of the virtual object stored in the asset storage system.

13. The system of claim 12, wherein the at least one physical computing device further:

receives, from a virtual scene control system, a command to modify the maintained data representative of the plurality of virtual entities; and

modifies, in response to the receipt of the command, the maintained data representative of the plurality of virtual entities in accordance with the command.

14. The system of claim 13, wherein the modification of the maintained data representative of the plurality of virtual entities in accordance with the command is performed by at least one of:

adding an additional virtual object to the plurality of virtual entities included within the virtual 3D space of the virtual scene;

replacing the virtual object included within the plurality of virtual entities included within the virtual 3D space of the virtual scene with the additional virtual object;

removing the virtual object from the plurality of virtual entities included within the virtual 3D space of the virtual scene; and

modifying at least one property of the virtual object included in the plurality of virtual entities included within the virtual 3D space of the virtual scene.

15. The system of claim 13, wherein the modification of the maintained data representative of the plurality of virtual entities in accordance with the command is performed by at least one of:

adding an additional virtual vantage point to the plurality of virtual entities included within the virtual 3D space of the virtual scene;

modifying at least one of the plurality of virtual vantage points included within the plurality of virtual entities included within the virtual 3D space of the virtual scene; and

removing the at least one of the plurality of virtual vantage points from the plurality of virtual entities included within the virtual 3D space of the virtual scene.

16. The system of claim 12, wherein the at least one physical computing device maintains the data representative of the plurality of virtual entities by applying, to the virtual object included within the virtual 3D space of the virtual scene, at least one of a physics-based object behavior and an artificial intelligence-based (AI-based) object behavior.

17. The system of claim 12, wherein:

the virtual entity description frame representative of the state of the at least one virtual entity in the plurality of virtual entities at the particular point in the temporal sequence is a key description frame representing, at the particular point in the temporal sequence, a state of all the virtual entities in the plurality of virtual entities; and

the key description frame is associated with a sequence number indicative of a position of the key description frame in the temporal sequence with respect to other key description frames in the temporal sequence.

18. The system of claim 12, wherein:

the virtual entity description frame representative of the state of the at least one virtual entity in the plurality of virtual entities at the particular point in the temporal sequence is an update description frame representing, at the particular point in the temporal sequence, a state of only those virtual entities in the plurality of virtual entities that have changed since a previous key description frame was generated representing, at a previous point in the temporal sequence, a state of all the virtual entities in the plurality of virtual entities; and

the update description frame is associated with a sequence number indicative of a position of the update description frame in the temporal sequence with respect to the previous key description frame and with respect to other update description frames in the temporal sequence.

19. The system of claim 12, wherein:

the first 3D rendering engine to which the virtual entity description frame is provided is configured to render the first surface data frame based on the virtual entity description frame, the first surface data frame representative of color and depth data of surfaces of the virtual object visible from the first virtual vantage point at the particular point in the temporal sequence; and

the second 3D rendering engine to which the virtual entity description frame is provided is configured to render the second surface data frame based on the virtual entity description frame, the second surface data frame representative of color and depth data of surfaces of the virtual object visible from the second virtual vantage point at the particular point in the temporal sequence.

20. The system of claim 19, wherein:

the first surface data frame is included within a first sequence of surface data frames representative of color and depth data of the surfaces of the virtual object visible from the first virtual vantage point during the temporal sequence;

the second surface data frame is included within a second sequence of surface data frames representative of color and depth data of the surfaces of the virtual object visible from the second virtual vantage point during the temporal sequence; and

the plurality of server-side 3D rendering engines is communicatively coupled to a video data packaging system that

generates, based on the first and second sequences of surface data frames, a transport stream that includes a color video data stream and a depth video data stream for each of the virtual vantage points in the plurality of virtual vantage points, and

provides the transport stream for streaming to a client-side media player device associated with a user, the client-side media player device configured to generate, based on the color video data stream and the depth video data stream for each of the virtual vantage points included within the transport stream, a 3D representation of the virtual 3D space of the virtual scene to be experienced by the user from a dynamically selectable virtual vantage point selected by the user and corresponding to an arbitrary virtual location within the virtual 3D space.