Predictive bi-adaptive streaming of real-time interactive computer graphics content

• US 9,786,027 B1
• Filed: 01/09/2017
• Issued: 10/10/2017
• Est. Priority Date: 06/16/2016
• Status: Expired due to Fees

• Current Assignee: Waygate Inc.
• Original Assignee: Waygate Inc.
• Inventors: Cooley, Benjamin, Morgantini, Carlo
• Primary Examiner(s): Wang, Yi
• Application Number:
  US15/401,938
• Time in Patent Office: 274 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: G06T 2210/36 Level of detail
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/566 Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display

Abstract

A method including playing a first interactive 3D simulation for a first 3D location of a computer graphics (CG) content clip for at least a first predetermined length of time. The method also includes, while playing the first interactive 3D simulation and within the first predetermined length of time, determining a first subset of the plurality of 3D assets to be downloaded for rendering two or more second 3D locations of the plurality of 3D locations; adapting the level of detail for one or more first 3D assets of the first subset of the plurality of 3D assets based at least in part on (a) a bandwidth for downloading the first subset of the plurality of 3D assets through the network, and (b) a processing power of the player device; and downloading the first subset of the plurality of 3D assets at the level of detail as adapted for the bandwidth and the processing power. The method additionally includes, after the first predetermined length of time, receiving a selection for a selected 3D location of the two or more second 3D locations. The method further includes playing a second interactive 3D simulation for the selected 3D location using at least a second subset of the first subset of the plurality of 3D assets, the second interactive 3D simulation being played devoid of delays for downloading and rendering the second interactive 3D simulation. Other embodiments are provided.

Claims

1. A player device comprising:

one or more processing modules in data communication with a server through a network; and

one or more non-transitory memory storage modules storing computing instructions configured to run on the one or more processing modules and perform;

playing a first interactive 3D simulation for a first 3D location of a computer graphics (CG) content clip for at least a first predetermined length of time, the CG content clip comprising a plurality of 3D locations renderable using a plurality of 3D assets each at a level of detail, and the plurality of 3D locations comprising the first 3D location;

while playing the first interactive 3D simulation and within the first predetermined length of time;

determining a first subset of the plurality of 3D assets to be downloaded for rendering two or more second 3D locations of the plurality of 3D locations, the two or more second 3D locations being selectable options for playback directly subsequent to playing the first interactive 3D simulation for the first 3D location;

adapting each of the levels of detail for one or more first 3D assets of the first subset of the plurality of 3D assets based at least in part on (a) a bandwidth for downloading the first subset of the plurality of 3D assets through the network, and (b) a processing power of the player device; and

downloading the first subset of the plurality of 3D assets at each of the levels of detail for the plurality of 3D assets, including each of the levels of detail, as adapted for the bandwidth and the processing power, for each of the one or more first 3D assets of the first subset of the after the first predetermined length of time, receiving a selection for a selected 3D location of the two or more second 3D locations; and

playing a second interactive 3D simulation for the selected 3D location using at least a second subset of the first subset of the plurality of 3D assets, the second interactive 3D simulation being played devoid of delays for downloading and rendering the second interactive 3D simulation based on each of the levels of detail for the one or more first 3D assets that was adapted for the bandwidth and the processing power during the first predetermined length of time.

2. The player device of claim 1, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets based at least in part on a current running average of the bandwidth.

3. The player device of claim 1, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

determining whether the first subset of the plurality of 3D assets can be downloaded through the network within the first predetermined length of time when each of the levels of detail for each of the first subset of the plurality of 3D assets is a highest level of detail; and

if the first subset of the plurality of 3D assets cannot be downloaded through the network within the first predetermined length of time when each of the levels of detail for each of the first subset of the plurality of 3D assets is a highest level of detail, decreasing each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets until the first subset of the plurality of 3D assets can be downloaded within the first predetermined length of time.

4. The player device of claim 1, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

using a bandwidth score for each of the levels of detail for the first subset of the plurality of 3D assets to determine whether the first subset of the plurality of 3D assets can be downloaded within the first predetermined length of time.

5. The player device of claim 1, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets based at least in part on a current running average of the processing power of the player device.

6. The player device of claim 1, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

determining whether a subset of the first subset of the plurality of 3D assets for rendering each of the two or more second 3D locations can be rendered within a predetermined framerate using each of the levels of detail, as currently set, for the subset of the first subset of the plurality of 3D assets; and

for each of the two or more second 3D locations, if the subset of the first subset of the plurality of 3D assets for rendering the second 3D location cannot be rendered within the predetermined framerate, decreasing each of the levels of detail for one or more 3D assets of the subset of the first subset of the plurality of 3D assets until the subset of the first subset of the plurality of 3D assets can be rendered within the predetermined framerate; and

the one or more first 3D assets comprise each of the one or more 3D assets of each of the subsets of the first subset of the plurality of 3D assets.

7. The player device of claim 1, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

using a processing score for each of the levels of detail for the first subset of the plurality of 3D assets to determine whether each subset of the first subset of the plurality of 3D assets for rendering each of the two or more second 3D locations can be rendered within a predetermined framerate.

8. The player device of claim 1, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

determining the one or more first 3D assets of the first subset of the plurality of 3D assets for adapting each of the levels of detail, based on priority levels of the first subset of the plurality of 3D assets.

9. The player device of claim 1, wherein:

determining the first subset of the plurality of 3D assets to be downloaded for rendering the two or more second 3D locations of the plurality of 3D locations further comprises;

determining one or more second 3D assets of the plurality of 3D assets for rendering the two or more second 3D locations that are already in a cache of the player device, such that the one or more second 3D assets are not included in the first subset of the plurality of 3D assets to be downloaded for rendering the two or more second 3D locations.

10. The player device of claim 1, wherein the computing instructions are further configured to perform:

after receiving the selection for the selected 3D location of the two or more second 3D locations, evicting from a cache of the player device a third subset of the first subset of the plurality of 3D assets, the third subset of the first subset of the plurality of 3D assets being devoid of any of the second subset of the first subset of the plurality of 3D assets.

11. The player device of claim 1, wherein:

determining the first subset of the plurality of 3D assets to be downloaded for rendering the two or more second 3D locations of the plurality of 3D locations further comprises;

using a location graph comprising a plurality of locations nodes each representing a different one of the plurality of 3D locations to determine the two or more second 3D locations of the plurality of 3D locations.

12. The player device of claim 11, wherein:

the location graph is a directed graph.

13. The player device of claim 11, wherein:

the location graph is a cyclic graph.

14. The player device of claim 1, wherein:

the level of detail for a first one of the plurality of 3D assets is different than a level of detail for a second one of the plurality of 3D assets.

15. A method being implemented on a player device via execution of computer instructions configured to run at one or more processing modules and configured to be stored at one or more non-transitory memory storage modules, the method comprising:

playing a first interactive 3D simulation for a first 3D location of a computer graphics (CG) content clip for at least a first predetermined length of time, the CG content clip comprising a plurality of 3D locations renderable using a plurality of 3D assets each at a level of detail, and the plurality of 3D locations comprising the first 3D location;

while playing the first interactive 3D simulation and within the first predetermined length of time;

determining a first subset of the plurality of 3D assets to be downloaded for rendering two or more second 3D locations of the plurality of 3D locations, the two or more second 3D locations being selectable options for playback directly subsequent to playing the first interactive 3D simulation for the first 3D location;

adapting each of the levels of detail for one or more first 3D assets of the first subset of the plurality of 3D assets based at least in part on (a) a bandwidth for downloading the first subset of the plurality of 3D assets from a server to the player device through a network, and (b) a processing power of the player device; and

downloading the first subset of the plurality of 3D assets at each of the levels of detail for the plurality of 3D assets, including each of the levels of detail, as adapted for the bandwidth and the processing power, for each of the one or more first 3D assets of the first subset of the plurality of 3D assets;

after the first predetermined length of time, receiving a selection for a selected 3D location of the two or more second 3D locations; and

playing a second interactive 3D simulation for the selected 3D location using at least a second subset of the first subset of the plurality of 3D assets, the second interactive 3D simulation being played devoid of delays for downloading and rendering the second interactive 3D simulation based on each of the levels of detail for the one or more first 3D assets that was adapted for the bandwidth and the processing power during the first predetermined length of time.

16. The method of claim 15, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets based at least in part on a current running average of the bandwidth.

17. The method of claim 15, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

determining whether the first subset of the plurality of 3D assets can be downloaded through the network within the first predetermined length of time when each of the levels of detail for the first subset of the plurality of 3D assets is a highest level of detail; and

if the first subset of the plurality of 3D assets cannot be downloaded through the network within the first predetermined length of time when each of the levels of detail for the first subset of the plurality of 3D assets is a highest level of detail, decreasing each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets until the first subset of the plurality of 3D assets can be downloaded within the first predetermined length of time.

18. The method of claim 15, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

using a bandwidth score for each of the levels of detail for the first subset of the plurality of 3D assets to determine whether the first subset of the plurality of 3D assets can be downloaded within the first predetermined length of time.

19. The method of claim 15, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets based at least in part on a current running average of the processing power of the player device.

20. The method of claim 15, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

determining whether a subset of the first subset of the plurality of 3D assets for rendering each of the two or more second 3D locations can be rendered within a predetermined framerate using each of the levels of detail, as currently set, for the subset of the first subset of the plurality of 3D assets; and

for each of the two or more second 3D locations, if the subset of the first subset of the plurality of 3D assets for rendering the second 3D location cannot be rendered within the predetermined framerate, decreasing each of the levels of detail for one or more 3D assets of the subset of the first subset of the plurality of 3D assets until the subset of the first subset of the plurality of 3D assets can be rendered within the predetermined framerate; and

the one or more first 3D assets comprise each of the one or more 3D assets of each of the subsets of the first subset of the plurality of 3D assets.

21. The method of claim 15, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

using a processing score for each of the levels of detail for the first subset of the plurality of 3D assets to determine whether each subset of the first subset of the plurality of 3D assets for rendering each of the two or more second 3D locations can be rendered within a predetermined framerate.

22. The method of claim 15, wherein:

adapting each of the levels of detail for the one or more first 3D assets of the first subset of the plurality of 3D assets further comprises;

determining the one or more first 3D assets of the first subset of the plurality of 3D assets for adapting each of the levels of detail, based on priority levels of the first subset of the plurality of 3D assets.

23. The method of claim 15, wherein:

determining the first subset of the plurality of 3D assets to be downloaded for rendering the two or more second 3D locations of the plurality of 3D locations further comprises;

determining one or more second 3D assets of the plurality of assets for rendering the two or more second 3D locations that are already in a cache of the player device, such that the one or more second 3D assets are not included in the first subset of the plurality of 3D assets to be downloaded for rendering the two or more second 3D locations.

24. The method of claim 15, further comprising:

after receiving the selection for the selected 3D location of the two or more second 3D locations, evicting from a cache of the player device a third subset of the first subset of the plurality of 3D assets, the third subset of the first subset of the plurality of 3D assets being devoid of any of the second subset of the first subset of the plurality of 3D assets.

25. The method of claim 15, wherein:

determining the first subset of the plurality of 3D assets to be downloaded for rendering the two or more second 3D locations of the plurality of 3D locations further comprises;

using a location graph comprising a plurality of locations nodes each representing a different one of the plurality of 3D locations to determine the two or more second 3D locations of the plurality of 3D locations.

26. The method of claim 25, wherein:

the location graph is a directed graph.

27. The method of claim 25, wherein:

the location graph is a cyclic graph.

28. The method of claim 15, wherein:

the level of detail for a first one of the plurality of 3D assets is different than a level of detail for a second one of the plurality of 3D assets.