Methods and systems for virtual reality streaming and replay of computer video games

• US 9,573,062 B1
• Filed: 10/13/2016
• Issued: 02/21/2017
• Est. Priority Date: 12/06/2015
• Status: Active Grant

• Current Assignee: Sliver VR Technologies, Inc.
• Original Assignee: Silver VR Technologies, Inc.
• Inventors: Long, Jieyi, Liu, Mitchell C.
• Primary Examiner(s): Harvey, David
• Application Number:
  US15/292,350
• Time in Patent Office: 131 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/497 Partially or entirely replaying previous game actions
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/60 Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G11B 27/00 Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18

Abstract

Methods and systems for generating spherical game videos for virtual reality (VR) streaming and replay are disclosed. The method, when executed by a processor, comprises first recording game environment around a virtual camera array inserted into a source computer game that lacks spherical video output, using a first subset of virtual cameras within the virtual camera array. Next, receiving game metadata of a source game play of the source game, configuring a local game play according to the received game metadata, recording a video of the local game play using a second subset of virtual cameras, wherein the second subset of virtual cameras is not equal to the first subset of virtual cameras, and generating a spherical video from the recorded game environment and the recorded local game play. The present invention aids the mass adoption of VR technology by easily generating VR content from existing computer games.

Claims

1. A system for generating spherical videos from computer games, comprising:

a server having access to at least one processor; and

a non-transitory physical medium for storing a source computer game and for storing program code, wherein the non-transitory physical medium is accessible by the server, and wherein the program code when executed by the processor causes the processor to;

record a game environment around a virtual camera array inserted into the source computer game, using a first subset of virtual cameras within the virtual camera array, wherein the virtual camera array includes a plurality of virtual cameras each facing a different direction, and wherein the source computer game lacks spherical video output;

receive a game metadata of a source game play of the source computer game;

configure a local game play of the source computer game according to the received game metadata;

record a video of the local game play using a second subset of virtual cameras within the virtual camera array, wherein the second subset of virtual cameras is not equal to the first subset of virtual cameras; and

generate a spherical video from the recorded game environment and the recorded video of the local game play.

2. The system of claim 1, wherein the source computer game is selected from the group consisting of 2D games and 3D games.

3. The system of claim 1, wherein the source game play is a live game play.

4. The system of claim 1, wherein the virtual camera array is positioned at a fixed location within the source computer game.

5. The system of claim 1, wherein the spherical video generation is by aligning and stitching the game environment recorded by virtual cameras in the first subset but not in the second subset, to the video of the local game play recorded by the second subset of virtual cameras.

6. The system of claim 1, wherein at least one virtual camera within the second subset of virtual cameras records the local game play at a spatial or temporal frequency different from those of the virtual cameras within the first subset of virtual cameras.

21. The system of claim 1, wherein the spherical video has a 360-degree horizontal viewing angle.

22. The system of claim 1, wherein the virtual camera array was inserted into the source computer game using a capture client provided by a Software Development Kit (SDK) or a game connector module.

23. The system of claim 1, wherein the program code when executed by the processor further causes the processor to:

superimpose the spherical video with a 2D broadcast of the source game play, wherein the spherical video and the 2D broadcast of the source game play are synchronized in time.

7. A method for generating spherical videos from computer games, comprising:

recording a game environment around a plurality of virtual camera arrays inserted into a source computer game, wherein for each virtual camera array, a game environment is recorded using a first subset of virtual cameras within the virtual camera array, wherein each virtual camera array includes a plurality of virtual cameras each facing a different direction, and wherein the source computer game lacks spherical video output;

receiving a game metadata of a source game play of the source computer game;

configuring a local game play of the source computer game according to the received game metadata;

selecting a virtual camera array from the plurality of virtual camera arrays based on an optimized perspective or a user input;

recording a video of the local game play using a second subset of virtual cameras within the selected virtual camera array, wherein the second subset of virtual cameras is not equal to the first subset of virtual cameras within the selected virtual camera array used to record the game environment around the selected virtual camera array; and

generating a spherical video from the recorded game environment around the selected virtual camera array and the recorded video of the local game play.

8. The method of claim 7, wherein the optimized perspective is optimized over locations of the plurality of virtual camera arrays within a game map with respect to a set of positions within the game map, and contents of the game map.

9. The method of claim 7, wherein the optimized perspective is determined by analyzing a player field of view.

10. The method of claim 7, wherein the optimized perspective is optimized over a distance and a viewing angle towards a player position.

11. The method of claim 7, wherein the user input is from a broadcaster, a player, or a spectator.

12. The method of claim 7, wherein the selection of the virtual camera array is limited to a proper subset of the plurality of virtual camera arrays.

13. The method of claim 7, wherein the source computer game is selected from the group consisting of 2D games and 3D games.

14. The method of claim 7, wherein the source game play is a live game play.

15. The method of claim 7, wherein each of the plurality of virtual camera arrays is positioned at a fixed location within the source computer game.

16. The method of claim 7, wherein the spherical video generation is by aligning and stitching the game environment recorded by virtual cameras in the first subset but not in the second subset of virtual cameras within the selected virtual camera array, to the video of the local game play recorded by the second subset of virtual cameras within the selected virtual camera array.

17. A non-transitory computer-readable storage medium for generating spherical videos from computer games, the storage medium comprising a source computer game and program code stored thereon, that when executed by a processor causes the processor to:

record a game environment around a plurality of virtual camera arrays inserted into the source computer game, wherein for each virtual camera array, the game environment is recorded using a first subset of virtual cameras within the virtual camera array, wherein each virtual camera array includes a plurality of virtual cameras each facing a different direction, and wherein the source computer game lacks spherical video output;

receive a game metadata of a source game play of the source computer game;

configure a local game play of the source computer game according to the received game metadata;

select a virtual camera array from the plurality of virtual camera arrays based on an optimized perspective or a user input;

record a video of the local game play using a second subset of virtual cameras within the selected virtual camera array, wherein the second subset of virtual cameras is not equal to the first subset of virtual cameras within the selected virtual camera array used to record the game environment around the selected virtual camera array; and

generate a spherical video from the recorded game environment around the selected virtual camera array and the recorded video of the local game play.

18. The non-transitory computer-readable storage medium of claim 17, wherein the optimized perspective is optimized over locations of the plurality of virtual camera arrays within a game map with respect to a set of positions within the game map, and contents of the game map.

19. The non-transitory computer-readable storage medium of claim 17, wherein the optimized perspective is determined by analyzing a player field of view.

20. The non-transitory computer-readable storage medium of claim 17, wherein the optimized perspective is optimized over a distance and a viewing angle towards a player position.

24. The non-transitory computer-readable storage medium of claim 17, wherein the spherical video has a 360-degree horizontal viewing angle.

25. The non-transitory computer-readable storage medium of claim 17, wherein the plurality of virtual camera arrays was inserted into the source computer game using a capture client provided by a Software Development Kit (SDK) or a game connector module.

26. The non-transitory computer-readable storage medium of claim 17, wherein the program code when executed by the processor further causes the processor to:

superimpose the spherical video with a 2D broadcast of the source game play, wherein the spherical video and the 2D broadcast of the source game play are synchronized in time.