Three-dimensional model creation and rendering with improved virtual reality experience

• US 10,672,189 B2
• Filed: 01/25/2017
• Issued: 06/02/2020
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: Umbra Software Oy (Amazon.com, Inc.)
• Inventors: Makinen, Otso, Hatala, Antti, Saransaari, Hannu, Muurimaki, Jarno, Bushnaief, Jasin, Muszynski, Johann, Pulkki, Mikko, Jaba, Niilo, Laulajainen, Otto, Aijala, Turkka, Truong, Vinh
• Primary Examiner(s): He, Yingchun
• Application Number:
  US15/415,183
• Publication Number:
  US 20170287199A1
• Time in Patent Office: 1,224 Days
• Field of Search: None
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/837 Shooting of targets
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/005 Tree description, e.g. octree, quadtree
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2200/16 : involving adaptation to the client's capabilities
• CPC Class Codes: G06T 2210/36 : Level of detail
• CPC Class Codes: G06T 2210/61 : Scene description
• CPC Class Codes: G06T 3/4092 : Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: G09G 2350/00 : Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2370/022 : Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/16 : Use of wireless transmission of display information
• CPC Class Codes: G09G 5/006 : Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/4312 : involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/47 : End-user applications

Abstract

A cloud network server system, a method, and a software program product for compiling and presenting a three-dimensional (3D) model are provided. An end 3D model is composed from at least two pre-existing 3D models stored in the cloud network server system by combining the pre-existing 3D models. The end 3D model is partitioned into smaller cells. The system and method allow a drawing user to view and draw the end 3D model for example of a computer game, via a drawing user terminal computer. Based on a virtual location of the drawing user in the end 3D model, parts of at least one version of the end 3D model are rendered to the drawing user. The system and method render a more lifelike virtual reality gaming experience with substantially lesser time lag, lesser memory footprint requirement, and lesser production effort.

Claims

1. A method for compiling and streaming a three-dimensional graphical computer model, using at least one graphics engine distributed cloud network server system comprising a plurality of servers, and at least one user terminal computer, the method comprising:

composing an end three-dimensional graphical computer model on said at least one graphics engine distributed cloud network server system from at least two pre-existing 3D computer models stored in said at least one graphics engine distributed cloud network server system by combining said at least two pre-existing 3D computer models, the end three-dimensional graphical computer model being stored in a data storage on the at least one graphics engine distributed cloud network server system;

partitioning the end three-dimensional graphical computer model into smaller cells;

connecting a drawing user to a cloud server network with the at least one user terminal computer that is a drawing terminal computer installed with an application or a browser configured to;

determine dynamic location information of the user within the end three-dimensional graphical computer model from user inputs and transmit the dynamic location information to the cloud server network; and

view and draw into at least one end three-dimensional graphical computer model cell from said cloud server network;

streaming to the at least one user terminal only an amount of data of the end three-dimensional graphical computer model determined to provide a predetermined display resolution at a viewing distance of a viewing user;

displaying, based on a virtual location of the drawing user within the end three-dimensional graphical computer model, only the at least one cell of the end three dimensional graphical computer model visible from said virtual location of the drawing user; and

receiving input from the drawing user, using a third-party 3D authoring tool integrated with the graphics engine, and drawing into the virtual location within the at least one cell of the end three-dimensional graphical computer model.

2. The method as claimed in claim 1, further comprising:

connecting the viewing user to the cloud network with a viewing user terminal computer installed with a viewing application configured to view the at least one end three dimensional graphical computer model cell from said cloud server network; and

displaying, based on a virtual location of the viewing user within the end three-dimensional graphical computer model, only the at least one cell of the three dimensional graphical computer model visible from said virtual location of the viewing user.

3. The method as claimed in claim 2, wherein the viewing user terminal computer is a mobile terminal computer, such as a tablet and/or mobile phone, and the viewing application is a virtual reality software application.

4. The method as claimed in claim 1, wherein additional non-spatial information is included with spatial and visibility information of the three dimensional graphical computer model during drawing.

5. The method as claimed in claim 4, wherein the additional non-spatial information included with the spatial and visibility information of the end three-dimensional graphical computer model comprises any of the following:

triangle groups, reflectance and/or refractance attributes of light at a particular location in the three-dimensional graphical computer model, reflectance of sound at a particular location in the three-dimensional graphical computer model, smell, temperature, wind, quantity of natural light at a certain time, and/or air speed.

6. The method as claimed in claim 1, wherein the end three-dimensional graphical computer model is a voxel grid with portal graphs.

7. The method as claimed in claim 1, wherein an information structure of the end three dimensional graphical computer model in the end three dimensional graphical computer model that is drawn and uploaded to the cloud server network is configured to be transferred to and preserved in multiple different user applications using said end three-dimensional graphical computer model.

8. The method as claimed in claim 1, wherein the drawing terminal computer is a mobile terminal computer, such as a tablet and/or mobile phone or a desktop computer, and

drawing takes place via the viewing and drawing application or via a web browser.

9. A graphics engine cloud server network system for compiling and streaming a three dimensional graphical computer model comprising:

a plurality of servers; and

at least one user terminal computer,

an end three-dimensional graphical computer model is configured to be composed on said graphics engine cloud network server system from at least two pre-existing 3D computer models stored in said graphics engine cloud network server system by combining said at least two pre-existing 3D computer models, the end three-dimensional graphical computer model being stored in a data storage on the cloud network server system,

the end three-dimensional graphical computer model is configured to be partitioned into smaller cells,

a drawing user is configured to connect to the cloud server network with the at least one user terminal computer that is a drawing terminal computer installed with an application or a browser configured to;

determine dynamic location information of the user within the end three-dimensional graphical computer model from user inputs and transmit the dynamic location information to the cloud server network; and

view and draw into at least one end three-dimensional graphical computer model cell from said cloud server network,

wherein the graphics engine cloud server network system is configured to stream to the at least one user terminal only an amount of data of the end three-dimensional graphical computer model determined to provide a predetermined display resolution at a viewing distance of a viewing user;

based on a virtual location of the drawing user within the end three-dimensional graphical computer model, only the at least one cell of the end three-dimensional graphical computer model visible from said virtual location of the drawing user are configured to be displayed, and

input is received from the drawing user, using a third party 3D authoring tool integrated with the graphics engine, and the virtual location within the at least one cell of the end three-dimensional graphical computer model is drawn thereinto.

10. The cloud server network system as claimed in claim 9, wherein

the viewing user is configured to connect to the cloud server network with a viewing user terminal computer installed with a viewing application configured to view the at least one end three-dimensional graphical computer model cell from said cloud server network, and

based on a virtual location of the viewing user within the end three-dimensional graphical computer model, only the at least one cell of the three-dimensional graphical computer model visible from said virtual location of the viewing user are configured to be displayed.

11. The cloud server network system as claimed in claim 10, wherein the viewing user terminal computer is configured to be a mobile terminal computer, such as a tablet and/or a mobile phone, and

the viewing application is a virtual reality software application.

12. The cloud server network system as claimed in claim 9, wherein additional non-spatial information is configured to be included with spatial and visibility information of the three-dimensional graphical computer model during drawing.

13. The cloud server network system as claimed in claim 12, wherein said additional non-spatial information included with the spatial and visibility information of the three-dimensional graphical computer model is configured to comprise any of the following:

triangle groups, reflectance and/or refractance attributes of light at a particular location in the three-dimensional graphical computer model, reflectance of sound at a particular location in the three-dimensional graphical computer model, smell, temperature, wind, quantity of natural light at a certain time, and/or air speed.

14. The cloud server network system as claimed in claim 9, wherein the end three dimensional graphical computer model is configured as a voxel grid with portal graphs.

15. The cloud server network system as claimed in claim 9, wherein an information structure of the end three-dimensional graphical computer model in the end three-dimensional graphical computer model that is drawn and uploaded to the cloud server network is configured to be transferred to and preserved in multiple different user applications using said three-dimensional graphical computer model.

16. The cloud server network system as claimed in claim 9, wherein the drawing terminal computer is configured to be a mobile terminal computer, such as a tablet and/or a mobile phone or a desktop computer, and

drawing takes place via the viewing and drawing application or via a web browser.

17. A software program product stored in a non-transitory computer readable medium for compiling and presenting a three-dimensional graphical computer model is executable in at least one graphics engine distributed cloud network server system including a plurality of servers and at least one user terminal computer, the software program product causing a processor to execute a method comprising:

composing an end three-dimensional graphical computer model on said at least one graphics engine distributed cloud network server system from at least two preexisting 3D computer models stored in said at least one graphics engine distributed cloud network server system by combining said at least two pre-existing 3D computer models, the end three-dimensional graphical computer model being stored in a data storage on the at least one distributed cloud network server system;

partitioning the end three-dimensional graphical computer model into smaller cells;

connecting a drawing user to a cloud server network with the at least one user terminal computer that is a drawing terminal computer installed with an application or a browser configured to;

determining dynamic location information of the user within the end three-dimensional graphical computer model from user inputs and transmit the dynamic location information to the cloud server network; and

view and draw into at least one end three dimensional graphical computer model cell from said cloud server network;

streaming to the at least one user terminal only an amount of data of the end three-dimensional graphical computer model determined to provide a predetermined display resolution at a viewing distance of a viewing user;

displaying, based on a virtual location of the drawing user within the end three-dimensional graphical computer model, only the at least one cell of the end three dimensional graphical computer model visible from said virtual location of the drawing user; and

receiving input from the drawing user, using a third party 3D authoring tool integrated with the graphics engine, and drawing into the virtual location within the at least one cell of the end three-dimensional graphical computer model.

18. The software program as claimed in claim 17, further comprising:

connecting the viewing user to the cloud server network with a viewing user terminal computer used by a viewing user and installed with a viewing application configured to view the at least one end three-dimensional graphical computer model cell from said cloud server network; and

displaying, based on a virtual location of the viewing user within the end three-dimensional graphical computer model, only the at least one cell of the three dimensional graphical computer model visible from said virtual location of the viewing user.

19. The software program as claimed in claim 18, wherein the viewing user terminal computer is a mobile terminal computer, such as a tablet and/or a mobile phone, and

the viewing application is a virtual reality software application.

20. The software program as claimed in claim 17, wherein additional non-spatial information is included with spatial and visibility information of the three-dimensional graphical computer model during drawing.

21. The software program as claimed in claim 20, wherein said additional non-spatial information included with the spatial and visibility information of the end three-dimensional graphical computer model comprises any of the following:

triangle groups, reflectance and/or refractance attributes of light at a particular location in the end three-dimensional graphical computer model, reflectance of sound at a particular location in the end three-dimensional graphical computer model, smell, temperature, wind, quantity of natural light at a certain time, and/or air speed.

22. The software program as claimed in claim 17, wherein the end three-dimensional graphical computer model is a voxel grid with portal graphs.

23. The software program as claimed in claim 17, wherein an information structure of the end three-dimensional graphical computer model in the end three-dimensional graphical computer model that is drawn and uploaded to the cloud server network is configured to be transferred to and preserved in multiple different user applications using said end three-dimensional graphical computer model.

24. The software program as claimed in claim 17, wherein the drawing terminal computer is a mobile terminal computer, such as a tablet and/or a mobile phone or a desktop computer, and

drawing takes place via the viewing and drawing or via a web browser.