Three-dimensional modelling with improved virtual reality experience

• US 10,713,845 B2
• Filed: 03/25/2019
• Issued: 07/14/2020
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: Umbra Software Oy (Amazon.com, Inc.)
• Inventors: Makinen, Otso, Hatala, Antti, Saransaari, Hannu, Muurimaki, Jarno, Bushnaief, Jasin, Muszynski, Johann, Pulkki, Mikko, Jaba, Niilo, Laulajainen, Otto, Aijala, Turkka, Truong, Vinh
• Primary Examiner(s): Wu, Chong
• Application Number:
  US16/363,179
• Publication Number:
  US 20190221038A1
• Time in Patent Office: 477 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/837 Shooting of targets
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/005 Tree description, e.g. octree, quadtree
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2200/16 : involving adaptation to the client's capabilities
• CPC Class Codes: G06T 2210/36 : Level of detail
• CPC Class Codes: G06T 2210/61 : Scene description
• CPC Class Codes: G06T 3/4092 : Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: G09G 2350/00 : Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2370/022 : Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/16 : Use of wireless transmission of display information
• CPC Class Codes: G09G 5/006 : Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/4312 : involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/47 : End-user applications

Abstract

A cloud network server system, a method, and a software program product for experiencing a three-dimensional (3D) model are provided. 3D model data associated with a 3D video game is uploaded to the cloud network server system. The system and method are used to design for example a computer game that renders non-spatial characteristics such as, smell, reflection and/or refraction of light, wind direction, sound reflection, etc., along with spatial and visibility information associated with 3D objects displayed in the 3D video game. Different versions of the 3D model are created based on memory, streaming bandwidth, and/or processing power requirements of different user terminal computers. Based on a virtual location of a user in the 3D model, parts of at least one version of the 3D model are rendered to the user.

Claims

1. A method for experiencing a three-dimensional graphical computer model, comprises at least one distributed cloud network server system comprising a plurality of servers, at least one model upload computer and at least one user terminal computer, wherein,

the three-dimensional graphical computer model data is uploaded to the cloud network from the at least one model upload computer,

the three-dimensional graphical computer model is composed from the data and is spatially subdivided into an octree data structure of cells, wherein higher nodes in the octree data structure fill larger regions in space and have less detailed computational parameters,

a user connects to the cloud network with the at least one user terminal computer installed with an application configured for viewing the three-dimensional graphical computer model from said cloud server network,

based on a virtual location of the user within the three-dimensional graphical computer model, active tree nodes of the octree data structure are traversed and objects in the octree data structure that are optimized for a selected viewing distance from the virtual location are prioritized for streaming to the at least one user terminal computer and are displayed on a screen of the at least one user terminal computer to the user, and

additional non-spatial information is included with spatial and visibility information of the three-dimensional graphical computer model.

2. A method as claimed in claim 1, wherein said additional non-spatial information included with the spatial information of the three-dimensional graphical computer model comprises any of the following:

reflectance and/or refractance attributes of light at a particular location in the three-dimensional graphical computer model, reflectance of sound at a particular location in the three-dimensional graphical computer model, smell, temperature, wind, quantity of natural light at a certain time and/or air speed.

3. A method as claimed in claim 1, wherein parts of the three-dimensional graphical computer model and associated non-spatial information are streamed to the application on said at least one user terminal computer.

4. A method as claimed in claim 3, wherein said streaming prioritises the cell the user is currently in, or cells that are visible from the cell that the user is currently in, or are near the cell the user is currently in.

5. A method as claimed in claim 4, wherein the complexity of the three-dimensional graphical computer model is adjusted based on the available streaming bandwidth to the at least one user terminal computer and/or memory and/or processing power of said at least one user terminal computer.

6. A method as claimed in claim 5, wherein different versions of different sizes are created from the three-dimensional graphical computer model for different bandwidth, memory and/or processing power environments.

7. A method as claimed in claim 6, wherein the plurality of different versions of the three-dimensional graphical computer model are hosted in the cloud network server system.

8. A method as claimed in claim 1, wherein the cells are connected by portals that appear as doors or corridors in the three-dimensional graphical computer model.

9. A method as claimed in claim 1, wherein the three-dimensional graphical computer model is a voxel grid with portal graphs.

10. A method as claimed in claim 1, wherein the at least one user terminal computer is a mobile terminal computer, including one or more of a tablet or mobile phone, and wherein the viewing application is a virtual reality software application.

11. A method as claimed in claim 1, wherein an information structure of the three-dimensional graphical computer model in the model upload computer and cloud server network can be transferred to and preserved in multiple different user applications using said three-dimensional graphical computer model.

12. A software program product stored on a non-transitory computer readable medium for providing a three-dimensional graphical computer model is executable in at least one distributed cloud network server system comprising a plurality of servers, at least one model upload computer and at least one user terminal computer, wherein,

three-dimensional graphical computer model data is configured to be uploaded to the cloud network from the at least one model upload computer,

the three-dimensional graphical computer model is configured to be composed from the data and is spatially subdivided into an octree data structure of cells, wherein higher nodes in the octree data structure fill larger regions in space and have less detailed computational parameters,

a user connects to the cloud network with the at least one user terminal computer installed with an application configured for viewing the three-dimensional graphical computer model from said cloud server network,

based on a virtual location of the user within the three-dimensional graphical computer model, active tree nodes of the octree data structure are traversed and objects in the octree data structure that are optimized for a selected viewing distance from the virtual location is sent are prioritized for streaming to the at least one user terminal computer and are displayed on a screen of the at least one user terminal computer to the user, and

additional non-spatial information is included with spatial and visibility information of the three-dimensional graphical computer model.

13. A software program product as claimed in claim 12, wherein said additional non-spatial information included with the spatial information of the three-dimensional graphical computer model comprises any of the following:

reflectance and/or refractance attributes of light at a particular location in the three-dimensional graphical computer model, reflectance of sound at a particular location in the three-dimensional graphical computer model, smell, temperature, wind, quantity of natural light at a certain time and/or air speed.

14. A software program product as claimed in claim 12, wherein parts of the three-dimensional graphical computer model and associated non-spatial information are streamed to the application on said at least one user terminal computer.

15. A software program product as claimed in claim 14, wherein said streaming prioritises the cell the user is currently in, or cells that are visible from the cell that the user is currently in, or are near the cell the user is currently in.

16. A software program product as claimed in claim 15, wherein the complexity of the three-dimensional graphical computer model is adjusted based on the available streaming bandwidth to the at least one user terminal computer and/or memory and/or processing power of said at least one user terminal computer.

17. A software program product as claimed in claim 16, wherein different versions of different sizes are created from the three-dimensional graphical computer model for different bandwidth, memory and/or processing power environments.

18. A software program product as claimed in claim 17, wherein the plurality of different versions of the three-dimensional graphical computer model are hosted in the cloud network server system.

19. A software program product as claimed in claim 12, wherein the cells are connected by portals that appear as doors or corridors in the three-dimensional graphical computer model.

20. A software program product as claimed in claim 12, wherein the three-dimensional graphical computer model is a voxel grid with portal graphs.

21. A software program product as claimed in claim 12, wherein the at least one user terminal computer is a mobile terminal computer, including one or more of a tablet or mobile phone, and wherein the viewing application is a virtual reality software application.

22. A software program product as claimed in claim 12, wherein the information structure of the three-dimensional graphical computer model in the model upload computer and cloud server network can be transferred to and preserved in multiple different user applications using said three-dimensional graphical computer model.

23. A cloud server network system for providing a three-dimensional graphical computer comprising a plurality of servers, at least one model upload computer and at least one user terminal computer, wherein,

three-dimensional graphical computer model data is configured to be uploaded to the cloud network from the at least one model upload computer,

the three-dimensional graphical computer model is configured to be composed from the data and is spatially subdivided into an octree data structure of cells, wherein higher nodes in the octree data structure fill larger regions in space and have less detailed computational parameters,

a user is configured to connect to the cloud network with the at least one user terminal computer installed with an application configured for viewing the three-dimensional graphical computer model from said cloud server network,

based on a virtual location of the user within the three-dimensional graphical computer model, active tree nodes of the octree data structure are traversed and objects in the octree data structure that are optimized for a selected viewing distance from the virtual location are prioritized for streaming to the at least one user terminal computer and are configured to be displayed on a screen of the at least one user terminal computer to the user, and

additional non-spatial information is included with spatial and visibility information of the three-dimensional graphical computer model.

24. A cloud server network system as claimed in claim 23, wherein said additional non-spatial information included with the spatial information of the three-dimensional graphical computer model is configured to comprise any of the following:

reflectance and/or refractance attributes of light at a particular location in the three-dimensional graphical computer model, reflectance of sound at a particular location in the three-dimensional graphical computer model, smell, temperature, wind, quantity of natural light at a certain time and/or air speed.

25. A cloud server network system as claimed in claim 23, wherein parts of the three-dimensional graphical computer model and associated non-spatial information are configured to be streamed to the application on said at least one user terminal computer.

26. A cloud server network system as claimed in claim 25, wherein said streaming is configured to prioritise the cell the user is currently in, or cells that are visible from the cell that the user is currently in, or are near the cell the user is currently in.

27. A cloud server network system as claimed in claim 26, wherein the complexity of the three-dimensional graphical computer model is configured to be adjusted based on the available streaming bandwidth to the at least one user terminal computer and/or memory and/or processing power of said at least one user terminal computer.

28. A cloud server network system as claimed in claim 27, wherein different versions of different sizes are configured to be created from the three-dimensional graphical computer model for different bandwidth, memory and/or processing power environments.

29. A cloud server network system as claimed in claim 28, wherein the plurality of different versions of the three-dimensional graphical computer model is configured to be hosted in the cloud network server system.

30. A cloud server network system as claimed in claim 23, wherein the cells are configured to be connected by portals that may appear as doors or corridors in the three-dimensional graphical computer model.

31. A cloud server network system as claimed in claim 23, wherein the three-dimensional graphical computer model is configured as a voxel grid with portal graphs.

32. A cloud server network system as claimed in claim 23, wherein the at least one user terminal computer is configured to be a mobile terminal computer, including one or more of a tablet or mobile phone, and wherein the viewing application is configured to be a virtual reality software application.

33. A cloud server network system as claimed in claim 23, wherein the information structure of the three-dimensional graphical computer model in the model upload computer and cloud server network is configured to be transferred to and preserved in multiple different user applications using said three-dimensional graphical computer model.