System and method for augmented and virtual reality

• US 10,021,149 B2
• Filed: 08/16/2016
• Issued: 07/10/2018
• Est. Priority Date: 10/28/2011
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Meky, Moustafa M
• Application Number:
  US15/238,657
• Publication Number:
  US 20160359930A1
• Time in Patent Office: 693 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 65/4015 : where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/14 : Multichannel or multilink protocols

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A system for enabling at least one user to interact within a virtual world comprising virtual world data, comprising:

a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; and

a first user device, configured to be operated by a first user, comprising an environment-sensing system and a user-sensing system,

wherein the first user device is operatively coupled to the computer network,

wherein the environment-sensing system is configured to capture a local environment audio input,

wherein the user-sensing system is configured to capture a user audio input from the first user,

wherein at least a first portion of the virtual world data is the local environment audio input,

wherein at least a second portion of the virtual world data is the user audio input from the first user;

wherein the virtual world further comprises a visual rendering by at least one of the computer servers or the one or more computing devices; and

wherein the visual rendering of the virtual world is presented in a three-dimensional format.

2. The system of claim 1, wherein at least a portion of the virtual world changes in response to a change in the virtual world data.

3. The system of claim 1, wherein at least a portion of the virtual world changes in response to a change in the audio input captured by the first user device.

4. The system of claim 1, wherein the visual rendering of the virtual world is presented in a two-dimensional format.

5. The system of claim 1, wherein the user device is operable to provide an interface for enabling interaction between a user and the virtual world in an augmented reality mode.

6. The system of claim 1, wherein the user device is operable to provide an interface for enabling interaction between a user and the virtual world in a virtual reality mode.

7. The system of claim 1, wherein the user device is operable to provide an interface for enabling interaction between a user and the virtual world in a combination of augmented and virtual reality modes.

8. The system of claim 1, wherein the virtual world data is transmitted over a data network.

9. The system of claim 1, wherein the computer network is operable to transmit the first portion of the virtual world data to a second user device for presentation to a second user, such that the second user may hear the audio input of the first portion of the virtual world data at a location of the second user.

10. The system of claim 9, wherein the first and second users are in different physical locations.

11. A system for enabling at least one user to interact with a virtual world comprising virtual world data, comprising:

a first user device operatively coupled to a computer network, the computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process a first portion of the virtual world data;

a user-sensing system operatively coupled to the first user device and configured to capture at least a first input from a first user, wherein the user-sensing system is configured to transmit the first input to the computer network; and

an environment-sensing system operatively coupled to the first user device and configured to capture at least a second input from a local environment of the first user device, wherein the environment-sensing system is configured to transmit the second input to the computer network,

wherein the computer network is configured to alter virtual world data based on at least one of the first input or the second input,

wherein the first user device is configured to present virtual content to the first user based on altered virtual world data

wherein the virtual world further comprises a visual rendering by at least one of the computer servers or the one or more computing devices; and

wherein the visual rendering of the virtual world is presented in a three-dimensional format.

12. The system of claim 11, wherein the virtual content is selected from a group consisting of visual content, audio content, or haptic content.

13. The system of claim 11, wherein the first input from the first user is selected from a group consisting of gesture data, audio data, geographic location data, or optical characteristic data.

14. The system of claim 11, wherein the second input from the local environment is selected from a group consisting of audio data from the local environment and visual data from a physical object in the local environment.

15. The system of claim 11, further comprising a gateway coupled between the computer network and the first user device.

16. The system of claim 11, further comprising a second user device configured to receive the virtual content.

17. The system of claim 16, wherein the second user device is configured to present the virtual content to a second user.