System and method for augmented and virtual reality

• US 10,637,897 B2
• Filed: 10/07/2019
• Issued: 04/28/2020
• Est. Priority Date: 10/28/2011
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Meky, Moustafa M
• Application Number:
  US16/594,655
• Publication Number:
  US 20200053137A1
• Time in Patent Office: 204 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 65/4015 : where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/14 : Multichannel or multilink protocols

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A wearable display system comprising:

a display lens configured to direct projected light toward an eye of a user of the wearable display system, the display lens further configured to selectively allow outside light from an outside environment to be transmitted through the display lens toward the eye of the user;

a rendering engine configured to generate rendering information describing virtual objects to be presented to the user through the projected light; and

a processor communicatively coupled to the display lens and the rendering engine, the processor executing instructions that cause the processor to perform operations comprising;

determining, from a plurality of available functional modes, a functional mode in which to operate the wearable display system; and

responsive to determining the functional mode;

i) instructing the display lens to selectively transmit at least a portion of the outside light through the display lens according to the determined functional mode, and ii) instructing the rendering engine to selectively generate, according to the determined functional mode, the rendering information that describes at least one virtual object to be presented through the projected light.

2. The wearable display system of claim 1, wherein the functional mode is determined automatically.

3. The wearable display system of claim 1, wherein determining the functional mode includes receiving a selection of the functional mode that is made by the user through an interface associated with the wearable display system.

4. The wearable display system of claim 1, wherein:

the plurality of available functional modes includes an augmented mode; and

responsive to determining the augmented mode as the functional mode, instructing the display lens to be substantially transparent to provide a view of the outside environment, and instructing the rendering engine to generate the rendering information that describes the at least one virtual object to be presented with the view of the outside environment.

5. The wearable display system of claim 4, wherein the at least one virtual object is presented at a position that is associated with at least one physical object that is present in the outside environment.

6. The wearable display system of claim 1, wherein:

the plurality of available functional modes includes a virtual mode; and

responsive to determining the virtual mode as the functional mode, instructing the display lens to block a view of the outside environment, and instructing the rendering engine to generate the rendering information that describes the at least one virtual object to be presented without the view of the outside environment.

7. The wearable display system of claim 1, wherein:

the plurality of available functional modes includes a blended mode; and

responsive to determining the blended mode as the functional mode, instructing the rendering engine to generate the rendering information that describes the at least one virtual object as a virtual rendering of at least one physical object that is present in the outside environment.

8. The wearable display system of claim 7, further comprising at least one sensor that generates sensor data describing the at least one physical object present in the outside environment, wherein the rendering information is generated based at least partly on the sensor data.

9. The wearable display system of claim 1, wherein instructing the display lens to selectively transmit at least a portion of the outside light through the display lens includes instructing the display lens to alter its opacity.

10. The wearable display system of claim 1, wherein instructing the display lens to selectively transmit at least a portion of the outside light through the display lens includes instructing the display lens to manipulate a cover that at least partly blocks the outside light from the display lens.

11. A method for controlling operation of a wearable display system, the method comprising:

determining, from a plurality of available functional modes, a functional mode in which to operate the wearable display system that includes a display lens configured to direct projected light toward an eye of a user of the wearable display system, the display lens further configured to selectively allow outside light from an outside environment to be transmitted through the display lens toward the eye of the user, the wearable display system further include a rendering engine configured to generate rendering information describing virtual objects to be presented to the user through the projected light; and

responsive to determining the functional mode;

i) instructing the display lens to selectively transmit at least a portion of the outside light through the display lens according to the determined functional mode, and ii) instructing the rendering engine to selectively generate, according to the determined functional mode, the rendering information that describes at least one virtual object to be presented through the projected light.

12. The method of claim 11, wherein the functional mode is determined automatically.

13. The method of claim 11, wherein determining the functional mode includes receiving a selection of the functional mode that is made by the user through an interface associated with the wearable display system.

14. The method of claim 11, wherein:

the plurality of available functional modes includes an augmented mode; and

responsive to determining the augmented mode as the functional mode, instructing the display lens to be substantially transparent to provide a view of the outside environment, and instructing the rendering engine to generate the rendering information that describes the at least one virtual object to be presented with the view of the outside environment.

15. The method of claim 14, wherein the at least one virtual object is presented at a position that is associated with at least one physical object that is present in the outside environment.

16. The method of claim 11, wherein:

the plurality of available functional modes includes a virtual mode; and

responsive to determining the virtual mode as the functional mode, instructing the display lens to block a view of the outside environment, and instructing the rendering engine to generate the rendering information that describes the at least one virtual object to be presented without the view of the outside environment.

17. The method of claim 11, wherein:

the plurality of available functional modes includes a blended mode; and

responsive to determining the blended mode as the functional mode, instructing the rendering engine to generate the rendering information that describes the at least one virtual object as a virtual rendering of at least one physical object that is present in the outside environment.

18. The method of claim 17, wherein:

The wearable display system further comprises at least one sensor that generates sensor data describing the at least one physical object present in the outside environment; and

the rendering information is generated based at least partly on the sensor data.

19. The method of claim 11, wherein instructing the display lens to selectively transmit at least a portion of the outside light through the display lens includes instructing the display lens to alter its opacity.

20. The method of claim 11, wherein instructing the display lens to selectively transmit at least a portion of the outside light through the display lens includes instructing the display lens to manipulate a cover that at least partly blocks the outside light from the display lens.