System and method for augmented and virtual reality

• US 10,629,003 B2
• Filed: 03/11/2014
• Issued: 04/21/2020
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Abovitz, Rony
• Primary Examiner(s): Patel, Nitin
• Assistant Examiner(s): Stone, Robert M
• Application Number:
  US14/205,126
• Publication Number:
  US 20140306866A1
• Time in Patent Office: 2,233 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A user display device, comprising:

a housing frame mountable on a head of a user having eyes;

a first pair of cameras coupled to the housing frame to track a movement of the eyes and to estimate a depth of focus based on the movement of the eyes;

a projection module having a light generating mechanism to generate and focus, based on at least the estimate of the depth of focus, a projected light associated with a display object;

an environment sensing system to generate a digital rendering of at least a portion of an environment of the user when the user display device is in a virtual reality mode, wherein the display object is a rendered physical body part of the user as it appears in the environment of the user when the user display device is in a virtual reality mode, the digital rendering of the physical body part being generated in real time based on a captured field-of-view image and being generated in response to a determination that the physical body part is in a field-of-view of the user, the environment sensing system generating the rendered physical body part using an image-based 3D reconstruction software to digitally reconstruct the physical body part as the digital rendering;

a lens mounted on the housing frame, a first portion of the lens configured to transition from a transparent setting to an opaque setting to block visible light from an outside environment from passing through the lens as part of entering the virtual reality mode from an augmented reality mode, wherein the first portion of the lens is configured to transition from the transparent setting to the opaque setting in response to a selection of the virtual reality mode corresponding to projecting light associated with the display object and blocking light from the outside environment;

wherein the lens further comprises at least one transparent mirror positioned in front of the eyes to reflect the projected light into the eyes; and

a processor communicatively coupled to the projection module to communicate data associated with a display image to the projection module.

2. The user display device of claim 1, wherein the at least one transparent mirror is configured to selectively allow a transmission of light from the outside environment.

3. The user display device of claim 1, further comprising a second pair of cameras mountable on the housing frame to capture a field-of-view image of an eye corresponding to each of the second pair of cameras.

4. The user display device of claim 3, wherein the processor is configured to calculate a head pose of the user based on the field-of-view images.

5. The user display device of claim 1, wherein the projection module comprises a scanned laser arrangement to modify the projected light associated with the display object based on the estimate of the depth of focus.

6. The user display device of claim 5, wherein the projected light is a light beam having a diameter less than 0.7 mm.

7. The user display device of claim 1, wherein the first pair of cameras comprises infrared cameras paired with infrared light sources to track the movement of the eyes.

8. The user display device of claim 1, further comprising a sensor assembly comprising at least one sensor to sense at least one of a movement of the user, a location of the user, a direction of the user and an orientation of the user.

9. The user display device of claim 8, wherein the at least one sensor is at least one of an accelerometer, a compass and a gyroscope.

10. The user display device of claim 8, wherein the processor is configured to estimate a head pose of the user based on the at least one of the movement of the user, the location of the user, the direction of the user, and the orientation of the user.

11. The user display device of claim 1, further comprising a GPS system.

12. The user display device of claim 1, wherein the processor is communicatively coupled to a computer network to transmit at least a portion of a virtual world data, and receive another portion of the virtual world data.

13. The user display device of claim 1, further comprising an audio speaker module mountable on the housing frame to output sounds.

14. The user display device of claim 1, further comprising a microphone mountable on the housing frame to capture sounds local to the user.

15. The user display device of claim 1, wherein the projection module is configured to modify other projected light associated with a second object that is not the display object such that the second object appears blurred.

16. The user display device of claim 1, wherein the processor is configured to render frames of the display object at a rate of at least 60 frames per second.

17. The user display device of claim 1, further comprising a haptic interface device communicatively coupled to the projection module to provide tactile feedback.

18. The user display device of claim 1, wherein the display object is at least one of a virtual object, a rendered physical object, an image and a video.

19. The user display device of claim 1, further comprising an environment sensing system to digitally reconstruct an environment of the user.

20. The user display device of claim 1, wherein the lens is configured to transition from the transparent setting to the opaque setting by employing a cover to block the lens.

21. The user display device of claim 1, wherein the lens is configured to transition from the transparent setting to the opaque setting based at least in part on a selected mode of the user display device.

22. The user display device of claim 21, wherein the selected mode is a virtual reality mode, and wherein the lens is in the opaque setting.

23. The user display device of claim 1, wherein the augmented reality mode is selected from at least an augmented reality mode and a virtual reality mode.