SYSTEM AND METHOD FOR AUGMENTED AND VIRTUAL REALITY

• US 20140306866A1
• Filed: 03/11/2014
• Published: 10/16/2014
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Abovitz, Rony
• Granted Patent: US 10,629,003 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

Claims

1. A user display device, comprising:

a housing frame mountable on a head of a user;

a first pair of cameras coupled to the housing frame to track a movement of the user'"'"'s eyes and to estimate a depth of focus based on the tracked eye movements;

a projection module having a light generating mechanism to generate and modify, based on the estimated depth of focus, a projected light associated with a display object such that the display object appears to be in focus;

a lens mounted on the housing frame; and

a processor communicatively coupled to the projection module to communicate data associated with the display image to the projection module.

2. The user display device of claim 1, wherein the lens further comprises at least one transparent mirror positioned in front of the user'"'"'s eyes to bounce the projected light into the user'"'"'s eyes.

3. The user display device of claim 2, wherein the at least one transparent mirror selectively allows a transmission of light from the local environment.

4. The user display device of claim 1, further comprising a second pair of cameras mountable on the housing frame to capture a field-of-view image of an eye corresponding to each of the second pair of cameras.

5. The user display device of claim 4, wherein the processor calculates a head pose of the user based on the captured field-of-view images.

6. The user display device of claim 1, wherein the projection module comprises a scanned laser arrangement to modify the projected light associated with the display object based on the estimated depth of focus.

7. The user display device of claim 6, wherein the projected light is a light beam having a diameter less than 0.7 mm.

8. The user display device of claim 1, wherein the first pair of cameras comprises infrared cameras paired with infrared light sources to track a movement of each of the user'"'"'s eyes.

9. The user display device of claim 1, further comprising a sensor assembly comprising at least one sensor to sense at least one of a movement of the user, a location of the user, a direction of the user and an orientation of the user.

10. The user display device of claim 9, wherein the at least one sensor is at least one of an accelerometer, a compass and a gyroscope.

11. The user display device of claim 9, wherein the processor estimates a head pose of the user based on the at least one of the movement of the user, the location of the user, the direction of the user, and the orientation of the user.

12. The user display device of claim 1, further comprising a GPS system.

13. The user display device of claim 1, wherein the processor is communicatively coupled to a computer network to transmit at least a portion of a virtual world data, and receive another portion of the virtual world data.

14. The user display device of claim 1, further comprising an audio speaker module mountable on the head frame to output sounds.

15. The user display device of claim 1, further comprising a microphone mountable on the housing frame to capture sounds local to the user.

16. The user display device of claim 1, wherein the projection module modifies another projected light associated with another object that is not the display object such that the other object appears blurred.

17. The user display device of claim 1, wherein the processor renders frames of the display object at a rate of at least 60 frames per second.

18. The user display device of claim 1, further comprising a haptic interface device communicatively coupled to the projection module to provide tactile feedback.

19. The user display device of claim 1, wherein the display object is at least one of a virtual object, a rendered physical object, an image and a video.

20. The user display device of claim 1, further comprising an environment sensing system to digitally reconstruct an environment of the user.

21. A method, comprising:

tracking a movement of a user'"'"'s eyes;

estimating a depth of focus of the user'"'"'s eyes based on the tracked eye movements;

modifying a light beam associated with a display object based on the estimated depth of focus such that the display object appears in focus; and

projecting the modified light beam into the user'"'"'s eyes.

22. The method of claim 21, further comprising:

selectively allowing a transmission of light from a local environment of the user based on a visualization mode of the display object, wherein the visualization mode is one of an augmented reality mode, a virtual reality mode, and a combination of augmented and virtual reality modes.

23. The method of claim 21, further comprising:

capturing a field-of-view image of each of the user'"'"'s eyes.

24. The method of claim 23, further comprising:

estimating a head pose of the user based on the at least one field-of-view image of the user'"'"'s eyes.

25. The method of claim 23, further comprising:

converting at least one physical object in the captured field-of-view image to a physically rendered virtual object; and

displaying the physically rendered virtual object to the user.

26. The method claim 23, further comprising:

extracting a set of points in the captured field-of-view image; and

creating a fiducial for at least one physical object in the captured field-of-view image based on the extracted set of points.

27. The method of claim 26, further comprising:

transmitting the at least one of the extracted set of points and the created fiducial to a cloud computer; and

tagging the at least one of the extracted set of points and the created fiducial to a type of object.

28. The method of claim 27, further comprising:

recognizing a different physical object as belonging to the type of object based on at least one of the tagged set of points associated with the type of object and the tagged created fiducial associated with the type of object.

29. The method claim 21, wherein the projected light is a light beam having a diameter less than 0.7 mm.

30. The method of claim 21, further comprising:

sensing at least one of a movement of the user, a location of the user, a direction of the user and an orientation of the user; and

calculating a pose of the user based on the at least one sensed movement, sensed location, sensed direction and sensed orientation.

31. The method of claim 30, wherein the at least one sensor is at least one of an accelerometer, a compass and a gyroscope.

32. The method of claim 21, further comprising:

processing a virtual world data associated with the display object to a computer network; and

transmitting at least a portion of the virtual world data associated with the display object to a second user located at a second location such that the second user may experience the at least portion of the virtual world data associated with the display object at the second location.

33. The method of claim 32, further comprising:

sensing a physical object; and

modifying, based on a predetermined relationship with the sensed physical object, at least a portion of the virtual world data associated with the display object.

34. The method of claim 33, presenting the modified virtual world data to the second user.

35. The method of claim 21, further comprising:

modifying another light associated with another object that is not the display object such that the other object appears blurred.

36. The method of claim 21, further comprising;

receiving user input through a user interface; and

modifying the display object based on the received user input.

37. The method of claim 36, wherein the user interface is at least one of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device and an audio input device.

38. The method of claim 21, wherein the display object is at least one of a virtual object, a rendered physical object, an image and a video.

39-87. -87. (canceled)